Testing is Getting Noisy

If you ’ re reading this, you are believably already mindful that the spotlight on testing is brighter than ever before. The research is clear that customer experience, and therefore application quality, is paramount to the success of a digital business. Here are just a few stats:

• According to McKinsey & amp; Company, 25 % of customers will vacate a brand after just one bad experience.

• The Harvard Business Review reported that client who had the better retiring experience spend 140 % more than customers who had the poorest retiring experience.

• Qualtrics found that 41 % of customer either decrease spending or stop spending completely after a bad experience.

Now more than ever, leadership desire to realise how their quality scheme is mitigating the risk of a poor user experience, and helping build digital confidence throughout their organisation. “ Automate more! ” they say confidently, knowing that manual testing creates bottlenecks to turn new features to client. But as test automation begin to scale, new questions arise. Mainly, how do we know if our testing efforts are positively impacting the business? Is this work?

In the Sauce Labs 2019 Continuous Testing Benchmark, it was reported that 80 % of our exploiter have a trial pass rate of less than 90 % (please see above icon). This means that for most users, at least 10 % of their tests need to be inquire. And for client that run decade, or even hundreds of thousands of test a day, this is not an peanut subset. Leadership sees failure rates, and wants to translate why these failures are befall and how promptly they can be fixed. Developers want to keep writing new feature and introduce, not track down and fixing problem tests in what seems to be a never-ending project. It ’ s a constant balancing act, and a common conflict for teams.

To aid ease this burden, data insights can aid proactively measure testing efficaciousness, and can still uncover where failures occur. But that ’ s where most of these measurements end— they tell you where the problems are, but not how to fix them. To truly make a conflict:

SUSA automates exploratory testing with persona-driven behavior, catching bugs that scripted automation misses.

• Developers and QA take a smart way to translatewhytests are failing, and insight intohow often those types of failures are hap across the test suite. This way, they can focus on the most permeant issues, and get the most “ smash for their dollar ” when fastening tests.

• Management want a single panorama that shows the on-going health of their quality strategy, and how it justifies their spend (on tools, citizenry, and mechanisation in general), and builds confidence that test is mitigating the risk of poor user experience.

What is Sauce Doing About This?

Our team has been interviewing customers of all sizing, and across manufacture, to understand how our Uninterrupted Testing Cloud can help make them best testers. One of the most commonly discussed theme with our customers is the concept of failure analysis. How can we leverage test datum with technology to volunteer brainwave into where the most pervasive issue are in a test suite, and help make sense of all of the noise test failures can create?

To that end, we are happy to announce the accessibility of our newest Insights feature: Failure Analysis. Using sophisticated machine learning that act with test pass/fail data, Failure Analysis surfaces the most common test errors, prioritizes them establish on how frequently they are occurring, and gives you insight into where you can focalize endeavor to improve your walk rates. This better developer efficiency, as they are repair the most pervasive issues more quickly. And leadership can see the plus impingement of testing sweat, and rest ensure that their investment in mechanisation is directly influencing user experience.

Trying Failure Analysis

Our team is stir to offer Failure Analysis to our customers and get your feedback on this new and evolving feature. Read below to see if you restrict for early access to Failure Analysis:

• Failure Analysis currently works with any Selenium or Appium tests. It does not act for XCUITest or Espresso at this time.

• For now, this feature is usable to our Enterprise contract customers. Anyone on our online subscription program, free trial, or Open Sauce will get access at a ulterior date.

• To get utilize Failure Analysis, you must configure your tests to send pass/fail data to Sauce Labs. This feature uses machine discover to examine pass/fail data and present common failure patterns. To enable this, please reference this in our documentation.

If you ’ re interested in trying out Failure Analysis, please send an email to, and our team will respond with more information on getting started.