In the creation of software testing, tests sometimes fail. That ’ s a fact of living.

And when those failure happen, your first thought as a QA engineer might be to Slack the developers, say “ test failed—try again thx! ” and call it a day. After all, the principal role of the QA team is to make sure that the package is ready to ship. Fixing problems that are discovered during testing is usually developers ’ responsibleness.

However, the fact is that the QA team needs to do more than merely record failures and alert developers to them. Whenever a examination fails, QA engineers should occupy the extra footstep of performing test failure analysis. Doing so not only help the QA team provide insights to developers that might allow them to resolve an topic faster, but can too make testing operations smoother at the like time.

What is test failure analysis?

Test failure analysis is what it sounds like: it ’ s the process of analyzing a failed test to figure out what went wrong.

The exact nature of your test failure analysis process should be orient to your motivation, but typically, the analysis should allow you to answer the following query:

Application vs. tryout failure

Did the test fail because of a problem with the software that you were testing, or because of a problem with your test? This is the first and most canonical question that you need to answer; after all, before you go telling the developers that their code has a bug, you should make certain that the job was not caused by a test that you wrote.

Root cause

Regardless of whether the trouble lies with the software or with your test, you need to cognize the root cause of the issue. The ways in which a test failure manifests itself on the surface may or may not reflect the beginning cause of the job. For model, perhaps your test reveals that a menu in your application fails to load on a certain browser. There are multiple potential source causes of this issue -- a corrupted CSS file, a permissions trouble with the file, or a bug in the browser (which your developers will ask to work around), to call exactly a few possibilities. You need to figure out what the nucleus effort of the subject is to assist developers direct it quickly.

Failure scope

If the failure was caused by a problem with your application ’ s code, how many builds or configurations are touch by the failure? If you have run tests on every build and configuration, then this will be an easy interrogation to respond. But if -- like most QA teams -- you don ’ t have the resources to test every single potential environment, your test failure analysis should include an assessment of how many environments are likely to experience the same problem.

For autonomous testing across multiple user personas, check out SUSATest — it explores your app like 10 different real users.

Failure significance

A final question that your test failure analysis should be capable to answer is: how significant is the failure? If it was caused by a trouble with the application, is it significant plenty that you need to delay deployment until it is fixed? Or is it a relatively minor topic that doesn ’ t warrant canceling a unhurt deployment?

If, alternatively, the trouble dwell with your test, you should measure how seriously the issue affects your examine pipeline, and whether you need to drop everything and reference it ASAP or if you can live with it for a little while.

Getting the most out of test failure analysis

You might be thinking: “ Test failure analysis sounds great, but who has the time to run a elaborated analysis of every failed test? Not me! ”

Fair enough. You shouldn ’ t be doing a tryout failure analysis every single time a examination betray. Instead, consider the following strategies, which can help you name which failures take a total analysis and which you can snub.

Parallel testing

Parallel testing offers many benefits, including the ability to assess the probable significance of a exam failure quick. This is because, when you run tests in parallel, you ’ ll know as soon as they are complete how many tests for a given build are failing.

If you run respective dozen automate tests in parallel and only one or two fail, there ’ s a becoming chance that the trouble is comparatively minor. That doesn ’ t mean you should inevitably ignore it, but it make afford you a sense of how to approach the issue before you still get any kind of analysis.

If, on the other hand, a significant component (say, more than 20 pct) of your tests fail at the same clip during a parallel run, you know that you either have a problematic bug on your manus that will probably require you to cancel deployment, or a widespread job in your tests that you involve to address straightaway before you do any more testing.

Auto-restarts

Despite theold clichéthat the definition of insanity is repeating the same thing and require different results, the fact is that package is a fickle thing. Sometimes, extend the like tryout twice will yield different results. Maybe a net connectedness failed temporarily on the first run, or a server crashed on your test grid. Weird and unpredictable things happen from time to time, even in package testing environments where we endeavor for consistency.

For that reason, configure your test to resume mechanically when a failure come is one way to reduce the number of potential failure that you need to analyze. By auto-restarting test, you can be sure that a test is truly failing and requires you to yield it a deeper look.

Failure playbook

As virtually anyone who works in QA knows, eubstance is the mother (or one of them, at least) of calibre. That rule certainly holds true when it arrive to test failure analysis. You want a consistent, predictable summons for appraise and respond to failure.

That ’ s why you should develop a failure analysis “ playbook ” that lays out the process you will postdate for determining whether a failure warrants analysis, as well as how you ’ ll execute the analysis if it does. Your playbook should besides specify what you ’ ll do after analysis is complete by identify the process for things like contacting developers and determining whether to retest.

Conclusion

Your testswillfail, at least sometimes. The way you react to the failures plays a pivotal role in shaping the effectivity of your overall testing strategy. Instead of simply sending failed code back to developers and expecting them to handle it, you should have a consistent plan in place for analyzing exam failures and reacting to them.

Sauce Labs has made a number of changes to our Insights platform to help our user become better tester, with the well-nigh significant change be the addition of a new Failure Analysis feature. Failure Analysis uses machine learning to work with pass/fail data and Selenium require logs to unveil and aggregate common failures, and provides insights into how they are affecting the test suite as a unit. This leads to improved productiveness, as developers and QA can prioritize fixing the most pervasive problems to ensure that freeing speed doesn ’ t stall. To learn more, ensure outthis little demo.

Chris Tozzi has worked as a journalist and Linux scheme administrator. He has exceptional interest in open source, agile infrastructure and networking. He is Older Editor of content and a DevOps Analyst at Fixate IO. His latest record,For Fun and Profit: A History of the Free and Open Source Software Revolution, was published in 2017.