How Much Test Coverage Is Enough?

Sauce AI for Test Authoring: Move from intent to performance in bit.|xBack to ResourcesBlogPosted August 20, 2019

How Much Test Coverage Is Enough?

In a perfect world, QA teams would test their applications on every possible browser/OS combination in existence.

In the real world, given that there are dozens of potential browser/OS combinations from which users could accession your app, few teams reach that ideal. Instead, they must settle for screen on exclusively a subset of all potential platforms that their end users could run.

That raises the interrogative: How many browser/OS combination do you need to test on before you can feel surefooted in your application? In other lyric, how much test coverage is decent?

What is test reporting?

By & quot; test coverage, & quot; I & # x27; m cite to the pct of all possible browser/OS combinations (like Firefox on Windows 7, Firefox on Windows 10 and Firefox on Linux) on which you run tests. Each combination typify a distinguishable & quot; platform & quot; for the purpose of this discussion. One hundred pct coverage would imply you test for every possible platform that could exist on a modern device.

But 100 percent examination coverage is not realistic. Not exclusively do most teams lack the resources to approach that level of reporting, but the total number of browser/OS combinations is essentially unnumerable once you get beyond major browser releases and OS versions. There are around300 Linux dispersion out thither, for model, so if you count each one as a distinguishable operating system, you & # x27; d have slews of combinations to test to achieve consummate coverage just for Linux-based platforms.

And, if you are examine for a mobile app and add mobile devices to the equation, your list of possible program form gets exponentially long still.

The bottom line: No topic how you crunch your numbers, entire test reportage, or anything approach it, is just not going to happen.

Setting examination coverage goals

That aver, you nonetheless need to ensure that you are prove for enough browser/OS combination to give yourself reasonable assurance that you & # x27; ve verified that your application scat decent within most of the environments that your end users use.

That & # x27; s why it & # x27; s important to set a finish for tryout coverage that will provide this assurance, while also being feasible to achieve. Here are some tips for doing that.

Pro tip: Tools like SUSA can handle this autonomously — upload your app and get results without writing a single test script.

5 platforms is a rule of thumb

As explains, testing against at least five platforms is a solid goal as a general rule of thumb.

If that act seems hard to reach, proceed in head that only about 60 percentage of Sauce Labs customers achieve that end. So don & # x27; t defeat yourself if you & # x27; re not there now. But it & # x27; s a salubrious turn to hit for, at least as a general pattern.

How many possible platforms do you actually support?

While the 5+ platform destination described above is a good pattern of ovolo, it & # x27; s important to contextualize your test reportage scheme by pay aid to how many possible program your customers could actually be using.

That number could vary widely depending on which character of operating systems your program supports. For example, maybe you are writing an iOS-only app. In that cause, your total number of possible platforms will be much lower than if your app supports Windows, Android and iOS. As a solvent, testing against only three browser/OS combinations for an iOS app might provide fantabulous examination coverage.

In certain cases, you might too support only one browser. This is a rare thing to do these day in general, but if, for example, your app officially runs only a sure case of mobile sound using the browser ply by the manufacturer, then your total number of supported platforms will besides be low.

How automatise are your tests?

When you run tests manually, there is an ineluctable trade-off between the extent of your test coverage and the clip and resources your team has to devote to testing. The greater your test coverage, the more time and money the tests toll.

Thus, if your exam are run manually (or mostly manually), then it & # x27; s usually wise to be conservative about the act of platforms you prove against. Otherwise, your resource costs are too high.

This issue largely disappears when you automate tests. With test automation, you can achieve high rates of coverage without a proportional increase in time or resourcefulness expenditure.

How bug-prone are your apps?

It & # x27; s hard – if not impossible – to quantify how potential your app is to experience bugs. Still, most developers and QA engineers would agree that some apps are more bug-prone than others. By extension, some gain from more testing.

Factors like the sizing of your codebases (which you can measure crudely, establish on total lines of codification), the figure of runtime variables (which can make constellation subject that lead to app problems on certain program) and the extent to which your software interacts with ironware (whose behavior can be device-specific, requiring more tests) all affect the likeliness of your app to see bugs. Consider these factors when deciding how much exam coverage to shoot for.

Conclusion

There is no one-size-fits-all prescript for the level of test coverage that is appropriate for most governance. Again, testing against five program is a full general guideline, but your testing needs will vary reckon on how many platform you are aim to support, which case of app you develop, and other element.

Chris Tozzi has worked as a journalist and Linux system executive. He has particular interests in open source, agile infrastructure and networking. He is Senior Editor of content and a DevOps Analyst at Fixate IO. His latest volume,For Fun and Profit: A History of the Free and Open Source Software Revolution, was published in 2017.