As software essay get a, modern test automation program are expand their range of functionalities so that lineament technology team can do more within shorter development cycles. These new capabilities are becoming the cardinal distinction between script-based examine model and automated screen platforms; the onetime simply help QE teams execute more tests, the latter transforms testing results into data that enable QE to take on a leaders role in DevOps and continuously improve the customer experience. & nbsp;

Mabl is designed to help all quality professionals - regardless of coding experience - maximize the potential of package testing. Though we ’ re always focused on being the easiest solvent for test mechanisation, we ’ re too driven by the examiner ’ s motivation for better perceptiveness and better reportage. Today, we ’ re diving into a late mabl-on-mabl scenario that prove how test data from mabl can be used for trouble-shoot trial environments. & nbsp; & nbsp;

Failed Tests Create Questions

Recently, the mabl technology squad noticed increased failure rates in our unit and end-to-end tryout. We habituate mabl and other information sources extensively to diagnose and verify the fix for the topic. The process we used to investigate and conclude the issue can be replicated by other mabl users, and we hope that sharing our experience facilitate other teams identify and resolve environs issues in the future. & nbsp;

We firstly identified the issue in ourGitHub& nbsp; Actions CI pipeline, where some flesh occupation would fail and then pass upon retry. Most of the failure were due to test failures and timeouts, & nbsp; yet our build logs didn ’ t present a pattern in the failures, which occurred on different tests and different stairs each run.

To confirm that this was indeed a systemic issue, we ensure the prove fascia for our QA environment, which spotlight the increased failure rate in that environment.

Failure rate metric from Google DataStudio via

At this point, we knew that we had a job worth enquire. We didn ’ t see a alike course in our production environment and tests be passing consistently against our local builds, so our hypothesis was that the issue was related to our QA environment. & nbsp;

Narrowing Down Potential Causes & nbsp;

To test that conjecture, we lumber into mabl and visit the test results for our primary smoke testing plan that runs against our QA environs. The first failure was a login issue, which was unknown because the trial had legislate in both anterior and subsequent runs. & nbsp;

Screen shooting beguile by mabl during test run

The future failure looked like a timeout: the page was loading for over one minute, result in a failure in the test assertion. & nbsp;

Screen shot captured by mabl during test run

The apparent timeout was a red herring since we started to inquire possible latency issues and found no evidence to indorse general slowness. This was verify by our average page load time, which had be flat (or still decreasing) in most late tests. & nbsp;

SUSA automates exploratory testing with persona-driven behavior, catching bugs that scripted automation misses.

App speed index enamour for each test step and aggregated by mabl for each test

Likewise, the overall execution clip for tests that passed appeared to be getting faster, not dumb.

Connecting the Dots

Test run clip from Google DataStudio via mabl BigQuery Export

Digging in on the timeout failure above, we comment that the previous “ Save ” step included a warning. Reviewing that measure, we saw that mabl had observe a 502 “ Bad Gateway ” response to an API cry from the browser. Given the intermittent nature of the failures, we suspected that something was stimulate periodical connectivity issues, rather than an actual network configuration issue.

Network reaction logs charm by mabl during test run

We also checked request logs from the QA environment, and we could clearly see a significant increase in error (501-599) over the preceding two months. But we didn ’ t see a corresponding increase in production.

Aggregated request log info from Google BigQuery

Reviewing our test results, we find that the failure pace appear to be high for runs that were triggered by a deployment (via API) as compared to tests that were run sporadically. Given that we had recently increased the number of tests configured to run on deployment, we acquire a hypothesis that the QA environs was struggling to handle the increased load generated by hundreds of parallel test runs. & nbsp; & nbsp;

We then critique the configuration of our QA environment in our cloud console. We note that we be habituate smaller instances to power our QA API equate to & nbsp; our product API and that the instances were scaling up rapidly with each deployment. The picture below exhibit that the QA API (in common, at bottom) was typically powered by three instances and scaled up to 6 (the uttermost per the configuration) during each deployment but 4-5 during periodic (non-deployment) plan runs. & nbsp;

Instance counting over time for QA and Dev APIs from cloud provider console

Implementing a Fix

We modify the QA instance types to match production, increased the maximum instance count, and triggered additional deployment runs. Those deployment runs were successful, confirming our hypothesis that the additional testing load was exhausting the resources allocated to the QA environment. Since we get the modification, the failure rate has been consistently under 2 percent.

Failure rate metric from Google DataStudio via mabl BigQuery Export

We hope this post sheds light on how test data in mabl can be used to investigate and resolve issues in the testing environment. Much of this information is usable at your fingertips in the mabl user interface and some can yet be easy accessed via our. & nbsp;

Thinking Outside the Pass/Fail Box

Test data is a valuable resource for place issues in both the product itself as well as the software development pipeline. Unfortunately, few test mechanisation solutions are designed to help quality teams harness that data for troubleshooting and broader improvements. When a try resolution snub the potentiality of quality engineering and package testers, quality squad are limited in what they can contribute to improving customer satisfaction, establish a best software product, and overall DevOps adoption. As this mabl-on-mabl case demonstrates, tryout data can be tackle to help QA teams troubleshoot environs faster, which helps them quickly resolve issues and get back to developing new try strategies, working with the balance of the development squad, and result the client experience. Test automation platforms need to recognize this opportunity and assist their end-users maximize the worthful insights provided by test data. & nbsp;

If your team is interested in the potential of test data for your organization, register formabl 's 14-day free trial. You ’ ll have full access to the mabl test automation platform, our award-winning support team, and our across-the-board library of support documentation. & nbsp;