One of our end hither at mabl is to make test mechanisation easygoing for everyone in the development organization, includingmanual quizzer, QA engineers, developers, and production owners. In tardy 2022, we introducedperformance testingto our platform ’ scross-browser and API essaycapabilities. We conducted customer interviews, refined essential, germinate design, implemented, and tested the early access release of the merchandise. As more teams adopt low-code API performance testing with mabl, we wanted to share a few lessons we learned while germinate this exciting characteristic.

From Functional Testing to Load Testing & nbsp;

Prior to this effort, we offered functional API testing. Here is the high-level architecture of that result:

API Functional Testing Architecture

When our scheduling scheme mold that an API trial should be executed, it publishes a message to a Pub/Sub topic containing all of the info required to execute the test. We then invoke a Cloud Function that execute the test, pen the yield to Cloud Storage, and calls rearward to our API with results. This architecture has be working reliably for functional API tests since it was introduced. For load testing, our initial design was to simply extend that architecture to support parallel executions.

Adapting the Functional API Test Architecture to Load Testing & nbsp;

A few changes would necessitate to be made to accommodate the functional API testing architecture for load testing. The virtually significant change from an architecture perspective was the gain of a new component telephone theOrchestratorwhich would be responsible for managing the Cloud Function representative associated with a load test. This part would depart up the required number of Cloud Functions via Pub/Sub trigger and replace them in the event of a crash or if the test needed to run long than the maximum Cloud Function runtime bound.

Our Initial Load Testing Architecture & nbsp;

Taking the above changes into chronicle, we began enforce and testing the limited functional examination architecture:

Initial cargo testing architecture

Verifying Load Test Performance

Once we had implemented the initial load testing architecture, we needed to control that it was performing adequately. The first job we had when trying to control the performance was finding a worthy target for the load. There are several sites contrive to attend in writing functional API tests, such aspostman-echo.com or httpbin.org, but these sites are not intended to be used for freight testing.

We were able to quickly build and deploy a shipment prove target service based onnginxand GKE. & nbsp; By leveraging nginx 's support for Lua scripting, we could pass in different parameters on the request query string in order to have the server return specified error codes with certain probabilities or induce randomised latency.

Challenge 1: Minimizing Latency and Variance

As we began to ramp up our testing to larger figure of load generators, we noticed that the metric we be measuring were not in-line with our expectations. Latencies were high than require, throughput was correspondingly lower, and several metric seemed to have a higher than require variant. Below is an example of one of these former test runs:

Request latencies be waver by C of msec and averaging over 500ms per request.

For autonomous testing across multiple user personas, check out SUSATest — it explores your app like 10 different real users.

We spent the next twosome of weeks investigating agency to meliorate performance. & nbsp; We tested different variables by changing the node libraries, changing which cloud services be used to generate the load, and eliminating as many network hops as possible. First we tried generating load expend the same client code but running in different execution environments: local non-virtualized ironware, plain GCE VMs, and GKE.

We did n't observe performance issues when running in these alternative environment, which led us to suspect that the Cloud Functions execution environment was a conducive factor. & nbsp; To confirm, we then generated payload in Cloud Functions using a different client library written in a different lyric, and we observed like performance anomalies. Although we were not able to determine the actual source of those performance anomalies, we suspect that several divisor such as shared infrastructure and the underlying instance character contributed to this behavior.

Next, we had to decide what to do with these findings. Although the performance was not what we expected, changing a key constituent of the architecture was a significant risk to the timeline. & nbsp; We researched alternatives like Cloud Run and GKE, and we estimated that migrating to GKE would not be too much effort since we were already employ it for the other components. & nbsp; We ultimately decided that achieving high performance was so critical to the success of the product that it was deserving the extra time investment.

Migrating from Cloud Functions to GKE was a fairly easy process. We simply had to define a new node pool for load generators and box the code in an image so that it could be deployed as a KubernetesStateful Set. The entire operation took roughly one week.

Running the load source on GKE give much lower latency and discrepancy:

Load trial with load source executing on GKE

After get these changes we be last seeing consistent performance for the bulk of the test duration. However, there was one anomaly that was yet bothering us.

Challenge 2: Achieving Consistent Throughput & nbsp;

After resolving the main execution problems with the load tryout base we noticed one other strange anomaly that would consistently seem near the beginning of the test:

Abbreviated throughput dip near the offset of load exam signal a problem…somewhere

We would much see throughput driblet lasting less than a min that would occur shortly after the starting of the test. Using the same debugging techniques we had used antecedently, we place one lay whose value would affect the throughput drops: connection reuse.

When sending many HTTP petition, recycle the same TCP connection can significantly improve performance by avoiding the TCP handshake overhead each time. However, we had originally wanted to avoid connection reuse in order to more accurately simulate manyuniqueusers accessing the quarry system simultaneously. Unfortunately, we base that the high number of unequalled connections was stimulate performance issues when unite with Cloud NAT. Our suspicion was that Cloud NAT had to horizontally scale up after a few minutes of sustained shipment with many alone connections, and that this scaling operation result in a temporary performance degradation.

After enable TCP reuse and performing some early Cloud NAT constellation tune to better handle large numbers of connections, we were able to execute a 3,000-user test with smooth, consistent execution:

Smooth ramp-up and consistent execution with 3,000 load generators

Our Final Load Testing Architecture & nbsp;

After making the changes line above, we terminate up with a simpler architecture relying on few services and achieving best performance at lower cost:

Final load testing architecture with GKE-based consignment generators

At mabl we use an, reiterative development methodology. Early in the development form of the API execution try product we prefer result that would allow us to get the functionality into customers ' hands sooner in order to obtain feedback and validate the nucleus lineament. Later on, we revisited some of those earlier decisions and made a few modification that we thought were necessary for the product to gain its total potential. The end outcome was a product that provided value to our Friends of mabl customer community and foster a signified of pride across the mabl team. & nbsp;

See mabl 's API Testing Capabilities in Action & nbsp;

Join Google Cloud (Apigee), GitLab, mabl, and 66degrees on Thursday, October 19th to learn how to up-level your API strategy through:

• API management best practices
• Integrating APIs into your CI/CD pipelines
• API testing

Register for costless.