What is mobile app performance testing, exactly? What should we like about in terms of app execution and how can we test app performance as constituent of our automated test cortege? This is a topic I ’ ve been thinking a lot about recently, and so I convinced Brien Colwell (CTO of HeadSpin) to sit down with me so I could pester him with some questions. I took our conversation and turned it into a variety of FAQ of. None of this is specifically tied to Appium – stop tuned for future edition where we seem at how to achieve some of the goals discussed in this clause, utilise Appium by itself or in conjunction with other tools and services.

What is performance testing all about?

Really, we could think of performance examination as a big part of a panoptic concept: UX testing (or). The thought hither is that a user ’ s experience of your app goes beyond the input/output functionality of your app. It depart beyond a lot of the things we normally associate with Appium (though of course it include all those things too – an app that doesn ’ t work does not provide a good experience!) It reverberate the state of the app grocery, where the world is so crowded with nearly-identical apps, that small improvements in UX can mean the difference between life and death for one of these startups.

Years ago, I consideredexecution testingto be exclusively in the domain of metric like CPU or memory utilization. You solely did performance testing when you wanted to be sure your app had no memory leaks, that kind of thing. And this is withal an important part of performance testing. But more generally, performance testing now focuses on attributes or behaviors of your application as they interface with psychological facts about the exploiter, like how long they are prepared to wait for a view to load before moving on to another app. One quick way of summarizing all of this is to delimit performance testing as ensuring that your app is responsive to the directions of the user, across all property.

What performance metrics should we like most about?

It ’ s true that classical specters of pitiable package performance, like memory wetting or spinning CPUs, can plague mobile app experience. And there is full reason to measure and profile these metrics. However, the primary campaign of a bad user experience these days run to be network-related. So much of the wandering app experience is dominated by the requirement of loading data over the network, that any inefficiency there can cause the user to have sorely frustrating delays. Also, quiz metrics like memory or CPU usage can oftentimes be adequately accomplished topically during development, whereas network prosody need to be screen in a variety of conditions in the battleground.

Read:

To this end, we might track metric like the pursual:

• DNS resolution (did the name resolve to an IP that was close by?)
• Time till 1st byte incur for a given response
• Time delay between send / acknowledge in the TCP watercourse
• TLS handshake time

SUSA automates exploratory testing with persona-driven behavior, catching bugs that scripted automation misses.

Beyond web metrics, there are a number of early UX metrics to reckon:

• Time-to-interactivity (TTI): how long does it take for an app to become usable after the user has launched it? (This is an extremely important UX metric)
• Full time a blank screen is prove during a user session
• Entire clip loading animations / spinner are shown during a exploiter session
• Video quality ()
• Time to full burden (for progressively-loaded apps)

Also read:

What are some mutual mistakes mobile app developers make that Pb to pitiful performance?

When developing an coating, we are often doing so on top-of-the-line desktop or laptop figurer and devices, with fast corporate internet. The execution we get during development may be so good that it masks topic see by the average set of users. Here are a few mutual fault (again, largely network-oriented) that developer create which can radically impact performance:

• Using HTTP/2 or a impost TCP communication groove instead of sticking with the simplicity of HTTP v1 and optimizing traffic. (I.e., there aren ’ t as many gains as you might expect from a more modern or complicated meshing hatful)
• Having dependent/blocking network requests that are executed serially; whenever possible, network requests should be executed in parallel.
• Having too many mesh asking on a individual panorama. This can be a import of a microservices / micro API architecture where to get the appropriate data, mickle of requests are required. For roving it is essential to make as few requests as possible (ideally only 1 per view), and therefore to sacrifice purity / composability of API answer in order to aggregate all the data necessary to build a given view.
• Misconfigured DNS causing devices to talk to waiter that are unduly far away.
• Unoptimized images, video, or audio being sent to regions or web that can ’ t cover the bandwidth.

Recommended Post:

How should we think about absolute vs comparative execution measurements?

In general, it can often be more useful to track performance relative to a certain baseline, whether that is an accepted touchstone baseline, or just the initiative point at which you started measuring performance of your app. However, tracking relative performance can also be a challenge when testing across a range of devices or networks, because relative measures might not be comparing apples to apple. In these cases, looking at absolute values side-by-side can be rather useful as easily.

Are there any absolute touchstone performance targets mostly discern as helpful?

It ’ s true that each team defines UX for their own app. Acceptable TTI measures for an e-commerce app might dissent by an order of magnitude or more from acceptable measures for AAA punt rubric. Still, there are some helpful rules of thumb base on HCI (Human-Computer Interaction) inquiry:

• Any delay over 500ms becomes a “ cognitive ” event, meaning the user is aware of time having passed.
• Any delay over 3s becomes a “ reflective ” event, meaning the user has time to reverberate on the fact of time having pass. They can become distrait or choose to go do something else.

These are not hard-and-fast truths that make sense in every case. And of line, nobody actually has a universal resolution, but again, it ’ s helpful to treat that 500ms number as a good target for any interaction we require to feel “ whipping ”.

(To follow up on some of this research, say up on thehuman processor model or powers of 10 in UX)

How widely should we require performance to vary across different devices?

In other language, when should we be really concerned about departure in performance between different devices or web? Actually, it ’ s fairly common to see dispute of about 30 % as quite common between devices. This level of difference doesn ’ t unremarkably indicate a wicked performance number, and can (with all appropriate caveats) be regard as division.

True performance problems can cause departure of 10-100x the baseline measurement – just think how long you ’ ve waited for some app view to charge when they are downloading too much content over a slow network!

How do you decide which devices to focus on for execution testing?

The answer hither is simple if not practical: that bring in the greatest revenue! Obviously this implies that you have some kind of understanding of your userbase: where are they located? What devices do they use? What is their typical network hurrying? And so on. If you don ’ t have this information, try to begin tail it so you can cross-reference with whatever sales metrics are significant for your product (point purchased, time spent in app, whatever).

At that point, if you can cull the top 5 device that meet these standard in a yield region, you ’ re well positioned to secure a potent UX.

Also assay:

Conclusion

“ Performance ” turns out to be quite a broad subcategory of UX, and of course, what we care about at the end of the day is UX, in a holistic way. The more elements of the UX we can begin to measure, the more we will be able to read the impact of changes in our application. We ’ ll even finally get to the point where we ’ ve name solid app-specific metrical mark, and can fail chassis that don ’ t meet these prey, guaranteeing a minimum high level of UX quality for our users. Oh, and our exploiter? They won ’ t know any of this is happening, but they ’ llloveyou for it.

4 Parts

Connet Now