So far in my series, we ’ ve looked at mobile execution testing in general, as well as. Now it ’ s time to discusspaidtools. Obviously, Appium is free and exposed source, and it ’ s nice to focus on and advertize complimentary tools. But at the end of the day, sometimes pay tools and services can be valuable, particularly in a business setting where the budget is less significant than the end result. I want to make sure you have some decent and reliable information about the unscathed ecosystem, which includes paid tools, so will from time to time let you know my idea on this as easily!

Disclaimers: This is not intend to be totally comprehensive, and yet for the service I cover, I will not undertake to give a full overview of all their features. The point here is to exercise down specifically into performance testing choice. Some service might not be really consummate when it comes to their execution testing offer, but that doesn ’ t mean they don ’ t have their potent point in former areas. For each tool, I ’ ll likewise note whether I ’ ve personally used it (and am therefore speak from experience) or whether I ’ m merely using their publicly-available marketing info or medico as research. Finally, I ’ ll note for transparence that HeadSpin is an Appium Pro sponsor, and in the retiring I worked for Sauce Labs.

Most interesting for us as Appium users will be service that allow the collection of performance data at the like clip as you run your Appium tests. This is in my opinion the ‘ holy Sangraal ’ of execution testing–when you don ’ t experience to do anything in particular to get your performance test data! In my research I found four companies that in one way or another touch on some of the topics we ’ ve discussed as part of their Appium-related products.

HeadSpin

HeadSpin is the most performance-focused of all the service supplier I ’ ve seen. Indeed, they began life with performance testing as their lonesome commission, so this is not surprising. Along the way, they added support for Appium and have been heavy contributors to Appium itself in late years. HeadSpin supports in various geographic locations around the cosmos, connected to a multifariousness of networks.

When you run any Appium test on HeadSpin, you get access to a report that looks something like the one hither. Basically, HeadSpin tag a ton of different metrics in what they call a. The linked doc highlights some of the metric captured:

mesh traffic (in HAR and PCAP format), device metrics including CPU, retention, battery, signal strength, screen rotation, frame rate, and others, device logs, client-side method profiling, and contextual info such as Appium require issued during the test, what app was in the foreground, and other contextual metadata.

On top of this, they try to show warnings or do suggestions based on industry-wide quarry for various prosody. And this is all without you have to instrumentate your app or get specific bidding to retrieve specific metrics. HeadSpin covers the most metrics out of any product or company I ’ m familiar with, including really difficult-to-measure quantities like the of a video as a proxy for video caliber.

One former feature utile for performance testing is what they call the ‘ HeadSpin span ’, which allows bridging remote devices to your local machine via their hs CLI tool. Concretely, this gives you the power to run adb on the remote device as if it were local, so any Android-related performance testing tools that work on a local device will also work on their cloud devices, for example theCPU Profiler. The bridge works likewise for iOS, allowing step debugging orInstruments profilingon remote devices, as if they were local.

In terms of network conditioning, HeadSpin offers a number of options that can be set on a per-test cornerstone, including mesh shaping, DNS spoofing, header shot (which is super utile for tracing network calls all the way through backend infrastructure), etc… This is in increase to requesting devices that exist in sure geographic regions.

Finally, it ’ s worth point out that HeadSpin tries to do it easy for teams to use the execution data captured on their service by spin up a customs “ Replica DB ” for each squad, where performance data is mechanically publish. This “ unfastened data ” approach means it ’ s easy to build custom dashboards, etc…, on top of the data without having to pre-process it or host it yourself.

Sauce Labs

Historically, Sauce Labs was focus principally on cross-browser web testing, and so it & # x27; s no surprise they have some performance-related info usable about their web browser sessions. That & # x27; s not what we & # x27; re considering today, notwithstanding, but rather Sauce & # x27; s roving offer. They experience a practical device cloud of iOS simulators and emulators, but as we mentioned before, performance testing of virtual devices is not peculiarly meaningful.

Sauce & # x27; s real gimmick cloud supports a feature called Device Vitals which provides a broad set of performance metrics, including CPU, memory, and performance information. It works with live/manual examination, but is also supported via their Appium interface when a special capability recordDeviceVitals is set. At this point of the product it seem the only output for is a CSV file which can be downloaded, meaning the user would be creditworthy for extracting and displaying the performance data itself. However, the live/manual usage shows a graph of performance information in real time:

Sauce also supports a practical device bridge, which they call Practical USB. It is generally available for Android and in beta for iOS, and works more or less the same as HeadSpin & # x27; s bridge lineament.

In my experience, and as far as I can tell, Sauce execute not give exploiter the power to fix web conditions for twist connections, so test app performance in a smorgasbord of lab conditions is not potential. However, it is potential to direct Sauce devices to use a proxy connexion, including something like Charles Proxy (see below).

AWS Device Farm

AWS has its own Appium cloud, which works on a bit of a different framework than HeadSpin or Sauce. With AWS, you compile your whole testsuite into a bundle and send it off to Device Farm, which then unpack it and uses it to run tests locally. Theoretically, this yield you a lot of flexibility about how you can connect to a device during the test, and it & # x27; s likely potential (though I haven & # x27; t tested this) to run some of your own execution testing diagnostics on the device under test, even as your Appium book is going through its paces.

Luckily, you may not demand to care about this because Device Farm does get some performance information as your test is executing, namely CPU, memory usage, and thread count over clip:

Device Farm does allow you to sham specific network conditions by creating a & # x27; network profile & # x27; (that acts basically the like as the profiles we discourse in the clause on simulating mesh weather for virtual devices.

Experitest

Information retrieved from companionship website

Experitest is a roving testing company with Appium support, who also publicize performance datum collection. They say they support different network conditions and geographical locations for their devices, as well as execution metrics like:

Pro tip: Tools like SUSA can handle this autonomously — upload your app and get results without writing a single test script.

• Transaction duration
• Speed Index
• CPU consumption
• Memory Consumption
• Battery Consumption
• Network data (upload and download)

I am not entirely certain what a & # x27; transaction & # x27; is, but it appears to be a user-definable span of time or sequence of app events.

I could not find more specific information in the docs about how this datum is captured, and whether it is done so automatically as a portion of an Appium test or whether it is a freestanding feature/interface. The better I can figure out, during trial recording the users define start/stop points for execution & # x27; transaction & # x27; transcription, though this does not explain whether it is possible with Appium tests that you have brought yourself and did not develop in Experitest & # x27; s & # x27; Appium Studio & # x27; product.

Like AWS Device Farm, Experitest apparently has the ability to manually create network & # x27; profiles & # x27; that can be applied to certain tests or projects.

Perfecto Mobile

Information find from companionship site

Perfecto Mobile is probably the oldest fellowship in the entire space, and they & # x27; ve had a lot of time to build out their support for performance testing. It looks like Perfect focuses on network condition simulation via their concept of Personas, which are essentially the same types of network profile we have seen in other places.

You can set the Persona you require your Appium exam to use via a special desired potentiality. Given my perusing of the DoC, it seem potential to set & quot; points of interest & quot; in your trial, that you can use for confirmation after on. It wasn & # x27; t clear, but it & # x27; s possible that setting one of these & quot; points of involvement & quot; might capture some specific execution data at that time for afterwards review, however I couldn & # x27; t find any specific mention of the types of performance metrics that Perfecto captures, if any.

In increase, I & # x27; m not quite sure what & quot; Appium & quot; mean exactly for Perfecto. From their Appium landing page, we see this substance:

Appium testing has limitations that may prevent it from be genuinely enterprise ready. Perfecto has re-implemented the Appium server so it can deliver stronger automated examination controls, mimic real user environs, and support the unparalleled motivation of today ’ s Global 1000 companies.

What this mean is that, if you & # x27; re using Perfecto, you & # x27; re not actually running the Appium open source project code. Whether that is a good or bad thing is up to you to determine, however I can alone imagine that it could yield confusion as their internal reimplementation inevitably drifts from what the Appium community is doing! How easy is it to migrate Appium tests running on Perfecto to another cloud? If they & # x27; re not employ standard open source Appium servers, this could be a difficult interrogation to answer. I & # x27; m a big fan of what Perfecto is trying to do with their product, but (as is probably not surprising given my office with the Appium projection) I don & # x27; t think it & # x27; s good for any cloud provider to & quot; reimplement & quot; Appium.

Other Products

Cloud services are not the alone types of paid products used for mobile performance try. We also have SDKs (which are construct into the app under exam), and standalone software utilize for various purposes. None of these have any particular relationship with or support with Appium, so you & # x27; re responsible for figuring out how to make them work in conjunction with your Appium tryout.

Charles Proxy

Information recover from company website

Charles Proxy is a networking debugging proxy that has special support for mobile devices. You can use it to inspect or adjust network traffic coming into or out of your devices. Charles comes with special VPN profiles you can instal on your devices to get around issues with trusting Charles & # x27; s SSL certificates, etc ...

Charles & # x27; s main features are: SSL proxying, bandwidth throttling (network conditioning), request replay, and request substantiation.

Firebase

Information regain from company site

Google & # x27; s Firebase is alotof things, apparently specify to be a sort of one-stop-shop especially for Android developers. It & # x27; s got databases and auth services, analytics and distribution tools. Firebase doesn & # x27; t have any officially supported Appium product, though they do support automatise functional testing of Android apps (via Espresso and UiAutomator 2) and iOS apps (via XCTest) through their Test Lab product.

Of more interest for us today is their performance monitoring merchandise. The way this works is by bunch in a particular SDK with your app which enables in-depth execution tracing viewable in the Firebase console. You start and stop traces using code that leverage the SDK in exam version of your app, and that & # x27; s it. Of course, this means modifying your app under test, and I couldn & # x27; t find any discussion of whether performance or network data is charm & quot; from the outside & quot; during tryout runs in the Firebase Test Lab.

New Relic

Information retrieved from company website

Another SDK-based option is New Relic Mobile. Embedding their SDK in your app enables tracking of a host of network and other execution data. Where New Relic appears to have an advantage is in exhibit the connector your app has with other New Relic-monitored service (backend sites, for example).

Otherwise, New Relic provides most of the same types of performance data you & # x27; d expect. They too give you a & quot; User Interaction Summary & quot; which appears to be similar to HeadSpin & # x27; s high level detail of possible UX topic. In improver, metrics can be catch in aggregated across twist eccentric, so it & # x27; s possible to easily spot performance course on the level of device type as good.

Since this is an SDK-based answer, you & # x27; ll have to alter your app under tryout to gather the data. Presumably you could then use it in your Appium test just fine!

Conclusion

In this article we examined a turn of cloud services and early type of paid software habituate for performance testing. A small smattering of these companies supports Appium as a first-class citizen, meaning you can use their cloud to run your Appium tests themselves, and potentially get a lot of useful execution information for free along with it.

In my mind, this is the best possible access for Appium users -- just write your functional tryout the way a user would walk through your app, then run the tests on one of these platform, and you & # x27; ll get execution reports for free! (Or rather, for hopefully noadditionalfee). The best kind of performance test is manifestly the one you didn & # x27; t have to write explicitly yourself, but which withal institute useful information that can easy be acted on in order to improve the overall UX of your app.

Oh, and did I miss a good candidate for this clause? It & # x27; s entirely possible I did! This is a big and growing industry. Let me know about it and I & # x27; ll update this list if I find a relevant product.

FAQs

Q: What are some performance screen KPIs?

Performance testing identifies an application & # x27; s hurrying, reliability, and robustness, so the KPIs should reflect these features. For example: -

Speed KPIs: Load multiplication, response time, Hits/second.

Reliability KPIs: Number of concurrent exploiter, bit of concurrent asking.

Performance KPIs should besides be linked to your application & # x27; s line goals.

Q: What is spike testing, and why is it important?

Spike testing determines an app & # x27; s ability to handle the sudden growth in load. Spike testing aid understand the peak usage of the application and its ability to battle DoS and DDoS attacks.

Q: What is package performance benchmarking?

Performance benchmarking examination an application & # x27; s performance against the industry measure or against similar applications. Performance benchmark create sure that the coating & # x27; s hurrying, reliability, robustness, or scalability either lucifer or exceed the accepted industry standard or user essential.

Q: What are spread lashings in execution examination?

Load testing evaluates an application & # x27; s performance by increasing the lading (either the routine of cooccurring users or transactions) on the waiter. It determines the load that the coating can handle.

Distributed load essay simulates the load from a declamatory act of users, figurer, and device. Distributed loads provide a closer simulation of the real-world traffic and a best representation of the application ’ s performance.

4 Parts

Connet Now