Let & # x27; s face it, have sometimes been accused of being slow and unreliable. In some slipway the accusation is true: there are fundamental speed limits to the automation engineering Appium relies on, and in the world of full-fledged there are a legion of environmental problems which can lead to test instability. In other ways, the accusation is misplaced, because there are strategies, we can use to get certain our test don & # x27; t run into common pit.

This clause is the initiative in a multi-part serial on test hurrying and reliability, instigate by a webinar I give recently on the same subject (you cancatch the webinar hither). The webinar was so jam-packed with content that I barely had the chance to get my breath in between topics and I notwithstanding went over time! So, in this series we & # x27; re going to take each part a little slower and in more detail. For this first part, we & # x27; ll discuss the blood-pressure-raising whim of test flakiness.

& quot; Flakiness & quot;

SUSA automates exploratory testing with persona-driven behavior, catching bugs that scripted automation misses.

No give-and-take of functional test dependableness would be accomplished without speak the concept of & quot; flakiness & quot;. According to mutual usage, & quot; flakey & quot; is synonymous with & quot; unreliable & quot; -- -the test passing sometimes and betray early time. The blame here is much put on Appium -- -if a trial passes once when run topically, sure any future failures are due to a problem with the automation engineering? It & # x27; s a tempting position to take, especially because it guide us (the test authors) and our apps out of the crosshairs of blame and let us to place responsibility on something outside that we don & # x27; t control (scapegoat lots?).

Check out:

In reality, the situation is much more complex. It may indeed be the case that Appium is responsible for unreliable behavior, and regrettably this does happen in reality. But without an investigation thatprovesthis to be the case for your particular test, the problem may with equal probability lie in any number of former country, for example:

• Unwarranted supposition get by the tryout author about app or device speed, app state, screen size, or dynamical content
• App unbalance (maybe the app itself exhibit erratic behaviour, even when used by a human!)
• Lack of compute or memory resources on the machine hosting a simulator/emulator
• The mesh (sometimes HTTP requests to your backend just fail, due to load or issues outside of your team & # x27; s control)
• The twist itself (as we all know, sometimes existent devices but do odd things)

Also chit: Don ’ t Rely on iOS Emulators & amp; Android Simulators.

Furthermore, even if it can be evidence that none of these areas are elusive, and therefore & quot; Appium & quot; is responsible, what does that mean? Appium is not one monolithic beast, even though to the user of Appium it might look that way. It is in fact a unhurt stack of technologies, and the erratic behavior could exist at any layer. To illustrate this, have a look at this diagram, showing the various spot of the flock that come into play during an:

The component of this stack that the Appium squad is creditworthy for is really not that deep. In many cases, the job lies deeper, potentially with the mechanisation tools provided by the mobile vendors (XCUITest and UiAutomator2 for example), or some other automation library.

Read:

Why go into all this explanation? My primary point isn & # x27; t to take the blame out from Appium. I want us to understand that when we say a test is & quot; bizarre & quot;, what we truly mean is & quot; this test sometimes passes and sometimes fails, and I don & # x27; t know why & quot;. Some testers are OK stopping thither and countenance the build to be flakey. And it & # x27; s true that some measure of instability is a fact of living for functional tests. But I want to boost us not to bond our heads in the sand -- -the instability we can & # x27; t do anything about is relatively small compare to the flakiness we oft decide for out of an dodging of a unmanageable probe.

My rule of thumb is this: only allow outre tryout whose flakiness iswell understood and can not be addressed. This means, of course, that you may need to get your paw dirty to figure out what exactly is going on, including coming to the Appium team and asking questions when it looks like you & # x27; ve pinned the problem down to something in the mechanization stack. And it might mean ringing some alarum doorbell for your app dev team or your backend squad, if as a effect of your investigation you learn job in those areas. (One mutual problem when extend many tests in parallel, for representative, is that a build-only backend service might be underpowered for the number of requests it receives during testing, result to random instability all over. The solution hither is either to run fewer tests at a time, or best yet, get the backend team to beef up the imagination available to the service!)

Like any kind of debugging, probe into gonzo tests can be daunting, and are led as much by intuition as by method. If you keep your eyes open, however, you will probably do the critical observance that move your investigation forward. For example, you might notice that a certain variety of flakiness is not isolated to one test, but rather pops up across the whole build, seemingly randomly. When you examine the log, you discover that this kind of daftness always happens at a certain clip of day. This is great information to take to another team, who might be able to interpret it for you. & quot; Oh, that & # x27; s when this actually expensive cron job is running on all the machines that host our! & quot;, for example.

We & # x27; ll dig into the topic of debugging failed tests in a succeeding part in this series. For now, my concrete recommendation for handling flakiness in a CI environment in general is as follows:

1. Determine whether a test is flakeybeforepermanently bestow it to your frame. In a pure world, this would look like mechanically catching any functional test that & # x27; s be devote and lam it many times (maybe 100?) to build a dependableness profile for it. If it pass 100 % of the time, great! Merge that perpetrate to dominate and off you go.
2. If the test doesn & # x27; t ever pass, it & # x27; s treacherous or flakey. But we can & # x27; t cease there, because & quot; flakey & quot; is a codeword for ignorance. Time to dig in and find out why it & # x27; s treacherous. Usually with a little investigation it & # x27; s possible to see what went incorrect, and perhaps adjust an element locator or an denotative wait to handle the problem. At this point, Appium log and step-by-step screenshots are essential.
3. Once you discover the cause of flakiness, you & # x27; ll either be able to resolve the flakiness or not. If you can do something to resolve it, it is incumbent on you to do so! So, get that tryout passing 100 % of the time. If you determine that there & # x27; s zip you can do about it (and no, filing a bug account to Appium or to Apple is not & quot; nothing & quot;), you have two options: either forfeit the test if it & # x27; s not locomote to provide more value than headache in the long run, or annotate it such that your CI system will retry the trial once or twice before considering it failed. (Or just run it before a freeing, when you feature time to manually control whether a failure is a & quot; flake & quot;).
4. If you take the approach of keeping the test in your build and permit the soma to rehear it on failure, youmusttrack statistics about how much each test is retried and receive some reliability threshold above which a new investigating is triggered. You don & # x27; t want tests creeping up in flakiness over clip, because that could be a signal of a real trouble with your app.

Also read:

Keep in mind that Appium tests are functional tests, and not unit test. Unit tests are hermetically seal off from anything else, whereas functional tests live in the real world, and the real world is lots messier. We should not aim for complete codification coverage via functional testing. Start small, by covering critical exploiter flows and let value out of catching glitch with a few tests. Meanwhile, make sure those few tests are as rock solid as possible. You will learn a lot about your app and your whole surroundings by hardening even a few tests. Then you & # x27; ll be able to invest that learning into new tests from the outset, rather than having to fix the like sort of flakiness over and over again down the route.

Ready for more test robustness? Head on over to, where we & # x27; ll talk about apace and dependably chance elements in your apps!

4 Parts

Connet Now