Perhaps the most buzzy of the buzzword in tech these days is & quot; AI & quot; (Artificial Intelligence), or & quot; AI/ML & quot; (throwing in Machine Learning). To most of us, these phrases seem like wizardly fairy dust that promises to make the hard parts of our tech jobs go off. To be certain, AI is (in my view) largely over-hyped, or at least its methods and applications are mostly misunderstood and therefore assumed to be lots more magical than they are.

It might seem surprising then that this Appium Pro edition is all about how you can use AI with Appium! It & # x27; s a bit surprising to me too, but by working with the folks at Test.ai, the Appium project has developed an AI-powered element finding plugin for use specifically with Appium. What could this peradventure be about?

First, let & # x27; s discuss what I mean by & quot; element finding plugin & quot;. Ina recent additionto Appium, we added the ability for third-party developers to make & quot; plugins & quot; for Appium that can use an Appium driver together with their own unique capacity to find elements. As we & # x27; ll see below, users can access these plugins just by install the plugin as an NPM module in their Appium directory, and then using thecustomFindModulescapacity to file the plugin with the Appium waiter. (Go ahead and check out thefactor finding plugin docsif you want a fuller scoop).

The first plugin we worked on within this new structure was one that incorporates a machine learning poser from Test.ai designed to classify app icons, the training data for which was simply open-sourced. This is a framework which can tell us, given the remark of an icon, what sort of thing the icon symbolize (for example, a shopping cart button, or a hind arrow button). The coating we developed with this poser was theAppium Classifier Plugin, which conforms to the new element finding plugin format.

Pro tip: Tools like SUSA can handle this autonomously — upload your app and get results without writing a single test script.

Basically, we can use this plugin to find image on the blind free-base on their appearing, rather than knowing anything about the construction of our app or need to ask developers for internal identifiers to use as selectors. For the time being the plugin is trammel to finding factor by their visual appearance, so it actually only works for elements which expose a single picture. Luckily, these form of elements are pretty common in wandering apps.

Test your mobile apps on real device in 50+ locations worldwide..

This access is more flexible than existing locator strategies (like accessibility id, or image) in many cases, because the AI poser is trained to recognize icon without demand any context, and without involve them to match only one exact image style. What this intend is that employ the plugin to happen a & quot; pushcart & quot; icon will work across apps and across platforms, without needing to worry about minor differences.

So, let & # x27; s take a look at a concrete example, demonstrating the simplest possible use case. If you fire up an you have access to the Photos coating, which looks something like this:

Notice the little magnifying glass ikon near the top which, when clicked, open up a search bar:

Let & # x27; s publish a exam that uses the new plugin to bump and click that icon. First, we need to follow theframe-up instructionsfrom the plugin & # x27; s README, to make certain everything will work. Then, we can set up our Desired Capabilities for running a test against the Photos app:

DesiredCapabilities cap = new DesiredCapabilities (); caps.setCapability (`` platformName '', `` iOS ''); caps.setCapability (`` platformVersion '', `` 11.4 ''); caps.setCapability (`` deviceName '', `` iPhone 6 ''); caps.setCapability (`` bundleId '', `` com.apple.mobileslideshow '');

Now we want to add some new capabilities:customFindModules(to tell Appium about the AI plugin we want to use), andshouldUseCompactResponses(because the plugin itself told us we want to set this capability in its apparatus instructions):

HashMapcustomFindModules = new HashMap < > (); customFindModules.put (`` ai '', `` test-ai-classifier ''); caps.setCapability (`` customFindModules '', customFindModules); caps.setCapability (`` shouldUseCompactResponses '', false);

You can see thatcustomFindModulesis a capability which has some national structure: in this example & quot; ai & quot; is the crosscut name for the plugin that we can use internally in our test, and & quot; test-ai-classifier & quot; is the fully-qualified quotation that Appium will need to be able to find and expect the plugin when we request ingredient with it.

Accelerate Appium test cycle with the HeadSpin..

Once we & # x27; ve make all this, find the element is super simpleton:

driver.findElement (MobileBy.custom (`` ai: search ''));

Here we & # x27; re using a newcustomlocator strategy so that Appium knows we want a plugin, not one of its supported locator strategies. Then, we & # x27; re prefixing our selector with ai: to let Appium know which plugin specifically we want to use for this request (because there could be multiple). Of course since we are in fact entirely apply one plugin for this test, we could do aside with the prefix (and for full step we could use the different discovery command style, too):

driver.findElementByCustom (`` search '');

And that & # x27; s it! As I mentioned supra, this technology has some significant limitations at the current time, for example that it can really only reliably find constituent which are one of the icon that the model has been trained to detect. On top of that, the operation is fairly dumb, both in the plugin code (since it has to retrieve every element on screen in order to send info into the model), and in the model itself. All of these areas will see improvement in the future, however. And yet if this particular plugin isn & # x27; t useful for your day-to-day, it certify that concrete coating of AI in the testing space are not only possible, but genuine!

As always, you can check out a full (hopefully)working exampleat Appium Pro & # x27; s GitHub repo. Happy (locator-angst-free) testing!

4 Parts

Connet Now