‍

Ilya:So, today ’ s webinar is all about identifying mobile performance issues with Nimble, a HeadSpin company. And just to get started, we ’ re going to have an introduction by one of thefounders of the Nimble engineeringand then we ’ re going to bound into the point of and how to do that at build clip. We ’ re depart to talk about theproduction potentiality and go entire live demoso you can see for yourself. Then we ’ re move to cover somecustomer case studies. And so, without any farther ado, I ’ ll walk this over to Junfeng Yang. Are you with us?

Jungfeng: Excellent. Hello everyone. I ’ m Junfeng Yang, Chief Scientist at HeadSpin. Prior to that, I was the Cofounder and CEO of NimbleDroid, acquired by HeadSpin. Prior to that, I was a prof at Columbia University, act at Microsoft and also got my PhD from Stanford University. But for all of these 20 or so years, my passion has always be building better tools to facilitate engineer make software quicker, and improving the quality of the software, so that ’ s essentially my calling passion.

Ilya: asked me to talk a little bit about the background of our merchandise and how it came into existence. Around the year 2013, I had a bunch of students who calibrate and got into the job market and started working in mobile technology. They came back and complained that the mobile tooling ecosystem is so humbled, and that there are so many different devices out there, that the frameworks that are still reasonably young and fast-evolving, and that there only aren ’ t good tools around to help them figure out performance chokepoint and.

So, my PhD, then PhD, student Younghoon and I start working on some very cool technology to mechanically name performance issues use really deep system techniques and also machine learning or AI techniques and the resulting system gained a lot of attention in the mobile developer community. That ’ s when we consider about creating a product out of the technology so that we can benefit our gazillion of mobile developers. That ’ s all component of how the engineering the product, NimbleDroid, and the society. That ’ s the background. So yeah, Ilya, anything else that you wanted me to discuss?

Ilya:Thank you Junfeng, that was unadulterated as an entry on where this come from. Thank you very much sir.

So Junfeng is a Columbia professor, so he ’ s in New York City and thank you for joining us.

So, let ’ s talking about the existent product as it exists today. So specifically, let ’ s first with performance. So, we ’ re speak about performance of mobile apps. They could be native apps. They could be hybrid apps. So, we ’ re basically talking about Android and iOS.

Check out:

And why is performance so critical for your concern? Now, there ’ s not a ton of studies out there, but some of the key analyses done by folk like Amazon and Google – they basically agree that any time your users experience some wait in using the app, they ’ re unhappy.

Amazon ply 100 millisecond latency as apercent loss in sales.

Google ’ s talking about 500 msec latency is a20 % fall in user petition. From our client anecdotally, we ’ ve heard that anywhere from 250 to 500 milliseconds is significant enough thatusers will comprehend that.

So, if a user of your app is used to doing a certain function, you cognize, like, think of a shopping app where you ’ re going to search for a shirt, okay – or think of a finance app where I ’ m going to assure my investment portfolio.

If I ’ m used to that screen appearing in a certain time and there ’ s a new liberation of the app, and all of a sudden it conduct about 500 milliseconds more, that ’ s enough that it ’ s noticeable. Obviously the worse the delay, the less glad client you have.

So why is that? You know, peregrine execution is obviously complex and as Junfeng said, they looked at some of the reasons. So he named thing like the ecosystem be complex, lots of different devices.

Here at Nimble as a merchandise society, we ’ ve looked at what our customers are telling us and what we launch is the three big reasons why there ’ s this complexity in understanding performance is:

1. The developers are frequently making codification changes. And so by the clip you realise that there ’ s a performance impact, many commits have already happened. It ’ s very hard to track it down.
   
   
2. The second problem is just testing for performance regressions, right? For a human tester to do some manual testing, they won ’ t inevitably know that an app got 300 milliseconds slowdowns. But if you imagine about three subsequent liberation with 300 millisecond obtuse downs, now we ’ re talking about one mo of obtuse down, which clearly is intelligible by your users.
   
   
3. And the other problem, it ’ s not just for performance, flop? This is sort of a generic thing in software ontogeny, but software development, peculiarly these years, means you ’ re using a lot of third-party code: SDKs, library developed by open-source developers or by still early folks in your team, right? So there ’ s this poor visibility of how the execution of the SDKs or third-party libraries are impacting the overall app execution.

And we actually have some examples of each of these three things to shew you. So the other thing that we struggled with a lot is just tools, right? There ’ s mickle of tools on the market and so they all do different things, and they are better suited for certain things than others. So, APM tools for example, they ’ re great. They yield you a lot of visibleness into what your existent users are make. But when it comes to actually notice issues, it ’ s too late in the cycle, right? The app is already out and real users are encountering those issues. So, if you think about the common term these days, it ’ s shift left.

Also see:

If you want to reposition left on performance, you can ’ t wait until your users are experiencing some of these issues. Then the other thing we talked about already is only accurately measuring the performance. When you ’ re talking about a few hundred millisecond of difference, it ’ s really firmly to do that faithfully because if you open the same app a couple of multiplication on whatever gimmick you happen to have, it ’ s not going to take the same sum of clip.

So, identifying where performance is worsened versus just a glitch or a network problem is kind of a big deal.And we can show you what that look like.

And then of class Android and iOS have verydifferent toolingand very different ecosystems and come with their own challenges. So there ’ s that constituent of it and there ’ s a lot of cost in trying to do this yourself in price of become devices that are optimized for this kind of testing specifically for execution and getting people who are condition to look at performance issues.

One of the things that we encounter commonly with our customers is that many organisation do not hold a dedicated execution squad, although some do. And it ’ s really a question of both citizenry and time and technology hours and dollars finally on how you can well test your apps for performance and why you should.

So, the solution that Nimble App put together is meant to provide visibility and controlinto the performance of a given mobile app in the development stage. We fundamentally mix with CII so we can continuously monitor every body-build so we can quickly alert when there is a regression as soon as some debatable code is introduced. It go a lot easygoing to go back and understand what commit has caused this problem.

And then the early thing which I ’ ll show you today is the ability of our product to provide thesefine-grain diagnostics truly assist developer pinpointexactly which method or what constituent of their code is cause the slow down so they know where to look or where to start improving things and they can understand what variety of performance impact that ’ s going to have.

And so, the idea again is to switch left and to be able to identify issues originally in the cycle. And that ’ s what we ’ re talking about today.

Now typically, our users use our merchandise in a few different fashion:

1. One is a developer may want to just check their latest build against performance numbers and they can do that. The typical use case is CI desegregation where every time there is a pull request and a new build gets generated, it have profiled, and we can detect regressions in performance.
2. Also, some of our customers use it as a release criteria. So, they set up an actual budget of what execution figure are acceptable for certain distinctive UI interactions, and then they can check against those numbers.
3. And so finally, in production when you ’ re talking about a customer describe number, you can actually rerun it through Nimble and get some results to look at.

But as I said, the distinctive integration that we offer is a very, very easy andseamless integrating into the CI workflow. So you can see some logos here of some popular CI systems. We really integrate with any CI scheme – we haven ’ t met one yet that we can not integrate with. And it ’ s a rattling, very simple process. And as I explain farther what the execution metrics seem like, you ’ ll see for yourselves. It ’ s a very uncomplicated thing.So, that ’ s what we ’ re go to verbalise about today.

So, I ’ m move to really swap to a live demo and go through some of the scenarios so that you can see for yourself what our product looks like.

And again, for those folk who have joined us since the beginning, just a reminder, if you have a question, please use the interrogative tab on the right hand side – you can actually put your question in. And what we ’ re go to do is address them at the end of the Webinar. Please go ahead and do that if you feature any questions.

Okay. So, let ’ s talk about a few specific things that we ’ re going to look at today. So,identifying slowdowns in third party codification is real significant. It ’ s one of the three primary challenges that we found.

And here ’ s an model from a real customer we have: this is actually from Aaron ’ s personal blog, but Aaron is an engineer that works for a company that is one of our customers. And what he ’ s calling out is essentially that get resources stream could take up to two seconds to create this initializer. And he ’ s basically pointing out that there ’ s a very simple fix – you can just make the adapter lazily format and it saves anyplace from one to two seconds at a startup time.

So let me really show you what Nimble does and how it demo these kinds of number so you can see for yourself. So, this is the initiative thing we ’ re going to do.

So, as I mentioned, Nimble will go forward and integrate with your CRM system and every time there is an upload of the app,Quick is travel to analyze that app.

So, this first step that I ’ m display you is a shopping app. So, one of the chief things that we look at, honestly, one of the first thing that any organization that starts like about execution is going to look at is the cold startup number and cold startup is literally from when you found the app until the App UI is usable by your users. And so, each of these dit represents a specific build direct from plausibly their release branch.

Also check:

And you can see the results in terms of how long it took to get to the app and make it operable. And other scenarios over here, like adding an item to the shopping cart, for exemplar, or fetch some recommendation – these are custom trial scenarios, which either we, Nimble, or our customers themselves can build. And we ’ ll talk a little bit more in detail about the test frameworks that we endorse.

But the mind again isevery clip there ’ s a new build, we can get these performance figure.And so what that allows an organization to do is as soon as there ’ s a performance fixation, it is connected to the previous build and it makes it very elementary for developer to understand what incisively is get that regression.

In fact, if we expand this out to say all 22 uploads and reload the dashboard, you ’ ll see for yourselves basically for cold startup, the numbers are fairly like, pretty like, fairly similar. They ’ re not incisively the like as the code modify arrive in and out, but they ’ re very minimum regressions or pickle.

And then suddenly cold startup jumps to 2.5 second and then even 2.6 seconds. So each of these two data points would get generated an alert to the squad. And when the developer looks at this, it makes it very simple to understand what exactly regress because I can merely click on this and say, compare this upload with the current.

And by the way, you can compare any two arbitrary points, but thetypical use case is comparing the current or the next anatomy – the latest shape– with the previous builds. And so that ’ s what we see here. So, the app didn ’ t change too lots. If you appear at the file size or the method count, they ’ re really similar. But the timing of the cold inauguration definitely return and depart from 1.13 minute to 2.5 seconds.

And so, as I continue saying,Agile makes it very simple to see why the code return. And there ’ s two means we do that.

1) One is we actually identifyspecific reasons for retardation. So, in this case we ’ re looking at three different thing.

• One is anything that ’ s on the UI thread, which is locomote to actually block or hang the UI.
• Then we ’ re also able to find hung wait method, which fundamentally means a method on the UI thread is waiting for some background thread to stop.
• And then the last thing is basically any method which extend in the ground, but for longer than 100 milliseconds.

Pro tip: Tools like SUSA can handle this autonomously — upload your app and get results without writing a single test script.

So, if you, again, if you recollect that the unexpended side used to be fast than the correct side is where things got slow – it ’ s very clear that the splash activity on create exists in both instance and the timing is about the same 254 versus 253 milliseconds.

But there ’ s a new principal activity on create which was added hither and it ’ s very long. And specifically, if you look at the vociferation stack for this, the independent activity on create method is depart to call the principal activity initialize method, which itself is 2100 milliseconds.

And if you want more info than that, we can actually pull up thefull call lotfor the entire application like this. And so again, what you see is the UI thread on the left for example, was really flying, 312 milliseconds. And I can only zoom in and show everything that happens as part of the UI thread. And here it ’ s near 2.5 millisecond. And you can see that the large method or the method that takes the longest time to run is as big integer add method.

So, what this boil downwards to is adeveloper made a code change. They submitted their body-build, we realized that there ’ s a performance problem, and by looking at the previous build versus this build, it ’ s very clear that this new method which was impart is in fact the method that is have the problem.

And so nowdeveloper can go back and modification this method, get rid of this method, rewrite the code, whatever is the appropriate remedy, but they understand that this method and this timing information is the large problem here. There ’ s former methods being called by the onCreate method, but they ’ re very speedy, so they ’ re not causing us any problems.

We can also pull up a timeline view – not really necessary here. Plus of course this is a demonstration, so not a existent client model.

But the timeline view basically shows us the threads and it also shows us when certain threads are launched, and if the UI yarn is wait on something to finish, it shows that as well.

So here the UI thread is actually waiting for a couple of point from a background thread to finish. And this is causing some wait as well. So to summarise, adistinctive use case for Nimble is you tie it into CI. Every time there ’ s a new build, it become profiled for cold startup as well as any kind of custom tests. That info is pass to the squad, so that as soon as there is a performance regression, so we can communicate that to the users.

Now to further explain what ’ s happening here, actually one of the main things that we ’ ve notice is if you look at performance and you run the app a couple of times on a twosome of different devices, you will get very different readings.

So, another thing that Nimble offers, and it ’ s really the groundwork of our platform, is these thinking or thesenumbers are very reliable, run over run. And the intellect that they ’ re so reliable is, easily there ’ s two reasons.

1) One is we have existentphysical real devices in a farmwhich are used to install the app, run it and profile it. And those devices are all just the same. So if you think of a typical gimmick farm, it may have lots of different devices and that may be useful for answering the inquiry of how is my app do on this device versus that device.

But for this kind of frame over build performance examination, in order for it to make sensation, you have to have exactly the same position and this build as good as that built. Otherwise the numbers won ’ t match up on the trend graph.

And so Nimble provides that by having a gimmick farm comprised ofincisively the same devices, configure in exactly the same way, sitting on the same network pipe in the same locationso that when you submit a new body-build, if the number are different as they are, in this event, there ’ sonly two possibilities:

1. Either you havenetwork trafficthat guide a different amount of time
2. or you ’ ve get achange in the codeand we can shew you that.

And in fact if you look at thetiminginformation on the cry stack, it get very leisurely to recount whether this is network related or CPU colligate, because what we ’ re showing hither is the CPU timing for how long each method takes to execute. So if these two were exactly the like, then the issue would be on the network side. If these two are drastically different, then clearly the issue is in the codification itself.

So another thing thatNimble provides is the functionality for iOS. Now, iOS, as I ’ m sure anybody who ’ s work on iOS knows, Apple has some curveballs when it comes to third company tooling and running various things. And so we ’ ve essentially dealt with all of that. So we also feature a device farm of iOS devices and we can do exactly the same thing in iOS that we do for Android.

So, as you can see here, it ’ s kind of a safe shopping app, but in an iOS surround and I but want you to see that it work exactly the like. Every build comes in, it gets profile. We can certainly do a “ compare this upload with current ” – so you can see the difference. I actually compared this upload with current, sorry about that.

So you see that in this case this is an IPA that ’ s being run and there might be some departure in this one and the timing travel down. And so again, you can attract up a outcry stack and see what the differences are side over side. And so it makes it really easy to see this in iOS as well as Android.

Recently, we ’ ve hadsome requests for web apps. And so this is a demo which we do on Kohl ’ s. So all of these are basically pulling up the Kohl ’ s website inside a mobile browser. And again, we can basically profile it for clock information and show when there are differences. And when you look at the item for this, you can see that we ’ re also identifying lag.

In the case of a web app, they ’ re basically JavaScript functionalities. In the case of Android, they ’ re Java code and the case of iOS, it ’ s either Swift or Objective C.

But essentially by appear at the call stack and looking at the method timing, whatever the languages of each program, we can identify these slowdowns. And more importantly, we can profile this and give these a relative number from one run to another.

Another thing that commonly happens is when a customer wants to look at improving the functionality of a certain app, they can look at the answer that we provide, not by a build over build fascia, but by really looking at a unique set of results. And so I ’ d like to exhibit you what that looks like.

This is an app that I pulled from the app fund. So this is just one of the, not the latest, but one of the late versions of this app telephone Runtastic. We grab the APK from the Google Play Store and we ’ ve analyzed it.

One of the things that you can see hither is a login that takes 4.6 seconds, and arguably if I wanted to ameliorate that clip, the question naturally is: what ’ s causing it to be long and where should I centre my sweat?

What Nimble make, as you can see here, is we ’ re basically starting when the exploiter clicks the login button and we ’ re end when the literal logged-in screen or a version of the app is fully populated. This is driven by an automation test, which in this lawsuit we wrote. There ’ s 10 retardation and we can look at the actual details and see the real methods. These are on the UI thread if you recollect because these are the ones that are actually capable of hang the CPU and hanging the UI ribbon.

These are ground method. And so looking at this, we can say, if you require to commence working on improving this particular app and this particular flow specifically, this is what you ’ re doing. This is the thing that is the path in the code that is taken.

If you want it to start improving this app ’ s performance for login, this is where you would concentre. Here ’ s the full shout stack so you can see for yourself what ’ s involved in making these yell.

For example, here we ’ re pulling in some datum from maps and this is on the Ui thread and it ’ s almost a c milliseconds, right? So if we can offload that to a ground thread, that would be great, and that would cause a performance improvement here.

But there ’ s some former interesting thing we can see here.

We utter a small bit about third company codification. And the thing about third party code is, whether it ’ s open source or in a tumid organization – mayhap it ’ s a different team that ’ s create some capacity –traditionally, it ’ s difficult to read if the app performance is due to “ my code ” or “ somebody else ’ s codification. ”

In this case,this is how easy Nimble get it. So this is Gson. It ’ s an unfastened source library for parse Json. And in this scenario, which if you think, is about four and a half bit, almost a second is actually fagged parse Json use this Gson Library. So that ’ s variety of a strange thing. You would expect this to be very quick.

So again, if we go backwards to the original slowdowns, we can really trace it down that way. So I ’ m looking at some of these methods. You can really see where that Gson is taking place.

Here ’ s a outstanding representative of that and I ’ ll exactly go to the beginning of it up here. This is a background thread and it ’ s run in the background. So that ’ s not directly going to freeze the UI thread, but obviously it has to finish processing in order for the app to be utilitarian.

And this is what it ’ s doing – you can see that Gson is being called and the reason that it ’ s so slow, relatively speaking, is because Gson utilise rumination and on Android, expression is actually incredibly dumb.

In fact, we have a couple things further in this webinar that explicate that a little bit better. We likewise provide you some links that you can look at.

But in general, the Gson Third Party library is causing some slow downs here. And I can track down into the code and see that this is due to this reflection. But the thing I find most interesting is actually, if you look at the gens hither, com.newrelic.agent – so not only am I identifying a problem in the tertiary party library, which is Gson, but the really interesting thing is it ’ s actually being introduced not even by the native apps functionality – it ’ s really because they receive New Relic and New Relic is creating some slow Down.

Check

I mentioned this earlier, New Relic is an APM tool. They provide a lot of utile information. So it may be the case that they ’ re unforced to accept that retard down because it provides so much useful info and that ’ s great.

But what you can provide to New Relic itself now is a break down and really say “ Hey, we love your product but you ’ re have us some job. Is there anything you can do to rush this up? ”

And New Relic would actually be wise to change out of Gson because there are other libraries for parse Gson and in the call stack, it ’ s really very clear that this happens a lot.

So here ’ s a ground yarn with a slight over one sec of runtime. And ultimately if you look at this, this is opening up some web server petition, which ultimately is going to phone New Relic agent, which is finally depart to name this to Json method, which is 300 msec here. And then guess what, this is another New Relic cry that ’ s 358 milliseconds over here and yet again over here for another 200. So, you see how this is all bestow up. And even though this is the ground thread, this is actually responsible for nearly the one second of CPU time.

So that ’ s the kind of thing that you can get from Nimble. Just to summarize, you secure it into CI or you could do it as a one-off bod. We verbalise about both of these scenarios and what it give you is this level of detail so that you can either work on amend the execution of a certain task or you can track it, build over body-build, then make certain that it does not regress.

In fact, a lot of our client will verbalize about make a baseline and then exactly making sure that nothing retroversion or gets slower from that baseline. So thither ’ s a match of different ways of using this. And, that ’ s form of the unrecorded demo.

I ’ m proceed to go back into the slides and cover the customer case studies and such. And again, if you hold any questions or if you want to appear at anything else, by all means, please put the questions into the question panel.

This is an illustration of name a tertiary party problem in Moshi Kotlin and some explanation thither.

This is the Gson example that I already testify you.

I get another case report here. This is from one of our client who did not agree to share their gens publicly, so I had to redact some of it, but this is a existent world example where we ’ re looking at three different variation of an iOS app, really popular one.

It basically has this huge spike which was later fixed. And so this is the worry one. And again, if we compare the before and after, it ’ s pretty clear to see that on the left hand side, the hot methods are 300 millisecond, 280 msec, On the right script side, there ’ s some database entree proceed on and it ’ s just very slow.

This is the kind of thing you can see with Nimble. This is an iOS app and this is what the full call mess looks like. And again, you can see under the UI application main, depart from 2000 milliseconds- so from two seconds to seven s, that ’ s a five second change and believe me, five sec is very perceivable to a user. I use this app. I ’ m sure a lot of you have to.

This is the sort of thing that we can place with Nimble and this is how it makes it possible for a developer to look at that and say, okay, I get it. I need to focus on vary this method, which ultimately will have this – ultimately, it ’ s either this guy or this guy that ’ s going to be my two biggest contributors and that ’ s what hap.

So that ’ s one existent model. But we receive some case studies here.

So Flipkartis a outstanding customer of ours – been habituate us for over two years now. They use this exactly the way I described. It ’ s establish into CI. They analyze every build and whenever we observe the regression for them, they use Agile to analyze what the issue is. You can see they restore it in two or three builds, improved it and made sure that it goes back to where it should be.

Flipkart is a big assistant because we deliver a lot of value to them. Of course, Flipkart has many customers. As a sales app, people tend to leave things in the shopping handcart. People tend to get mad and go use something else. So that ’ s a big thing for Flipkart.

Here ’ s another [case study] – they didn ’ t let us use their name, but this is a rattling turgid fellowship with lots of different teams.

What we launch is when you look at how frequently that many engineers make code changes, the sooner you can regain a problem, the better it is. After six month, we basically filed these credential. If you think about the engineering time that ’ s necessary to go rearward and understand why the performance alter – the more commits you feature to wade through, the more individual code changes you hold to analyze – the harder it ’ s depart to be.

Being able to look at it on a figure by build grade where the change is reasonably small is much better than being able to only look at edition to version comparison.

This is a great case study ofNew York Times– did this test with us. They wanted to improve their cold startup. Google recommends two sec or less for cold startup. As you can see here, theirs was hovering at 4.3 and then they started work with us to better it. Actually, it was even 5.6 initially.

So there is a link here – I ’ ll give you the URL at the end of the Webinar and we ’ ll obviously provide that with our follow-up, but they have a whole blog where they explained exactly what kinds of subject they found. This actually happened nigh two age ago, but it ’ s such a great case study.

For me as a customer facing engineer, what ’ s interesting is that even though this happened two years ago, I go to customers today and I see the like form of thing and just a looter: Gson and reflection is a big reason for why they had this slow startup time, which they fixed.

This blog post explains incisively why and how they determine it. As you precisely saw from the latest Runtastic version, people are still hitting those problems on Android today. So, this is not something that went away – still though this is this particular case studies two years old, it ’ s really still entirely valid based on what I see in the battleground. We ’ ll portion the tie at the end of the Webinar and we ’ ll postdate up with that. I recommend you take a look at that. If you have any Android development.

The former case study – a little bit newer, also a great blog post – which you can read from our friends atPinterest. They actually use us in a performance team and we are a big part of providing them a and iOS tryout and identify performance subject early.

They feature this nice blog post where they explained some of the thing that they ’ ve seen with us and what variety of issues they ’ ve been capable to bump and fix. And again, we ’ ll share the link and here they are: insert URLs here.

The three things that I showed in the webinar, these are the URLs. If you want to take a look at them and I do commend you guys occupy a look at it, we ’ ll share these links with you along with the transcription.

At this point, exactly to summarize,Nimble enables our customers to accurately monitor and profile every critical user flow of Android and iOS apps, and now even web apps. We integrate pretty seamlessly into your existing workflow, which signify it ’ s a set and forget sort of thing. You put it into CI and so it just works.

FAQs

Q:What are the key parameter to consider during performance examination?

Some of the critical element of performance testing are

• Memory usage
• Network Bandwidth
• Network queue and latency
• CPU interruption
• CPU usage
• Thread counts

Q: What is Performance Regression Testing?

Performance regression test determines how an application performs compared to previous versions. This test set a baseline performance and check that any changes in the application do not create new execution matter.

Q: How is client-side execution testing different from traditional execution testing?

Performance testing traditionally centre on the reaction from the server with test like lading tests, stress tests, spike tests. While extremely useful, traditional performance examination execute not test the behavior of an application on the web browser or a mobile covering.

Client side performance screen complement the traditional performance screen to supply an end-to-end execution. It includes screen the effects on web+mobile portion like JavaScript, images, picture, CSS, battery, localization lineament, networks on application performance.

4 Parts

Connet Now