HeadSpin ’ s Regression Intelligence is a amply automatise solution and provides a set of capabilities to help you discover regression early and amend character assurance. This article is the first part of the two series. The goal is to give you a hard-nosed idea of how Regression Intelligence deeds by direct you through create custom KPIs and put regression alerts. This knowledge is the understructure for realize the more advanced topics covered in, where we dive into statistic and the more profound production realm. Through this series, you will read all construct necessary to utilise Regression Intelligence where applicable and help you maximise your investment. For those look for BASIC, please refer to and the next materials:

• “”
• “”

In this article, we will cover:

1. What is a impost KPI?
2. Creating a custom KPI?
3. What alternative are available for regression detection and alerting?
4. Regression monitoring with Grafana
5. Regression monitoring with Watcher
6. Conclusion

What is a custom KPI?

A Key Performance Indicator (KPI) is a quantifiable index of performance that you have identified as important to measure the success of your application. HeadSpin indorse data accumulation frommore than 120 data point. You choose one or more KPIs from the standard set that HeadSpin supports out of the box, or make your own KPI if the KPI you demand is not found in the default setting.

‍

See how beautifully the data collected is plotted. The “ Max Current ” values collected from multiple sessions from the selected “ Samsung SM-j610G ” twist are highlighted in the orange line. Looking at the course, you can easy find spikes in the data set where the anomaly occurred and correlate them with over 100 KPIs at a glance. You can also jump to an individual session to open the particular page or switch highlights by selecting another device or region. For more details about Performance UI, see Performance Monitoring UI.

This is simply an introductory feature of Regression Intelligence. In the following, I will extend how to make this “ Max Current ” KPI and automatically receive notification when an anomaly or regression is detected.

Creating a custom KPI

Here, you will create a custom KPI called “ Max Current ”. Let ’ s get a session and collect data from the remote control. Here is the instruction. After the session is recorded, for this exercise, you will use summary statistics to get the max current by adding a “ time-series-request ” label as testify below.

Adding a time-series-request (stats) label via Web UI

‍

The label capability enable you to annotate the collected data and aid your analysis. There are multiple ways to create labels: by calling Session Label API or LOG functions, so labels can be created while the app or automatise test is extend. You can also use the COPY method to replicate labels across sessions.

Once the label is contribute successfully, you will see the result below.

‍

The undermentioned API will make the “ time-series-request ” label via Web UI:

Adding a time-series-request (stats) label via API

roll -- request POST 'https: // {{auth_token}} @ api-dev.headspin.io/v0/sessions/ {{session_id}} /label/add ' \ -- data-raw ' {`` label_type '': `` time-series-request '', `` name '': `` max current '', `` start_time '': 0, `` end_time '': 38.252, `` data '': {`` method '': `` stats '', `` time_series_key '': `` battery_current '', `` parameters '': {`` prosody '': [`` max '', '' mean '', `` percentile 10 '', `` percentile 97.5 '']}}} '

To hold the MAX calculation result, the Session Analysis API below must be telephone separately. The result can not be obtained from the Web UI as of this authorship. Specify the Label Group ID, which you noted earlier, as a parameter.

curlicue -- request GET 'https: // {{auth_token}} @ api-dev.headspin.io/v0/sessions/analysis/timeseries/stats/ {{session_id}}? label_group_id= {{label_group_id}} '

The following API is retrovert. In this suit, the maximal value observed in the galvanic current in this session is 176.116mA.

{`` summary_statistics '': [{`` time_series_key '': `` battery_current '', `` label_group_id '': `` b1150db6-cba9-4607-a1fd-25f08a43598f '', `` session_id '': `` c9cc7333-dd5c-11ec-96bb-025b84ee4231 '', `` metric '': `` max '', `` value '': 176.116, `` time_intervals_ms '': [[0, 38252]]}, {`` time_series_key '': `` battery_current '', `` label_group_id '': `` b1150db6-cba9-4607-a1fd-25f08a43598f '', `` session_id '': `` c9cc7333-dd5c-11ec-96bb-025b84ee4231 '', `` metric '': `` mean '', `` value '': 162.61328571428572, `` time_intervals_ms '': [[0, 38252]]}, {`` time_series_key '': `` battery_current '', `` label_group_id '': `` b1150db6-cba9-4607-a1fd-25f08a43598f '', `` session_id '': `` c9cc7333-dd5c-11ec-96bb-025b84ee4231 '', `` metric '': `` percentile 10.0 '', `` value '': 157.734, `` time_intervals_ms '': [[0, 38252]]}, {`` time_series_key '': `` battery_current '', `` label_group_id '': `` b1150db6-cba9-4607-a1fd-25f08a43598f '', `` session_id '': `` c9cc7333-dd5c-11ec-96bb-025b84ee4231 '', `` metric '': `` percentile 97.5 '', `` value '': 168.668, `` time_intervals_ms '': [[0, 38252]]}]}

Next, you will use the and add the retrieved value (i.e. 176.116) to a user flow. is a data repository for storing session data grow from the same user action. Typically, you will create a stream for each critical user operation (e.g. cold inauguration, shopping cart, payment transaction) where any failure triggers user frustration.

Adding a custom KPI “ Max Current ” to a user stream via API

curl -- request POST 'https: // {{auth_token}} @ api-dev.headspin.io/v0/perftests/upload ' \ -- data-raw ' {`` test_name '': `` {{user flow name}} '', `` session_id '': `` {{session_id}} '', `` status '': `` passed '', `` information '': [{`` key '': `` My Measurement '', `` value '': {{any value}}}, {`` key '': `` Max Current '', `` value '': 176.116}]} '

If the data collected through a session is not valid, set the “ position ” to either & quot; neglect & quot; or & quot; excluded & quot; to permeate out the session from the analysis.

That ’ s it. Here, we learnt how to create a “ Max Current ” KPI. Next, we will set up regression detection and alerts for this KPI.

What options are uncommitted for regression espial and alerting?

There are two approaches uncommitted; one employ Grafana, and the other uses Regression Intelligence. Depending on your use case, you will involve to cognise when to use which depending on your use case. See below are some guidelines:

Grafana

• Supports various charts and alert pick (e.g. email, webhook and more) that act on any KPIs, whether aggregated or not.
• Excellent selection to picture the relationship between different KPIs and discover hidden insights in the population.
• Knowledge of SQL and data representation skill are postulate.
• Fundamentally makes the detection of an anomaly session easier.

Regression Intelligence

• Supports the movement line prospect and email alert that works on network-based or duration-based KPIs (aggregated only).
• The goal is to predict the overall failure rate from the collected samples if the product be shipped as-is.
• Knowledge of statistic is required.
• Fundamentally makes comparisons of 2 data sets easier.

Not intuitively perceivable? In the following, I will walk you through two use lawsuit to interpret these differences visually.

Regression monitoring with Grafana

Here, you will see how to trigger an alarum when the electric current observed during a session exceeds 800mA. For this exercise, you will use the “ Max Current ” KPI refer earlier. Grafana is the correct choice for this task as neither statistical processing nor data collecting is command. It monitors all sessions and notifies when an anomaly with the “ Max Current ” KPI is detected.

The steps below assume you get created more than one session with the “ Max Current ” KPI store in a designated exploiter stream.

Setup Steps:

1. Open and follow the instructions to spark the Grafana lineament.

2. Navigate to a user flow where sessions with the “ Max Current ” KPI are stored.

3. Follow the illustrated steps below and export your exploiter flow to the Postgres Replica database.

‍

4. Navigate tohttps: //grafana-dev.headspin.io/grafana/login

5. Create a new splashboard.

6. Create a new panel similar to the image below.

‍

7. Optional: Create a data nexus and set it to

“ ”. With this,

you will be able to spring to an individual session of your interest.

‍

8. Set an e-mail alerting. Below is an example configuration

‍

9. Save and done. The final output should seem like the below.

‍

10. You will receive an email alarum when Grafana notice a session with a “ Max Current ” KPI greater than 800mA.

‍

Congratulations! With this bare use case, Grafana is sufficient. SQL Knowledge is command, but once the frame-up is done, you can let Grafana supervise fixation and you rivet on other anteriority tasks relevant to your business. Grafana endorse various graph, so explore alive sampleshereand imagine where else you can utilise this approach.

Succeeding, we will look at another use case where Regression Intelligence is better suited.

SUSA automates exploratory testing with persona-driven behavior, catching bugs that scripted automation misses.

Regression monitoring with Watcher

Here, we want to compare information collected from two different builds of your application, compute the middling full time where each build exceeds 400mA, and trigger an alert if the full time of the newer build increases by 10 %. In this case, use Regression Intelligence for the reasons below:

• Grafana doesn & # x27; t support the use of percentages for door.
• We need to findstatistically significant differencesin a large population, representing your entire customer groundwork, with clip length as a unit.
• We want to section the collected data from various perspective (e.g. progress, twist, region, tags) and perform comparability and analyses.
• It is not pragmatic to alter SQLs or panel settings in Grafana every clip a aspect needs to be added or updated.

It is not easy to understand without seeing it, isn ’ t it? Let ’ s deep honkytonk.

Setup Steps:

1. Start a session and amass data from the removed control. Here is.

2. Label the locating where the current value is great than 400mA.

For this undertaking, you will use the time-series method called “ Range ”. See. Below is how to do it via Web UI and API.

Adding a time-series-request (range) label via Web UI

‍

Adding a time-series-request (range) label via API

curl -- location -- request POST 'https: //api-dev.headspin.io/v0/sessions/ {{session_id}} f/label/add ' \ -- data-raw ' {`` gens '': `` over 400mA '', `` start_time '': 0, `` end_time '': 30.0, `` method '': `` range '', `` time_series_key '': `` battery_current '', `` parameters '': {`` lower_limit '': 400, `` include_lower_limit '': true}} '

To get the lean of available values for the “ time_series_key ” parameter, call the API below to regain the schema.

curl https: // {{auth_token}} @ api-dev.headspin.io/v0/sessions/timeseries/ {{session_id}} /info

2. Once the ” time-series-request ” label is created, confirm that “ time-series-result” labels are mechanically generated in the places where the current value is greater than 400mA. You will see a result similar to the one below.

‍

3.Optional:Calculate the entire duration of the three labels circled red circle in the picture above. Call the to retrieve a group of labels under the same Label Group ID, iterate through them, and sum up the total duration of the labels - 57.17 seconds. Run the following API to add the session with the KPI value to your exploiter flow. Remember having done this for the “ Max Current ” KPI before?

Adding a custom KPI “ insufferable battery usage ” to a user flow via API

coil -X POST https: // {{auth_token}} @ api-dev.headspin.io/v0/perftests/upload \ -- data-raw ' {`` test_name '': `` Mutual Operation (REQ 013) '', `` session_id '': `` {{session id}} '', `` status '': `` passed '', `` data '': [{`` key '': `` unacceptable battery utilisation '', `` value '' :57.17}]} ’

The image below illustrates how the trend of the KPI values is display on Web UI. Without this visual aid, it can be difficult to spot or filter out session that might confuse your analysis. If you want to be alerted when the & quot; unacceptable battery usage & quot; exceeds a fixed threshold or visualize correlations with early KPIs that do not have time as their X-axis, use Grafana & # x27; s alerts and information analysis to complement your needs.

‍

4. Add a build tag (e.g. build: 3.0) to the session.

‍

5. Repeat steps 1 - 4 to generate 80 sessions. The next step assume you have a exploiter flow having 40 sessions with the “ build: 3.0 ” tag and the remaining 40 sessions with the “ build: 4.0 ” tag.

6. Call to generate a regression report. Web UI doesn ’ t allow you to configure the “ limit ” and “ kpi ” parameters, so the use of API is necessary here - the default values of & quot; limit & quot; and & quot; kpi & quot; parameters are 15 and “ summary.network.regression. ” Network Latency ” respectively. For Network Latency, we will appear deep in.

Run a fixation analysis via API

lock -- location -- request POST 'https: //api-dev.headspin.io/v0/regression ' \ -- data-raw ' {`` a '': `` build: 3.0 '', `` b '': `` build: 4.0 '', `` limit '': 40, `` threshold '': 10, `` kpi '': `` labels.\ '' time-series-result\ '' .categories.battery_current.regression.Duration '', `` user_flow_name '': `` Mutual Operation (REQ 013) ''} '

There are three point with this API:

Point 1: What is “ limit ”?

The image below depicts the construct of group ‘ a ’ and ‘ b ’ and the impact of the & quot; limit & quot; parameter (e.g. limit=10). Usually, the higher the bound size is, the more exact, but the longer it takes to produce the report. As a rough estimate, it takes about 25 seconds for a 250 boundary and 220 seconds for a 1000 bound.

‍

In this exemplar, Regression Intelligence pseudo-randomly picks up ten sessions per set (20 session full). The “ pseudo-randomly ” substance as long as the tag/selector and the sizing of the population remain unchanged, the like sample will be select each clip; if new sessions are added to ‘ a & # x27; or ‘ b & # x27;, the sessions selected will change, even if the same selector is specify. This prevents the encroachment of human bias in the selection process.

Point 2: What is & quot; kpi & quot; and where to find the path?

Regression Intelligence monitors the value set in the & quot; kpi & quot; parameter to determine the fixation status. Finding the “ kpi ” path value is tricky. You should render a dummy report with limit = 1 and determine the path value to the mark KPI. Refer to the JSON output in Step 7 to locate the path.

Point 3: What is the syntax to select ‘ a ’ and ‘ b ’ sessions?

Regression detection Begin by extracting two groups from the self-possessed sessions, so-called ‘ a ’ and ‘ b ’ (e.g. “ build: 3.0 ” vs “ shape: 4.0 ”). Either or selectors can be used to extract sessions. In general, you opt tags, which do not postulate special knowledge. However, selectors can be very useful for making complex selections that require coherent or regular expressions or selector functions. See below for some useful expressions and examples.

Table:

build: 0.1.0 + day_of_week: Mon

build: 0.1.0 day_of_week: Mon

start & # 95; timestamp & # 58;
& lt; 1653487200 & # 43;
& gt; 1652104800

build:
prev_build (1),
soma: prev_build (0)

day_of_week:
=Sat, =Sun

day_of_week:
=Sat, =Sun

build: 0.1.0! geo: `` Mountain View, US ''

`` geo: \ `` Palo Alto, US \ `` ''

build: > 1.7.0

duration_seconds:
& gt; 60 & lt; 120

build: & gt; 100
& # 40; & lt; 200, & # 61; 300 & # 41;

geo: *US

build: 0.4. *

device_sku:
iPhone*

hostname: * proxy- * -lin.headspin.io

device_sku:
=Chrome,
=Microsoftedge,
=Firefox,
=Opera,
=Safari

build:
current_build ()

build:
prev_build (1)

end_timestamp:
& lt; minutes_ago & # 40; 15 & # 41;
& gt; minutes_ago & # 40; 30 & # 41;

end_timestamp:
> hours_ago (1)

end_timestamp:
> hours_ago (168)

[{`` build '': `` 1.0 ''},
{`` home_wifi_speed '': `` 100mbps ''},
{`` hostname '': `` dev-au-mel-1-proxy-2-lin.headspin.io ''},
{`` month_in_year '': `` 4 ''},
{`` start_timestamp '': `` 1651236780.327 ''},
{`` team_name '': `` HS ''}, {`` user_email_address '': '' shinji @ headspin.io ''},
{`` user_flow '': `` Max_Evaluation2 ''},
{`` user_flow_id '': `` 4243d3f0-967a-4bc3-bdce-1cb5f8ce3375 ''},
{`` geo '': `` Melbourne, Australia ''},
{`` end_timestamp '': `` 1651236820.663 ''},
{`` carrier '': `` Telstra ''},
{`` datetime_local '': '' 2022-04-29T22:53:00.327 ''},
{`` datetime_utc '': `` 2022-04-29T12:53:00.327 ''},
{`` day_of_week '': `` Fri ''}, {`` demonstration '': `` regression ''},
{`` device_id '': `` 17160522501498 ''},
{`` device_sku '': `` Zebra TC56 ''},
{`` duration_seconds '': `` 40.336 ''}, {`` year '': `` 2022 ''}]

‍

7. The regression report is returned in JSON format that looks like the below. Note that it contains “ Count ” and “ Sessions ” in the “ battery_current ” category section. You can set regression alert on them. In the next, we will set alerts on the “ Duration ” value.

‍

8. Find the account link in the result and open it to view the study in Web UI. On the correct side of the image, the source of the & quot; unacceptable battery usage & quot; label is spotlight.

‍

9. Let & # x27; s pause for a moment and see the report together.

• For each of & # x27; a & # x27; and & # x27; b & # x27;, the duration of the “ battery unacceptable usage ” labels for 40 pseudo-randomly selected sessions are aggregated, and the mean value is calculated.
• The mean value of ‘ a ’ and ‘ b ’ (13623ms vs 19720ms) are compare, and the difference of 6097.2ms is computed.
• Since the value of ‘ b ’ (19.7s) exceeds 14.96s, which is 10 % of ‘ a ’ (13.6s), the “ status ” shows “ failed ”, meaning regression has occurred.
• This means that the battery usage for the new build can be considered importantly worse than the old build. To confirm fixation, you might scrutinize the KPI distribution chart (see Step 3). If no clew is found, it could be a concurrence or irregular. Since asignificance gradeof 5 % is often used in statistics, you might ignore it and supervise the position until adequate alerting are find and a significant departure is reassert. You may too increase the & quot; limit & quot; parameter or reviewpopulation sizeto improve your confidence level. This topic will be discussed in more detail in.

10. Create Watchers utilise. The witness unceasingly monitor the reports for you and alerts you when regression occurs.

Set a pro-active performance regression monitoring agent via API

curl -- location -- request POST 'https: //api-dev.headspin.io/v0/regression/alert/watcher ' \ -- data-raw ' {`` gens '': `` Max battery regression watcher '', `` a '': `` build: 1.0 '', `` b '': `` build: 2.0 '', `` kpi '': `` labels.\ '' time-series-request\ '' .categories.battery_current.regression.Duration '', `` threshold '': 10, `` frequence '': `` * /10 * * * * '', `` ts_start '': 1652086900, `` user_flow_name '': `` Max_Evaluation ''} '

Note that HeadSpin allows you to create alerts in Web UI but there is no battleground to limit the & quot; kpi & quot; value, we need to use API here. If your target KPI is the nonpayment one, meaning “ Network Latency ”, use Web UI as evidence below.

‍

In the examples above, I used the cron expression“ * /10 * * * * ”,which means that the regression monitoring should be performed every 10 minutes. See the table below for various oftenness settings.

Table: cron expression commonly utilize

‍

11. Let ’ s test if the fixation monitoring and alerting are working as await. When a study go, the result will be display in three different colours: Green (success), Red (failure), and Yellow (skip, e.g. due to not enough samples). When a regression is detected and settle, you will find a apprisal at the time show by the email icon.

‍

Be mindful that the name and settings can not be changed once an alert has been set up. Deleting an alert will result in deleting all consort alert story and reports, so be heedful.

12. Optional: With the “ divider ” feature, you can set multiple regression alerts in a individual setting. In the following example, see the “ device_sku ” tag is set to partition. The “ device_sku ” tag bear five values, each representing a particular device manufacturer, including Galaxy S9. Regression Intelligence monitors each gimmick ’ s & quot; impossible battery usage & quot; KPI and issues an alert when a device surmount the 10 % threshold. You can set any exploiter tatter to the partition - not limited to “ device_sku ”. Very powerful, isn ’ t it? For more details, hunt “ partition ” on.

‍

Last, you might watch the following short picture to solidify your apprehension of regression alerts.

‍

Conclusion

Phew… Reading this far, I trust you understand how regression monitoring and alerting work. In this article, we focused on the confusing aspects of Regression Intelligence, include customer KPIs, when and how to use Grafana and Watcher, interpret regression study, and set up alerts in a practical manner. All the indispensable fabric and references are in one place, so bookmark this page and use it as a handbook or share it with your team members to expedite knowledge sharing. In, we will look into more advanced issue, include network latency and aggregation, percentile, percentile rank and more. Understanding these concepts will help you solve regression job in wider situations. See you following time!

4 Parts

Connet Now