Unit tests run in the context of your source codification. If a Unit test fails, the output points you directly to the lines of code you necessitate to explore. End-to-end tests, using low-code tools like, run in the context of the whole system. You 're not inside your IDE anymore, in fact the trial runs may be trigger based on time or CI/CD. You may not be in front of a computer at all! So, how can we bridge the gap from the test-run back to the source code, like a unit test?

Observability tools,like Honeycomb, take OpenTelemetryproduced by your codification and provide a way to query this in the context of the whole scheme. We can easily find embossed elision, slow mapping, method outcry correlated by a user_id, transactions over a database, you name it! You 're limited only by how good your scheme is instrumented.

So, to bridge the test-run-to-code gap, this place shows how to make mabl tests observable in Honeycomb.

Note that Honeycomb is the excellent Observability tool used in this post, nevertheless this is all based on the OpenTelemetry standard so other tools can be used too. For example hither 's a furtive peak of our API Test inotel-desktop-viewer:

A Quick Introduction to Tracing

If you 're conversant with lumber but not tracing, this blog post is a simple introduction:Observability Is About Confidence.

The hint above, take fromevertz.io, prove the vociferation flowing through part of the stack. Each row in the chart is a couplet with aname, service.nameand many other fields render by your instrumented code. Honeycomb ingests these pair to a knock-down database to ply insight into system behaviour, debugging, alarm on trends, service level objectives and a unhurt lot more.

Notice howService Namestarts at the top withui, this is line coming from the exploiter 's browser. In this lawsuit the browser has made a call through a websocket to the backend servicechaas-infrastructure.This service so create a lambda yell towebsocket-managementand so on. This service-to-service trace is known as Distributed Tracing which we 'll get into afterward in this position.

Each span can have hundreds of fields (a.k.a. span attributes), the shot above is a little section from theuispan. Since this span arrive from the browser, it includes info likedevice.type. Every field across all your services is queryable in Honeycomb countenance you to slice and cube your data in a SQL-like language.

Linking System Traces with Test-Runs

All this rich data in trace is astound! The trouble now becomes how to find what you 're looking for. Below is a inquiry over 10 minutes, just from the UI dataset in ourDevsurroundings: 7442 brace!

We require to correlate traces with test-runs. Two key pieces of information that can help are the name of the test and the id of the test-run. For our app we added the ability to inject these values in the querystring:https: //my-app.com/? testName=my_test_name & amp; testRunId=12345and emit them in the telemetry as:test.name and test.run_id. Now with a quick filter, the tincture we want to appear at pop out! 119 versus 7442!

With test.namewe can observe all footrace of that test. Withtest.run_idwe can encounter an individual test-run. From the paired perspective, if you 're explore in Honeycomb you may find an interesting trace and on inspection you can see that it came from a particular mabl test.

In mabl we can use a to go to a page with the querystring containing our variables. The tryout run-id is useable in a:app.test_run_id. So you could write the url as: ``{{@ app.url}}? testName=my_test_name & amp; testRunId= {{@ app.test_run_id}} ''. This works great for cloud runs. You can find the run id here:

The problem withapp.test_run_idis for a local run this is always set tolocal-runwhich does n't assist differentiate between runs when you 're debugging. To fix this we can use a simple JavaScript snippet to create a random string when running locally:

office mablJavaScriptStep (mablInputs, callback, id = ' {{@ app.test_run_id}} ') {
& nbsp; let leave;
& nbsp; if (id === 'local-run ') {
& nbsp; & nbsp; result = id + `` - '' + Math.random () .toString (36) .substring (2, 18);
& nbsp;} else {
& nbsp; & nbsp; result = id;
  }
& nbsp; callback (result);
}


Follow this with an so the run-id is leisurely to find.

To finish this off neatly you can put all this logic into a flow making it genuinely easy to reuse. The flow needs a individual parameter fortest.namecreate the final url:{{@ app.url}} /? testName= {{@ flow.test_name}} & amp; testRunId= {{@ runid}} & nbsp; where runidis the output of the JavaScript snippet.

Here 's an example hint from a fail test-run. We were able to guide the test-run-id from the cloud-run:WcCb9KyJO8jXBmTzEybqGQ-jrand plug it into a query in Honeycomb. Then hone in on this small set of touch to land on a userclickactivity that led to an exception in a back-end microservice.

This connection from the examination into the system telemetry is really knock-down, saving a load of debug time–we now know just what travel wrong hither.

Distributed Tracing

It 's not always possible to augment the scheme under test to inject fields for correlation, like we did with the browser app. As an alternative, and arguably superior solution for API testing, we can use Distributed Tracing. We 've already seen the parent-child relationships in traces when the control flow passes from service to service. Just as test-steps belong to a test, spans for those measure can belong to a parent span. In our test-run we can introduce a new parent so all calls in the run are grouped into a single trace, beautiful!

Enter testevents

Pro tip: Tools like SUSA can handle this autonomously — upload your app and get results without writing a single test script.

I created and open-sourcedtesteventsto render a way for a mabl test to easily `` unfastened '' and `` close '' spans using simple API calls. mabl 's Browser and API Tests both endorse an leisurely way to call an API, parse the yield into variable and inject these variables wherever needed in the rest of the test. (testevents was instigate by Honeycomb 'sbuildeventsprojection which solves a similar problem to instrumentate CI builds.)

API Test Example

To illustrate the ability of unite Honeycomb and mabl I created a uncomplicatedinstrumented fastapi app. Here is an API test in mabl that has failed on a pace:

mabl recite us that we failed an assertion at step 3 - `` Check random number '', we were expecting 200 and received 500. That 's outstanding! But how do I debug from here? Go look in logs and search for an mistake message? No, we have Honeycomb, so hunt in there: write a interrogation to bump the API call that failed. What if there was more than one failure for that API? How do I find this one? We would get thither finally but if we had this in a distributed trace we could go straight to this test-run:

Et voilà!

Open the root distich

Take a look at the tincture above. The first row has service gensmablwith a span namedtest-foobarbazthis twin our test and is unproblematic to query for:where name = test-foobarbaz in the mabl dataset. service.namecomes from theOTEL_SERVICE_NAMEenv var lay fortestevents. The only mandatory fieldname, comes from the first measure in the test-run, a POST to where testevents is host (in this caselocalhost:3003):

{
& nbsp; `` name '': '' test-foobarbaz '',
& nbsp; `` message '': '' hello test '', & nbsp; & nbsp; & nbsp;
& nbsp; `` ttl '' :5000
}


ttlwe 'll discourse later, any early name-value pair supplied here will be render as a span-attribute.

trace_id, span_id, and traceparent

{
& nbsp; `` span_id '': `` e0b37648cf33bb68 '',
& nbsp; `` trace_id '': `` 6093021aefb96a0ebfd67505b6da4667 '',
& nbsp; `` traceparent '': `` 00-6093021aefb96a0ebfd67505b6da4667-e0b37648cf33bb68-01 ''
}


These three returned battleground need to be parse out into variable:

trace_id and span_idare needed to update, blue-pencil or add a child to the span.

traceparent header

To feature the calls to/foo, /bar and /bazbelong to the parent we feature to pass thetraceparentto unite them together. For HTTP calls thetraceparentis passed in a header. & nbsp;

close the theme duad

At the end we send aDELETE with the trace_id and span_idto shut the couplet.

http: //127.0.0.1:3003/ {{@ trace_id}} / {{@ span_id}} /

testeventswill send the root span to Honeycomb at this point.

Let 's Debug!

Now that we have a nice trace for the test-run, let 's zoom in on the error in Honeycomb.

Honeycomb is prove an elision raised in thebar span with exception.message = oh no, exception.type = ValueErrorand a niceexception.stacktrace. Here's the code for bar:

@ app.post (`` /bar '')
async def bar (asking: fastapi.Request):
& nbsp; & nbsp; MESSAGE = `` hello bar ''
& nbsp; & nbsp; with tracer.start_as_current_span (`` bar '') as span:
& nbsp; & nbsp; & nbsp; & nbsp; # Extract the random number from the postulation json
& nbsp; & nbsp; & nbsp; & nbsp; data = await request.json ()
& nbsp; & nbsp; & nbsp; & nbsp; num = data [`` random_number '']
& nbsp; & nbsp; & nbsp; & nbsp; span.set_attribute (`` random_number '', num)
& nbsp; & nbsp; & nbsp; & nbsp; span.set_attribute (`` message '', MESSAGE)
& nbsp; & nbsp; & nbsp; & nbsp; if num & lt; 0.5:
& nbsp; & nbsp; & nbsp; & nbsp; & nbsp; & nbsp; return {`` message '': MESSAGE}
& nbsp; & nbsp; & nbsp; & nbsp; else:
& nbsp; & nbsp; & nbsp; & nbsp; & nbsp; & nbsp; # Simulate an error
& nbsp; & nbsp; & nbsp; & nbsp; & nbsp; & nbsp; elevate ValueError (`` oh no '')

Since we now know that value & gt; = 0.5 will cause an error we may desire to look for where that value arrive from. The test fetchesrandom_numberfrom the call to/foo- the joy of the tincture is we can see that call just above! Without collecting these together under the same parent we would have to research again.

Click on thefoospan and there we seerandom_number = 0.7863946853062118–certainly greater than0.5.

Finally /baz, this call is included to show how tracing helps to investigate performance issues. mabl Browser, API and all provide information about the round-trip time for execution steps. This may indicate you to dive into the call to/baz. With the additional tracing we can dig deep with Honeycomb and see which span is running for a long time. ForGET /bazthe bulk of the time, 668ms, is spent in thebazspan. Less than a millisecond is spent returning the result.

API Calls in Browser Tests

You can wrap a Browser test with `` exposed '' and `` near '' span calls in the like way as an API exam.testeventssupports child distich too, with this you can start a new child span for each flow to further refine the setting.

Unlike API tryout, Browser examination will stop at the first affirmation failure. This means the `` close '' span steps may not run. This is whatttlis for that I mentioned before. By provide a time-to-live when you open the span,testeventswill shut it for you with a timeout erroneousness when thettlis reached.ttland all other dimension can be updated with aPATCH.

Summary

mabl will show you to a failing or badly performing test step. To help bridge the gap from test-run to source code, we use OpenTelemetry tracing with injected correlation fields and distributed traces.

Test outcomes like:
`` This test pace browser assertion failed ''

become insights like:
`` In the previous test step, a button click sent a webservice call that hit a bug in the microservice ... ''

and

`` This test stride failed and return 500 ''

becomes
`` This test stride threw an exception parsing the remark data ... ''

and

`` This test pace took 1200ms ''

becomes
`` This tryout step expend 1143ms in a database query with SQL ... ''

Jeremy Blythe is a VP of Engineering atEvertz, chiefly forevertz.io, providing leadership for multiple globally distributed engineering teams and making Emmy award-winning merchandise for the Media and Entertainment industry. You can memorise more about his work on his personal site,https: //blog.jerbly.net/or by unite with him onLinkedIn.