This post discusses how we organise our exploratory testing sessions at mabl.
You ’ re welcome to download ourtemplatefor our “ Elmer Fudd ” session:

Quality is like Mom and apple pie - everybody thinks it ’ s a good thing!

When your product is an machine-controlled testing instrument like mabl 's, the bar for caliber is high. Ensuring the quality of our package is a top anteriority for the team, especially since at the end of the day it helps our customers render better production to their customers. It ’ s nucleus to our culture and we 're constantly working on fashion to improve.

No affair what your domain is, many bringing teams struggle with the ostensibly competing goals of frequently delivering new features that solve new client problems, and maintaining a eminent level of product quality so customers don ’ t experience the pain from bugs or early matter. Like many of you, we have to act on balancing the needs of both as well. We endeavor to deliver new features and enhancements as quickly as possible without losing direction on the quality of our yield. It 's an ongoing process for our team that we are always looking for new fashion to better upon.

Our ware is evolving at a rapid pace. The team is act rattling quickly, releasing new feature and improving existing unity. For example, there were over 70 new enhancements released in the last 90 day. To keep up with the pace of development while also meeting the high standards for quality, we originate a program called Elmer Fudd - a nod to that legendary Orion of Bugs Bunny. We 've added more structure around involving more team members in explorative testing of the product from a user experience point of view. Though of mabl, use a good exploratory testing session before a build locomote live, to find the edge cases not continue by existing automated tests, aid us afford our customers a better overall experience. & nbsp; & nbsp;

A small experiment

Our quality catalyst, Bertold Kolics, suggested an experiment: have volunteer from respective teams, including field engineers from sales support, client support and success, UX designers, ware handler, and merchandising squad member, enter in a one-hour “ bug hunt ” before releasing a new feature to beta or product. This was easy to try. No special testing noesis was needed. We got together, with a couple of us enter remotely. We had engineer, product managers, support team members, and more. Through ad hoc and exploratory examination, we found many opportunities to improve the production or fix an issue before the feature went live.

Everyone agreed the Elmer Fudd sessions were productive, and decided to continue them as needed. The sessions were informal. People could ask each other questions and discourse how features should behave. Issues found be noted in a dedicated Slack channel. Engineers depart back through the channel later to mention and prioritise bug to be secure or new narrative to write for any absent functionality noticed.

The Elmer Fudd session proceed, mostly at least erstwhile per week. Some focused on screen new feature, while others focused on making sure there were no fixation failures due to a new change.

Reflecting and retell with new experiments

After a few weeks of give Elmer Fudd sessions, we did a quick retrospective on what was going well in the session and what could be ameliorate. We had observed that multiple citizenry were frequently testing the like piece of functionality, while other aspects of a feature might not get examine at all during a session. Despite using the Slack channel, sometimes detail discovered during the sessions were getting lost. & nbsp;

We came up with some new mind to try as experiments. One was to do more focused exploratory examination, using charter free-base on Elisabeth Hendrickson ’ s template:

Explore & lt; target - a characteristic, requirement, conformation ... & gt;

with & lt; resources, such as tools, data, a proficiency, another feature & gt;

to learn & lt; information - about scheme behavior, execution, security, accessibility ... & gt;

(from ExploreIt! Reduce Risk and Increase Confidence with Exploratory Testingby Elisabeth Hendrickson)

Another way to be more mastermind about who was testing what is have people pair up to help get multiple perspectives on each exploratory charter.

Before our succeeding Elmer Fudd session, Bertold shared info on how to write explorative testing charters, pertain to Elisabeth Hendrickson ’ sExplore It!book (which we obtained for our library). & nbsp;

Pro tip: Tools like SUSA can handle this autonomously — upload your app and get results without writing a single test script.

He created a document for planning the session. Engineers added links to the stories to be tested and notes about what involve screen. They created exploratory essay charters in a separate spreadsheet. Personally, I find writing charters can be challenging, which isn ’ t a bad thing. The thought operation, such as thinking of imagination to use for prove, brings up fresh testing ideas. I paired with a couple of engineers to help them get started write charter.

When we part the next Elmer Fudd session, participants paired up and assigned themselves a charter. We nevertheless shared topic found in Slack, as well as recording line in the charter spreadsheet. One of the technologist volunteered to remote pair with me. At the end of the session, we had a quick debrief. We felt good about the measure of testing accomplished. The engineer who paired with me said he thought we had a lot of good testing ideas that he wouldn ’ t have had by himself. Others agreed they experience similar welfare.

Anyone can mastermind an effectual bug hunt

Thanks to Bertold, we now have a pretty smooth operation so that anyone can organise an Elmer Fudd session. He created a template we can use to make a doc for each session. It include a checklist of activities to be done:

The Elmer Fudd session document has a subdivision to specify the test environs and mabl trainer variation (since we may be testing a arm) to be used in the session. The schedule for the session is also laid out in the doc.

The next section is “ What ’ s in this release candidate? ” People can add the tickets and PRs along with other notes about what to test. This is followed by a subdivision about the exploratory try charter, using the format shown above, and a link to a spreadsheet for explorative testing charters. We have a template for creating this spreadsheet in Google Sheets:

The document likewise has guidelines for the session. These support full conversations during the session and assist see all valuable information discovered during testing is entrance.

Report all potential issues in Slack on # elmerfudd:

• Over-communicate so that info captured on the channel can be used in the post-session effectively

• Post screenshots, videos, logs as needed

• Use threadsfor discussing a trouble to forfend overloading the independent channel

The leave session document is pinned to our # elmerfudd Slack channel. The meeting organizer schedules the session and sends an invitation to the team requesting the Elmer Fudd, and to managers both in and outside of Engineering. A general announcement also goes out on Slack, and anyone can request an invitation. & nbsp;

Outcomes

Our squad tracks many different metrics on production utilization and customer-reported bug. And there are many different efforts go on to improve our production ’ s quality and user experience. As with many modern complex, distributed systems, it can be difficult to pinpoint course and their cause. & nbsp;

Overall, there ’ s been a positive trend. The rate of bugs in production that require interference by engineers is down. Field engineers report few roadblocks as they work with trial customers.

With all the information gathered during the Elmer Fudd sessions, everyone is more open about what testing was execute. We feel more confident about making a decision to merge change and get them deploy or unloose in production.

We 're all getting better at writing exploratory testing charter. Our engineers are getting a fortune to hone their own examination skills even more, so they can do more examination as they are coding new changes. & nbsp;

An important welfare to the Elmer Fudd sessions is that sales, customer support, and marketing can participate, help with the testing, and get a glimpse of what ’ s about to be release and see how the product is acquire. We have various feature teams, and each one is continually releasing changes to different parts of the merchandise. Participating in the Elmer Fudd helps everyone stay informed, and get hands-on with the up-to-the-minute lineament.

As we test a new characteristic in the Elmer Fudd, the engineers and decorator who developed that feature are in the room to answer questions. This has ensured that citizenry across all team part the like understanding of the end of each new feature and how that lineament work.

The Elmer Fudd sessions keep lineament at the forefront of everyone ’ s mind. It ’ s leisurely to get heads down encrypt new features, and this has given us a hazard to tread backwards, looking at the big picture, and think deliberately about how client might use the new modification. No doubt, as we reflect and learn from our testing activities, we will continue to improve them and do an even better job of building quality into our product.

You ’ re welcome to download ourtemplatefor our “ Elmer Fudd ” session papers.