How to run Parallel Tests with CircleCI

Testers oftentimes face large codebases with hundreds of tests that must run across different platforms, device, browsers, and operating systems. Running such a bulk of tests efficiently is only potential with the right approach: parallel examination. It ’ s the simplest way to handle program variability without bottlenecks, delays, or errors.

This article will research how to run parallel tests with a particularly democratic and useful CI/CD puppet like CircleCI.

What is Parallel Testing?

In, you test different modules or applications on multiple browser-device-OS combinations simultaneously rather than one after another. In a scenario where there are two versions of software available, and both must be examined to ensure stability and compatibility, it is easygoing to run tests on two version simultaneously, get results and detect bugs quicker

Parallel quiz reduces performance time and exploit, resulting in faster time to bringing. It is particularly useful in the case of,, fix, and.

Parallel CI Test Automation Workflow

Implementing parallel CI test mechanization workflow is essential to enhance the efficiency of pipelines. By dividing test suites into pocket-sized clump and running them parallelly across multiple environments, team can achieve quicker feedback, reduce bottlenecks, and overall improvement in productivity.

Steps to Implement Parallel CI Test Automation Workflow:

• Analyze Your Test Suite: Break down your test suite to identify slow-running and dependent tests. Categorize them establish on execution clip and priority.
• Enable Parallel Execution: Configure your CI pipeline to run multiple jobs in parallel. Most, like,, or GitHub Actions support this feature with minimum frame-up.
• Split Tests Dynamically: Use test splitting strategy (e.g., time-based, size-based, or file-based) to equally distribute tests across available environments. Tools like CircleCI & # 8217; s test splitting or custom scripts can automatise this.
• Optimize Environment Setup: Ensure test environments are consistent and lightweight for faster frame-up times. Tools like Docker or pre-built environments can aid streamline this process.
• Monitor and Fine Tune: Continuously monitor test execution clip, identify bottlenecks, and refine your splitting logic or infrastructure to maximize efficiency.

Benefits of Parallel CI Test Automation

Parallel CI test automation provides a clear itinerary to faster, more efficient development workflows. Key benefits include:

• Accelerated Test Execution: Running examination concurrently shortens overall build times, delivering flying feedback for developers.
• Scalability: Handle turn without bottleneck by distribute examination across multiple parallel environments.
• Efficient Resource Utilization: Make the most of available CI resources, ensuring machines are employ to their entire capacity.
• Other Bug Detection: Quickly surface issues by running tests on different environments simultaneously, trim the risk of hidden bugs.
• Enhanced Developer Productivity: Developers spend less clip wait for builds and can rivet on indite code, leading to faster delivery round.

Parallel CI automation directly contributes to higher-quality software and a more agile development procedure by enable faster, smarter, and more.

How to run Tests in Parallel with CircleCI

Before exploring parallel testing with CircleCI, it ’ s important to place a quick note on why it & # 8217; s important to run parallel trial on existent browsers and device.

While it may look easygoing to download and test on, they are riddled with inadequacies that forestall them from providing a reliable test environment. For example, they can not mimic low network conditions, low battery, or incoming calls. In general, they can not replicate all characteristic of existent browsers and devices, create the results of examination run on them prone to grave error.

Simply sign up for free, log in, select the browser-device-OS combinations to be tested on, give the relevant tests scripts and let the tests run themselves via mechanization.

Now, on to CircleCI.

CircleCI enables you to run tests in parallel, helping reduce test execution time.

By default, tests in CircleCI run on a single VM. As your test retinue grows, execution time increases. To rush this up, CircleCI allows you to distribute trial across multiple executors by setting a parallelism grade. This tells CircleCI how many executors to spin up for a job.

Once you set correspondence, you can split tryout files across executors using the CircleCI CLI or surroundings variable, control each machine runs a subset of the tests.

How to Set up Parallelism in CircleCI

Test entourage are defined at the job tier in the.circleci/config.ymlfile. Start by setting the parallelism level in the config file.

This correspondence key defines how many independent executors CircleCI will create to run your tryout. To do this, you just need to add this key before calling the steps to run your tests.

Depending on your CircleCI & # 8217; s pricing plan, the parallelism value can be increase to agree your requirements.

To run a job ’ s steps in latitude,set the parallelism key to a value greater than 1.

# ~/.circleci/config.yml version: 2 jobs: test: docker: - image: cimg/ & lt; language & gt;: & lt; version TAG & gt; auth: username: yourdockerhub-user password: $ DOCKERHUB_PASSWORD parallelism: 4

Now, in order to rive the real tests, the succeeding step is to use theCircleCI CLI (Command Line Interface)to phone extra commands to split the tests.

Example of Parallel Test Execution with CircleCI

Running examination in parallel using CircleCI helps speed up your CI/CD pipeline by divide your examination across multiple executors (machines).

Below is a step-by-step setup and explanation of how to configure CircleCI to run tests in parallel:

Step 1: Create or Modify .circleci/config.yml

In CircleCI, the constellation file (config.yml) defines how your job and workflow run. To set up execution, you involve to modify this file.

Here ’ s an exampleconfig.ymlfile that evidence parallel test performance:

variant: 2.1 jobs: test: docker: - image: cimg/python:3.8 # Use a Docker image with your test environment parallelism: 4 # Number of parallel executors (business) for this job step: - checkout - run: name: Install dependencies command: pip install -r requirements.txt - run: name: Run tests command: pytest tests/ # Run tests using pytest workflow: version: 2 test: chore: - test # Running the 'test ' job defined above

Step 2: Understanding the Configuration

The jobs section defines individual tasks in the pipeline, such as the examination job in this example.

1. Version and Jobs:

• The version:2.1indicates you & # 8217; re useCircleCI 2.1configuration syntax.
• The jobssection defines item-by-item tasks in the pipeline. Here, we have a job calledtest.

2. Docker Executor:

• This job habituate the Docker executor(docker),which specifies a Docker persona to run your examination. In this case, the picture used iscimg/python:3.8, which provides a Python 3.8 surroundings.

3. Parallelism:

• The parallelism: 4line limit the number of parallel instances for the job. Setting it to 4 instructs CircleCI to run four parallel jobs to execute the tests.

4. Steps:

• checkout:This step check out your codification from the depositary.
• run:These steps establish dependencies (pip install -r requirements.txt) and run the tryout expendpyteston the tests/ directory.

5. Workflows:

• The workflowssection delimit how occupation are orchestrate and accomplish. In this case, thetestworkflow extend thetest job.

Step 3: Enable Test Splitting (Optional)

For autonomous testing across multiple user personas, check out SUSATest — it explores your app like 10 different real users.

If you have a large number of tests, it can be beneficial to split them across the latitude jobs to ensure each job has an adequate load. CircleCI provides a built-in test-splitting feature to do this mechanically. You can split your tests up alphabetically (by name) (which is by nonpayment), by size, or by apply historic timing data.

Here ’ s how to enable test splitting:

task: exam: docker: - ikon: cimg/python:3.8 parallelism: 4 steps: - checkout - run: gens: Install dependance require: pip install -r requirements.txt - run: name: Split and run test in parallel command: | circleci tests split -- split-by=timing -- test-metadata=tests/test_metadata.json pytest tests/

The circleci trial splitcommand splits your test files based on timing information from previous footrace, so each job gets a rough equal number of tests.

Step 4: Push to CircleCI

Once you & # 8217; ve set up the config.yml, advertize your modification to your Git repository. CircleCI will automatically detect the configuration file and spark the grapevine.

Step 5: View Parallel Test Results

• After your tests run, you can see the results in the CircleCI dashboard. CircleCI will display the execution times and any test failure or successes across all parallel chore.
• You can also integrate with to streamline the planning, execution, and tracking of test instance. Additionally, offers a centralised survey of your examine attempt by consolidate the results of all parallel jobs to enhance debugging.

Talk to an Expert

How to split test files utilise CircleCI CLI

Splitting tests is an important practice to improve the efficiency of your. By breaking your test entourage into smaller, more manageable portion, you can take advantage of parallel executing to speed up testing and streamline your development process.

CircleCI supports robotic examination allocation across your container. The allocation is based on filename or class gens, look on the prerequisite of the test-runner used. It require the CircleCI CLI, which is automatically injected into your build at run-time.

The commands that will be employ for splitting the examination files areglob and split.

• A glob is basically a pattern that will match a list of filename to a specific figure that you set.
• The split command, by nonremittal, will split your examination harmonize to a list of filenames, but you can besides use other strategies such as splitting it by timing data or file size.

By using these two commands, you can split your trial into multiple executors that are self-governing of each other as you can see from the image above.

Globbing the trial

In order to aid in defining your test entourage, the CircleCI CLI endorse globbing trial files habituate the following patterns:

• * lucifer any sequence of quality (excluding path separators)
• * * check any sequence of characters (including way separators)
• ? matches any single character (exclude path separator)
• [abc] matches any fiber (excluding path separators) against quality in bracket
• {foo, bar, & # 8230;} matches a sequence of lineament if any of the alternative in braces matches

So to glob trial files, pass one or more practice to the CircleCI examination glob command.

Next, check out how to implement this in the configuration file.

circleci tests glob `` tests/unit/ * .java '' `` tests/functional/ * .java ''

You can use the glob dictation in your config file with the echo command as shown below, to check the results of pattern match:

# ~/.circleci/config.yml variation: 2 jobs: test: docker: - image: cimg/ & lt; language & gt;: & lt; version TAG & gt; auth: username: mydockerhub-user password: $ DOCKERHUB_PASSWORD # context / labor UI env-var mention parallelism: 4 steps: - run: bid: | echo $ (circleci tests glob `` foo/ * * / * '' `` bar/ * * / * '') circleci tests glob `` foo/ * * / * '' `` bar/ * * / * '' | xargs -n 1 replication

Splitting the tests

The split command, by nonremittal, will rive your tests according to a list of filenames but you can too use early strategies such as splitting it by timing data or file size. Below are all the splitting technique.

Split By Name

This is the default splitting technique. So in case you don ’ t specify a method utilise the& # 8211; split-byfleur-de-lis, CircleCI expects a list of filenames/classnames and split them alphabetically by test name.

There are a few ways to provide this listing:

• Create a text file with trial filename

circleci tests split test_filenames.txt

• Provide a path to the examination files

circleci tests split & lt; /path/to/items/to/split

• Pipe a lump of test files

circleci tests glob `` test/ * * / * .java '' | circleci tests split

Once the list of names has been provided, CLI seem up the number of available containers, along with the current container index, and so it uses the deterministic splitting algorithm to split the exam files across all available containers.

Split By Time

This is the good way to optimize your test suite across a set of parallel executors. This ensures that the exam are split in the evenest way, direct to overall reduced test time.

In order to split by test timings, use the& # 8211; split-byflag with the timing break type. The available timings datum will then be canvas and your tests will be break across the parallel-running containers as equally as possible.

circleci tests glob `` * * / * .go '' | circleci tryout split -- split-by=timings

On each successful run of a test suite, the timings datum is saved from the directory determine by the route in thestore_test_resultsstep. This timings data mentions how long each tryout direct to discharge per filename or classname, depending on the language employ.

One thing to be considered is if you do not usestore_test_results, there will be no timing data available for splitting your tests.

The CLI expects both filename and classnames to be present in the timing data produced by the test suite. By default, splitting nonpayment to the filename, but you can specify classnames by using the& # 8211; timings-type flag.

cat my_java_test_classnames | circleci tests split -- split-by=timings -- timings-type=classname

For partially institute test results, you can mechanically assign a random small value to any test you did not find timing data for. You can override this assigned value to a specific value with the& # 8211; time-default flag.

circleci tests glob `` * * / * .rb '' | circleci trial rive -- split-by=timings -- time-default=10s

Additionally, If you need to manually store and find timing data, use thestore_artifacts step.

Splitting by filesize

You can also split tests by filesize. To do this, use the& # 8211; split-byflag with the filesize split type.

circleci tests glob `` * * / * .go '' | circleci trial break -- split-by=filesize

How to split tests using environment variable

CircleCI ply us with two surround variables that can be used in spot of the CLI to configure each container individually.

They are CIRCLE_NODE_TOTAL and CIRCLE_NODE_INDEX.

• CIRCLE_NODE_TOTALis the total bit of parallel container being used to run your job.
• CIRCLE_NODE_INDEXis the index of the specific container that is currently running. The surround variables ensure us full control over parallelism.

How to run split tests

Now after globbing and splitting the tests, it & # 8217; s clip to run them. To unite exam grouping with test execution, you can consider saving the grouped tests to a file and so passing this file to your test runner.

circleci tests glob `` test/ * * / * .rb '' | circleci tests separate & gt; /tmp/tests-to-run bundle exec rspec $ (cat /tmp/tests-to-run)

The content of the file/tmp/tests-to-runwill be different in each container, establish on$ CIRCLE_NODE_INDEXand $ CIRCLE_NODE_TOTAL.

Applying test parallelism using CircleCI regardless of your underlying technical approach is constantly beneficial because it will speed up your overall execution time and in some cases, can even reduce the time by half.

Utilize the measure in this article to run parallel tests in CircleCI, thus getting quicker results and integrating speed into your CI/CD line with simpleness.

Test Reporting & amp; Management

Test reporting and management are key components of any CI/CD pipeline, ensuring that test results are easily accessible and actionable. When extend, especially in parallel, managing and consolidating the results can be challenging. This is where BrowserStack Test Management get in, offering a streamlined way to manage and report test solvent.

BrowserStack Test Management enables the integration of automated test run solvent through:

• Uploading JUnit-XML or BDD-JSON reports generated by TestNG utilize simple bash bid.
• via the BrowserStack SDK.

This ensures that your test results are automatically pushed to BrowserStack, providing a coordinated view of all your test executions.

Steps to Generate and Upload JUnit-XML Reports with TestNG:

1. Execute TestNG Test Cases: First, you can initiate your TestNG projection and execute test cases.

Here & # 8217; s an example of a basic TestNG test case:

parcel com.browserstack; import org.testng.Assert; import org.testng.Reporter; import org.testng.annotations.Test; public class BStackDemoTest {@ Test populace nullity testUpdateUserEmail () {Reporter.log (`` [[PROPERTY|id=TC-1]] \n '', true); // Test logic} @ Test world void addToCart () {Reporter.log (`` [[PROPERTY|id=TC-2]] \n '', true); // Test logic} @ Test public void checkout () {Reporter.log (`` [[PROPERTY|id=TC-3]] \n '', mistaken); // Test logic}}

2. Generate JUnit-XML Reports: TestNG mechanically generates JUnit-XML reports in the test-output folder after test executing.

Here & # 8217; s a sample JUnit-XML study:

& lt;? xml version= '' 1.0 '' encoding= '' UTF-8 ''? & gt; & lt; testsuite hostname= '' Sh * * * * * '' failures= '' 0 '' tests= '' 3 '' name= '' com.browserstack.BStackDemoTest '' time= '' 0.007 '' errors= '' 0 '' timestamp= '' 2023-04-20T10:46:28 IST '' skipped= '' 0 '' & gt; & lt; testcase classname= '' com.browserstack.BStackDemoTest '' name= '' testUpdateUserEmail '' time= '' 0.000 '' / & gt; & lt; testcase classname= '' com.browserstack.BStackDemoTest '' name= '' addToCart '' time= '' 0.006 '' / & gt; & lt; testcase classname= '' com.browserstack.BStackDemoTest '' name= '' checkout '' time= '' 0.001 '' / & gt; & lt; /testsuite & gt;

3. Upload the JUnit-XML Report to BrowserStack: To upload the generated trial report to BrowserStack Test Management, you need to set up the necessary environs variables in your terminus:

export TEST_MANAGEMENT_API_TOKEN= '' * * * * * * * * * * * * * 28a42 '' export TEST_MANAGEMENT_PROJECT_NAME= '' & lt; Project Name & gt; '' exportation JUNIT_XML_FILE_PATH= '' & lt; Report Path & gt; '' export TEST_RUN_NAME= '' & lt; Test Run Name & gt; ''

Once the variables are set, you can upload the report habituate a simple bash command with curl:

scroll -k -X POST https: //test-management.browserstack.com/api/v1/import/results/xml/junit \ -u $ TEST_MANAGEMENT_API_TOKEN \ -F project_name= '' $ TEST_MANAGEMENT_PROJECT_NAME '' \ -F `` file_path= @ $ JUNIT_XML_FILE_PATH '' \ -F test_run_name= '' $ TEST_RUN_NAME ''

After the upload completes, you will incur a confirmation message along with a URL to access the test run report:

{`` message '': '' File uploaded successfully. ``, '' url '': '' https: //test-management.browserstack.com/projects/ & lt; project id & gt; /test-runs/ & lt; test run id & gt; '', '' success '': true}

4. View Test Results in BrowserStack Test Management: Once the test account is uploaded, you can log in to & amp; view the results.

Navigate to your project, click on & # 8220;Test Runs& # 8221; and open the test run yield from your machine-driven test performance.

All the test cause and their various termination will be expose hither, making it leisurely to analyze, debug, and collaborate.

Conclusion

Applying test parallelism with CircleCI is a powerful way to raise the efficiency of your testing process. By escape multiple exam parallelly, you can significantly reduce the execution clip, sometimes reduce it in half, allowing for faster feedback and quicker releases. Integrating parallel tests into your CI/CD pipeline not solely speeds up the development cycle but besides ameliorate resource employment and scalability.

By postdate the steps in this clause, you can easily set up parallel testing in CircleCI & amp; integrate it with BrowserStack to view results in a centralised dashboard. BrowserStack Test Management provides a seamless & amp; intuitive way to track, analyze, and collaborate on tryout results. BothBrowserStack Test Management and BrowserStack Test Observabilityfeature aboriginal integration, and for effective debugging, you can use Test Observability. This will help optimize examine workflows, leading to faster and more reliable software delivery.

Frequently Asked Questions

1. What is the concurrency bound in CircleCI?

The concurrency in CircleCI is the bit of tasks that are being executed at any point in time. For example, CircleCI ’ s complimentary program provides a concurrence boundary of 30, meaning you can run up to 30 tasks at the same time.

2. What is the maximum runtime for CircleCI?

The maximum runtime bet on the plan you & # 8217; re using. Jobs have a maximum runtime of 1 (Free), 3 (Performance), or 5 (Scale) hours depending on the pricing plan.

Related Guides