A Comprehensive Guide to Parallel Testing with CircleCI

Let‘s start with the basics – what exactly is parallel testing and why should you care?

Sequential testing runs tests one after another on a single machine. As your test suite grows, execution time increases linearly. Parallel testing splits tests to run simultaneously across multiple containers, dramatically reducing overall runtime.

By the end of this guide you will be able to:

• Explain core concepts like containers, executors and workflows
• Configure parallelism in CircleCI with containers
• Glob and split test suites for allocation across machines
• Debug and monitor parallel runs
• Integrate splits with test runners like Selenium and Jest

Prerequisites

To follow along, readers should have:

• Existing test framework & suite (Selenium, Unit Test, etc)
• CircleCI account with project set up
• Comfort with CLI usage
• Understanding of CI/CD concepts

This guide focuses specifically on CircleCI but the parallel patterns apply to any CI/CD tool like TravisCI, Jenkins, etc.

Parallel Testing Benefits

1. Reduced Testing Time

Say you have 1000 Selenium tests, averaging 1 minute each to run sequentially. That is over 16 hours of total execution time!

By splitting these tests across 5 parallel containers, you can reduce this to around 3 hours instead.

2. Increased Feedback Velocity

Failures are detected rapidly rather than waiting for full suite to run. No time wasted debugging already passed tests when running sequentially.

3. Test More Browsers

Allocate tests for different browsers and viewports across machines. Catch cross-browser issues early.

However, parallel testing has downsides:

• Flaky tests become harder to reproduce/debug
• Increased infrastructure costs with more containers
• Needs sufficient tests – overhead can outweigh benefits for smaller suites

Overall though, the speed and efficiency gains make it very compelling.

Setting Up Parallelism

The parallelism key in CircleCI config.yml defines number of containers to use:

# config.yml
version: 2
jobs:
  test: 
    docker:
      - image: cimg/node:16.13
    parallelism: 5 # use 5 containers

Typically you pay per container usage so be mindful of costs!

Now let‘s look at ways to allocate tests across the containers.

Splitting Tests Across Containers

CircleCI CLI provides glob + split to groups tests for parallel runs.

Glob Files for Pattern Matching

First we glob to filters test files by pattern matching:

circleci tests glob "tests/**/*.test.js"

Supported patterns:

• * wildcard matches any # of characters
• ** matches nested directories
• ? matches single character
• {foo,bar} alternate matches

Split Test Files

The split command divides tests across containers:

circleci tests glob "tests/*.js" | circleci tests split 

Split techniques:

• split-by=name: Default, alphabetic
• split-by=timings: Optimized evenness
• split-by=filesize: Distribute by size on disk

View current container out of total:

echo $CIRCLE_NODE_INDEX $CIRCLE_NODE_TOTAL

Integrating Test Frameworks

Let‘s walk through setting up parallelism with Selenium Java framework and CircleCI.

Project Structure:

/tests
  - LoginTests.java
  - HomePageTests.java

/config.yml:

  parallelism: 4
  steps:
    - run: 
        command: |
          mvn test 
          circleci tests glob "src/test/java/*.java" | circleci tests split --split-by=timings | xargs mvn test

This splits Java test classes across 4 machines running Maven test runner with timing based splitting.

For JavaScript tests using Jest, we glob all test files starting from project root before splitting:

circleci tests glob "**/*.test.js" | circleci tests split | xargs npx jest

Best Practices

Here are some tips for smooth parallel testing:

1. Review Test Dashboard Frequently

Actively monitor the CircleCI dashboard for failing tests on various containers. Fix flakiness early.

2. Organize Test Artifacts

Structure screenshots and videos by test class name so failures can be tracked to source.

3. Reproduce Issues Locally

Binary search for flaky tests by running a subset of parallel files sequentially.

4. Use Real Devices

While CI/CD environments have improved browser support, only real devices can fully replicate native conditions.

Consider services like BrowserStack to run UI and API tests across 3000+ real mobile devices and browsers on the cloud.

Key Takeaways

• Parallel testing splits test execution across containers
• Reduce overall runtime by running tests concurrently
• Use CircleCI CLI to optimally group tests files
• Integrate splits with all major test runners like Selenium and Jest
• Monitor dashboards closely and organize artifacts carefully

I hope this guide gives you a methodology for implementing performant parallel testing workflows with CircleCI! Let me know if you have any other questions.