How to Reproduce and Fix Flaky Tests

A flaky test is a test that passes and fails non-deterministically on the same code. It is not a test that fails because of a bug; it is a test that fails because of entropy. The cost compounds: devel

Why Flaky Tests Destroy Velocity

A flaky test is a test that passes and fails non-deterministically on the same code. It is not a test that fails because of a bug; it is a test that fails because of entropy. The cost compounds: developers stop trusting CI, they rerun pipelines until green, real regressions slip through, and the test suite becomes a liability instead of a safety net.

The industry data is consistent. Google reports that 84% of their test failures are flakes. Microsoft found that flaky tests consume 20–30% of CI capacity. At scale, a 2% flake rate across 5,000 tests means 100 spurious failures per run. If each investigation takes 15 minutes, that's 25 engineering hours wasted daily.

The fix is not "rerun until pass." The fix is understanding the mechanics of non-determinism, building tooling to surface it, and applying targeted remedies per root cause. This guide covers the full lifecycle: taxonomy, reproduction, diagnosis, remediation, and governance.

---

Anatomy of a Flake: Root Cause Taxonomy

Every flake falls into one of six categories. The category dictates the reproduction strategy and the fix.

Timing / Async Waits: The Most Common Offender

Modern apps are asynchronous by default. React suspense, Android Jetpack Compose recomposition, iOS SwiftUI state updates — all introduce frames between action and observable result. A test that clicks and immediately asserts fails when the event loop yields.

# Flaky: assumes synchronous DOM update
def test_add_to_cart(page):
    page.click('[data-testid="add-to-cart"]')
    assert page.locator('[data-testid="cart-count"]').inner_text() == "1"

# Fixed: waits for the *effect*, not the action
def test_add_to_cart(page):
    page.click('[data-testid="add-to-cart"]')
    expect(page.locator('[data-testid="cart-count"]')).to_have_text("1", timeout=5000)

The pattern: wait for the observable consequence, not the trigger. Use polling assertions (expect in Playwright, WebDriverWait in Selenium, waitFor in Testing Library) with generous but bounded timeouts. Never sleep. Sleep is a flake factory: too short fails under load; too long wastes CI minutes.

Shared State Pollution: The Silent Killer

State leaks through every layer: database transactions, browser storage, in-memory caches, singleton services, module-level variables, filesystem temp directories.

// auth.service.ts — singleton with mutable state
@Injectable({ providedIn: 'root' })
export class AuthService {
  private currentUser: User | null = null; // DANGER: persists across tests
  
  setUser(user: User) { this.currentUser = user; }
  getUser() { return this.currentUser; }
  logout() { this.currentUser = null; }
}

If Test A logs in a user and Test B expects an unauthenticated state, Test B fails — but only when run after Test A. The fix is test-scoped dependency injection or explicit teardown.

// test-setup.ts
beforeEach(() => {
  TestBed.resetTestingModule(); // destroys singleton instances
  TestBed.configureTestingModule({ providers: [AuthService] });
});

For database state: wrap each test in a transaction and roll back. For localStorage/IndexedDB: clear in beforeEach. For filesystem: use unique temp directories per test (tmpdir fixture in pytest, testdir in Vitest).

Test Order Dependence: Implicit Coupling

Order dependence is shared state pollution with a specific trigger: test sequencing. The canonical test is pytest --randomly-seed=1234 or jest --shuffle. If the suite passes in default order but fails shuffled, you have order dependence.

# Find the minimal failing subset
pytest --forked --randomly-seed=42 --maxfail=1 -x
# Then bisect
pytest --forked --randomly-seed=42 --collect-only | head -20 > order.txt
# Use pytest-testmon or custom binary search to isolate the pair

Automate this. A nightly job that runs the suite in 10 random orders and reports flakes is worth its weight in gold.

Network / External Dependency: The Uncontrollable Variable

Third-party APIs — payment gateways, geocoding, email providers, OAuth — introduce latency, rate limits, schema drift, and outright outages. The only robust strategy: contract testing with recorded interactions.

# pact/consumer/user-service.json
{
  "interactions": [{
    "description": "get user profile",
    "request": { "method": "GET", "path": "/api/v1/users/42" },
    "response": {
      "status": 200,
      "headers": { "Content-Type": "application/json" },
      "body": { "id": 42, "name": "Alice", "email": "alice@example.com" }
    }
  }]
}

Replay via WireMock, MockServer, or Playwright's page.route with HAR files. Never hit real external services in CI. For end-to-end smoke tests that *must* hit production, tag them @smoke, run them separately, and accept they will flake — but don't let them block merges.

Animation / Rendering Race: The Visual Flake

Headless browsers often skip GPU compositing. Animations that run at 60fps in headed mode may complete in 0ms headless, or vice versa. Coordinate-based clicks (page.mouse.click(x, y)) fail when layout shifts.

// Flaky: coordinate click
await page.mouse.click(400, 300);

// Fixed: semantic selector + actionability wait
await page.getByRole('button', { name: 'Submit' }).click();

// For canvas/WebGL: wait for requestAnimationFrame cycle
await page.waitForFunction(() => window.__APP_READY__ === true);
await page.evaluate(() => new Promise(r => requestAnimationFrame(r)));

Disable animations in test config globally:

/* test-globals.css */
*, *::before, *::after {
  animation-duration: 0s !important;
  transition-duration: 0s !important;
  caret-color: transparent !important;
}

// playwright.config.ts
use: {
  extraHTTPHeaders: { 'Accept-Language': 'en-US' },
  locale: 'en-US',
  timezoneId: 'UTC',
  // Force reduced motion at OS level
  launchOptions: {
    args: ['--force-prefers-reduced-motion=reduce']
  }
}

Time / Randomness / Environment: The Hidden Inputs

Date.now(), Math.random(), crypto.randomUUID(), Intl.DateTimeFormat, timezone, locale, CPU core count, available memory — all are implicit inputs to your tests.

// Flaky: depends on wall clock
test('shows "Posted 5 minutes ago"', () => {
  const post = createPost({ createdAt: Date.now() - 5 * 60 * 1000 });
  render(<PostCard post={post} />);
  expect(screen.getByText('5 minutes ago')).toBeInTheDocument();
});

// Fixed: inject time
test('shows relative time', () => {
  const now = new Date('2024-01-15T12:00:00Z').getTime();
  vi.setSystemTime(now); // vitest/jest time travel
  const post = createPost({ createdAt: now - 5 * 60 * 1000 });
  render(<PostCard post={post} />);
  expect(screen.getByText('5 minutes ago')).toBeInTheDocument();
});

# pytest: deterministic UUID and random
import uuid, random
from unittest.mock import patch

@pytest.fixture(autouse=True)
def deterministic_randomness():
    with patch('uuid.uuid4', side_effect=lambda: uuid.UUID(int=next(_uuid_counter))):
        with patch('random.random', return_value=0.5):
            with patch('random.randint', return_value=42):
                yield

_uuid_counter = iter(range(1_000_000))

Run CI in containers with fixed resources (--cpus=2 --memory=4g) to eliminate "works on my 16-core machine, fails on 2-core CI runner" class flakes.

---

Building a Reproducible Test Matrix

You cannot fix what you cannot reproduce. A test matrix defines the dimensions of non-determinism you will exercise systematically.

Implementing the Matrix in CI

Don't run the full matrix on every PR — it's too slow. Use a tiered approach:

# .github/workflows/test.yml
jobs:
  test-fast:
    name: Fast Matrix (PR)
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
        order-seed: [0, 42, 123]  # 0 = default, others = random
        animation: [disabled]
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright test --shard=${{ matrix.shard }}/4 --seed=${{ matrix.order-seed }}

  test-comprehensive:
    name: Comprehensive Matrix (Nightly)
    runs-on: ubuntu-latest
    if: github.event_name == 'schedule'
    strategy:
      fail-fast: false
      matrix:
        include:
          - { network: '3g', cpu: 4, locale: 'de-DE', tz: 'Europe/Berlin' }
          - { network: 'chaos', cpu: 10, locale: 'ja-JP', tz: 'Asia/Tokyo' }
          - { network: 'offline', cpu: 1, locale: 'ar-SA', tz: 'Asia/Riyadh' }
          # ... 20 more combinations
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: |
          npx playwright test \
            --network=${{ matrix.network }} \
            --cpu-throttle=${{ matrix.cpu }} \
            --locale=${{ matrix.locale }} \
            --timezone=${{ matrix.tz }}

The fast matrix catches 80% of flakes (order, animation, seed) in 10 minutes. The comprehensive matrix catches the rest (network, locale, CPU) overnight.

---

Manual Reproduction Techniques

Before automating, you must reproduce locally. The goal: reduce a flaky test to a minimal, deterministic failure case.

1. The Rerun Loop

# Run a single test 100 times, stop on first failure
for i in {1..100}; do
  echo "Run $i"
  npx playwright test tests/checkout.spec.ts --reporter=line || { echo "FAILED on run $i"; exit 1; }
done

Add --retries=0 to disable internal retries. You want the raw flake.

2. Seed Control

# Playwright: fixed seed for test order
npx playwright test --seed=12345

# Jest: fixed seed for module order
npx jest --seed=12345

# Pytest: fixed randomization seed
pytest --randomly-seed=12345

If the test fails with seed 12345 but passes with 0, you have order dependence. Bisect to dependence on a specific predecessor.

3. State Snapshotting

For shared state flakes, snapshot global state before and after each test.

# conftest.py
import gc, sys, threading
from _pytest.fixtures import FixtureRequest

def snapshot_state():
    return {
        'modules': set(sys.modules.keys()),
        'threads': {t.ident: t.name for t in threading.enumerate()},
        'gc_objects': len(gc.get_objects()),
        'env': dict(os.environ),
    }

@pytest.fixture(autouse=True)
def detect_state_leaks(request: FixtureRequest):
    before = snapshot_state()
    yield
    after = snapshot_state()
    
    leaked_modules = after['modules'] - before['modules']
    leaked_threads = set(after['threads']) - set(before['threads'])
    obj_growth = after['gc_objects'] - before['gc_objects']
    
    if leaked_modules or leaked_threads or obj_growth > 1000:
        pytest.fail(
            f"State leak in {request.node.name}:\n"
            f"  Modules: {leaked_modules}\n"
            f"  Threads: {leaked_threads}\n"
            f"  Object growth: {obj_growth}"
        )

4. Network Chaos Injection

# Start toxiproxy
docker run -d -p 8474:8474 -p 8666:8666 ghcr.io/shopify/toxiproxy

# Create a proxy for your API
toxiproxy-cli create upstream --listen 0.0.0.0:8666 --upstream api.example.com:443

# Add latency, jitter, bandwidth limit, close connections
toxiproxy-cli toxic add upstream --name latency --type latency --attribute latency=2000 --attribute jitter=500
toxiproxy-cli toxic add upstream --name bandwidth --type bandwidth --attribute rate=50  # KB/s
toxiproxy-cli toxic add upstream --name close --type close --attribute probability=0.1

Point your test base URL to http://localhost:8666. Run the test suite. Flakes caused by network timing will surface immediately.

5. CPU Starvation

# Linux: run test under cpulimit (10% of one core)
cpulimit -l 10 -- npx playwright test

# Or use cgroups directly
sudo cgcreate -g cpu:/test_throttle
sudo cgset -r cpu.cfs_quota_us=10000 /test_throttle  # 10% of 100ms period
sudo cgexec -g cpu:/test_throttle npx playwright test

6. The "Bisect to Two Tests" Workflow

When you have order dependence but don't know which pair:

# 1. Get the full test list in the failing order
pytest --collect-only -q > all_tests.txt

# 2. Binary search script
cat > bisect.py << 'EOF'
import subprocess, sys

tests = [line.strip() for line in open('all_tests.txt')]
lo, hi = 0, len(tests)

while lo < hi:
    mid = (lo + hi) // 2
    subset = tests[:mid] + [tests[-1]]  # always include the failing test
    result = subprocess.run(['pytest', '-x'] + subset, capture_output=True)
    if result.returncode == 0:
        lo = mid + 1
    else:
        hi = mid
    print(f"Range [{lo}, {hi}), mid={mid}, passed={result.returncode==0}")

print(f"Minimal failing prefix: {tests[:lo]}")
EOF
python bisect.py

This isolates the exact test that pollutes state for the victim test.

---

Automated Flake Detection and Quarantine

Manual reproduction doesn't scale. You need automated detection, quarantine, and tracking.

Flake Detection Pipeline

# .github/workflows/flake-detection.yml
name: Flake Detection
on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM daily
  workflow_dispatch:

jobs:
  detect:
    runs-on: ubuntu-latest
    timeout-minutes: 120
    steps:
      - uses: actions/checkout@v4
      - name: Run suite 20 times with different seeds
        run: |
          for i in {1..20}; do
            SEED=$RANDOM
            npx playwright test --seed=$SEED --reporter=json > "run-$SEED.json" 2>&1 || true
          done
      - name: Analyze flakes
        run: |
          python << 'PYEOF'
          import json, glob, collections
          
          results = collections.defaultdict(lambda: {'pass': 0, 'fail': 0, 'seeds': []})
          
          for f in glob.glob('run-*.json'):
              seed = f.split('-')[1].split('.')[0]
              data = json.load(open(f))
              for suite in data.get('suites', []):
                  for spec in suite.get('specs', []):
                      for test in spec.get('tests', []):
                          key = f"{spec['title']} > {test['title']}"
                          if test['results'][0]['status'] == 'passed':
                              results[key]['pass'] += 1
                          else:
                              results[key]['fail'] += 1
                              results[key]['seeds'].append(seed)
          
          flakes = {k: v for k, v in results.items() if v['fail'] > 0 and v['pass'] > 0}
          
          print(f"Total tests: {len(results)}")
          print(f"Flaky tests: {len(flakes)}")
          
          for name, data in sorted(flakes.items(), key=lambda x: -x[1]['fail']):
              rate = data['fail'] / (data['pass'] + data['fail'])
              print(f"  {rate:.1%} flake rate | {data['fail']} fails | seeds: {data['seeds']} | {name}")
          
          # Write flake report for quarantine system
          with open('flake-report.json', 'w') as f:
              json.dump(flakes, f, indent=2)
          PYEOF
      - name: Update quarantine list
        run: |
          python << 'PYEOF'
          import json, os
          
          with open('flake-report.json') as f:
              flakes = json.load(f)
          
          # Load existing quarantine
          quarantine_path = 'quarantine.json'
          if os.path.exists(quarantine_path):
              with open(quarantine_path) as f:
                  quarantine = json.load(f)
          else:
              quarantine = {}
          
          # Update: add new flakes, increment counters, remove fixed
          for name, data in flakes.items():
              rate = data['fail'] / (data['pass'] + data['fail'])
              if rate >= 0.05:  # 5% threshold
                  if name not in quarantine:
                      quarantine[name] = {'first_seen': '2024-01-15', 'max_rate': rate, 'occurrences': 1}
                  else:
                      quarantine[name]['max_rate'] = max(quarantine[name]['max_rate'], rate)
                      quarantine[name]['occurrences'] += 1
              elif name in quarantine:
                  # Test stabilized — remove after 3 clean runs
                  quarantine[name]['occurrences'] = max(0, quarantine[name]['occurrences'] - 1)
                  if quarantine[name]['occurrences'] == 0:
                      del quarantine[name]
          
          with open(quarantine_path, 'w') as f:
              json.dump(quarantine, f, indent=2)
          
          # Fail if new flakes introduced
          new_flakes = [n for n in flakes if n not in quarantine or quarantine[n]['occurrences'] == 1]
          if new_flakes:
              print("::error::New flakes detected:")
              for n in new_flakes:
                  print(f"  {n}")
              sys.exit(1)
          PYEOF
      - name: Commit quarantine update
        uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: "chore: update flake quarantine [skip ci]"
          file_pattern: quarantine.json

Quarantine Policy

Quarantine is not "ignore forever." It is "remove from blocking CI while we fix."

// playwright.config.ts — quarantine integration
import { readFileSync } from 'fs';

const quarantine = JSON.parse(readFileSync('quarantine.json', 'utf-8'));
const quarantinedTests = new Set(Object.keys(quarantine));

export default defineConfig({
  projects: [
    {
      name: 'main',
      testMatch: '**/*.spec.ts',
      testIgnore: Array.from(quarantinedTests).map(t => `**/${t}.spec.ts`),
    },
    {
      name: 'quarantine',
      testMatch: Array.from(quarantinedTests).map(t => `**/${t}.spec.ts`),
      retries: 3,  // extra retries for visibility
      reporter: [['json', { outputFile: 'quarantine-results.json' }]],
    },
  ],
});

Confidence Scoring

Every test gets a confidence score (0–100). CI gates on the suite's aggregate confidence.

# confidence.py
def calculate_confidence(test_name: str, history: list[dict]) -> int:
    """
    history: list of {timestamp, seed, status, duration_ms, retries}
    """
    if not history:
        return 50  # unknown
    
    total = len(history)
    passes = sum(1 for h in history if h['status'] == 'passed')
    flakes = sum(1 for h in history if h['retries'] > 0)
    avg_duration = sum(h['duration_ms'] for h in history) / total
    p95_duration = sorted(h['duration_ms'] for h in history)[int(total * 0.95)]
    
    # Base score from pass rate
    pass_rate = passes / total
    score = pass_rate * 70
    
    # Penalty for flakes (retries > 0 means flaky pass)
    score -= flakes / total * 20
    
    # Penalty for slowness (slow tests flake more)
    if p95_duration > 30_000:  # 30s
        score -= 10
    elif p95_duration > 60_000:
        score -= 20
    
    # Bonus for determinism (same result across all seeds)
    seeds = set(h['seed'] for h in history)
    if len(seeds) > 10 and flakes == 0:
        score += 10
    
    return max(0, min(100, int(score)))

# Gate in CI
- name: Confidence Gate
  run: |
    python << 'PYEOF'
    import json, sys
    with open('test-history.json') as f:
        history = json.load(f)
    
    scores = {name: calculate_confidence(name, runs) for name, runs in history.items()}
    avg = sum(scores.values()) / len(scores)
    min_score = min(scores.values())
    
    print(f"Suite confidence: {avg:.1f} (min: {min_score})")
    
    if avg < 85:
        print("::error::Suite confidence below 85")
        sys.exit(1)
    if min_score < 50:
        print("::error::Individual test confidence below 50")
        for name, score in scores.items():
            if score < 50:
                print(f"  {name}: {score}")
        sys.exit(1)
    PYEOF

---

Fixing the Six Major Flake Classes

Each class demands a specific remediation pattern. Apply the pattern, verify with the matrix, move on.

Class 1: Timing / Async Waits

Root cause: Test asserts before app state settles.

Fix pattern: Replace all implicit waits with explicit, condition-based waits. Ban sleep, waitForTimeout, Thread.sleep.

// BAD
await page.click('#submit');
await page.waitForTimeout(2000); // flake factory
expect(await page.textContent('.result')).toBe('Success');

// GOOD — wait for the *specific* condition
await page.click('#submit');
await expect(page.locator('.result')).toHaveText('Success', { timeout: 10_000 });

// GOOD — wait for network idle if that's the trigger
await page.click('#submit');
await page.waitForResponse(r => r.url().includes('/api/submit') && r.status() === 200);
await expect(page.locator('.result')).toHaveText('Success');

// GOOD — wait for state in framework (React/Vue/Svelte)
await page.click('#submit');
await page.waitForFunction(() => window.__APP_STATE__.submitted === true);

Anti-pattern: "Wait for element to exist" when you need "wait for element to be interactive."

// FLAKY: element exists but is covered by overlay, disabled, or offscreen
await expect(page.locator('button.submit')).toBeVisible();
await page.click('button.submit');

// ROBUST: wait for actionability (Playwright auto-waits, but be explicit elsewhere)
await expect(page.locator('button.submit')).toBeEnabled();
await expect(page.locator('button.submit')).not.toBeHidden();
await page.click('button.submit');

Library-specific fixes:

Class 2: Shared State Pollution

Root cause: Mutable global state persists across tests.

Fix pattern: Isolate at the finest granularity possible.

// 1. Module-level state → dependency injection
// BAD: singleton
export const cache = new Map();

// GOOD: factory + per-test instance
export function createCache() { return new Map(); }

// In test:
const cache = createCache();
const service = new DataService(cache);

// 2. Database → transaction rollback
// pytest fixture
@pytest.fixture
def db_session(connection):
    transaction = connection.begin()
    session = Session(bind=connection)
    yield session
    session.close()
    transaction.rollback()  // <-- critical

// 3. Browser storage → clear in beforeEach
beforeEach(async () => {
  await page.evaluate(() => {
    localStorage.clear();
    sessionStorage.clear();
    indexedDB.databases().then(dbs => 
      dbs.forEach(db => indexedDB.deleteDatabase(db.name))
    );
  });
});

// 4. Singleton services → reset or re-instantiate
beforeEach(() => {
  TestBed.resetTestingModule(); // Angular
  // or
  jest.resetModules(); // Jest — clears module cache
  // or
  vi.resetModules(); // Vitest
});

// 5. Environment variables → snapshot/restore
const originalEnv = { ...process.env };
afterEach(() => {
  process.env = { ...originalEnv };
});

Verification: Run with --randomly-seed 50 times. Zero flakes = fixed.

Class 3: Test Order Dependence

Root cause: Test A creates state that Test B consumes (or breaks).

Fix pattern: Make tests hermetic. Each test sets up its own preconditions.

// BAD: Test B assumes user exists from Test A
test('user can view profile', async () => {
  await page.goto('/profile/42');
  await expect(page.locator('h1')).toContainText('Alice');
});

// GOOD: Test creates its own data
test('user can view profile', async () => {
  const user = await api.createUser({ name: 'Alice', email: 'alice@test.com' });
  await page.goto(`/profile/${user.id}`);
  await expect(page.locator('h1')).toContainText('Alice');
});

// GOOD: Use API to set up, UI to verify
test('admin sees all users', async () => {
  await api.createUser({ name: 'Bob', role: 'user' });
  await api.createUser({ name: 'Carol', role: 'admin' });
  await ui.loginAsAdmin();
  await page.goto('/admin/users');
  await expect(page.locator('tr')).toHaveCount(3); // header + 2 users
});

If you must share setup (expensive operations), use a fixture with explicit scope and document the dependency:

// test-fixtures.ts
export const test = base.extend<{ adminUser: User }>({
  adminUser: [async ({}, use) => {
    const user = await api.createUser({ name: 'Admin', role: 'admin' });
    await use(user);
    await api.deleteUser(user.id); // cleanup
  }, { scope: 'worker' }], // shared within worker process
});

Class 4: Network / External Dependency

Root cause: Real network is non-deterministic.

Fix pattern: Contract testing + recorded fixtures. Never hit real external APIs in unit/integration tests.

// 1. Record real interactions (once)
import { recordHar } from './har-recorder';

test('record payment flow', async () => {
  await recordHar('payment-flow.har', async () => {
    await page.goto('/checkout');
    await page.click('[data-testid="pay-with-stripe"]');
    // ... complete real Stripe