Smoke Testing vs Sanity Testing: Differences, When to Use Each

Software teams ship code continuously. Between a commit and a release, a handful of quick checks can stop obvious regressions before they reach users. Smoke testing and sanity testing are two of those

Smoke Testing vs Sanity Testing: Differences, When to Use Each

Software teams ship code continuously. Between a commit and a release, a handful of quick checks can stop obvious regressions before they reach users. Smoke testing and sanity testing are two of those checks, yet they are often confused or used interchangeably. This guide explains what each technique really means, where it belongs in the delivery pipeline, who typically owns it, and how to automate a lightweight but reliable gate that runs on every build.

---

1. What Smoke Testing Really Is

1.1 Definition and Goal

Smoke testing is a shallow, wide‑ranging verification that the most critical paths of an application start up and respond without fatal errors. The term comes from hardware: you power on a device and look for smoke; if none appears, you assume the basic circuitry works. In software, a smoke suite answers the question: “Can the system be used at all?”

A successful smoke run means:

• The application launches (or the service starts).
• Core navigation (home screen, main menu, login) is reachable.
• No uncaught exceptions, crashes, or ANRs (Application Not Responding) appear during the exercised flow.
• Basic integrations (database connectivity, authentication endpoint) return expected responses.

If any of these fail, the build is considered broken and should not proceed to further testing.

1.2 Typical Scope

A smoke suite touches a small percentage of total test cases—often 5‑15 %—but those cases cover:

The checks are deliberately independent of deep business rules. They do not validate calculations, edge‑case inputs, or UI polish.

1.3 Who Owns Smoke Tests?

• DevelopersDevelopers** write the initial smoke scripts while building a feature, because they know which calls must succeed for their code to be usable.
• QA engineers maintain the smoke suite as a shared asset, adding new critical paths as the product evolves.
• DevOps / Release engineers trigger the smoke run as the first gate in CI, often blocking promotion to staging if it fails.

---

2. What Sanity Testing Really Is

2.1 Definition and Goal

Sanity testing is a narrow, focused verification that a recent change—usually a bug fix or a small feature—has not broken the surrounding functionality. Unlike smoke, which asks “does the system start?”, sanity asks “did we fix what we intended, and did we not break anything else nearby?”

A sanity pass is performed after a developer has addressed a defect and before the change goes through full regression. It confirms that the specific area under work behaves as expected and that adjacent modules remain stable.

2.2 Typical Scope

Sanity targets the changed component plus its immediate dependencies. The set of checks is usually derived from the ticket description or the commit message. Typical sanity items include:

Because the scope is limited, a sanity suite can often be executed in under two minutes on a single device or container.

2.3 Who Owns Sanity Tests?

• Developers usually write sanity checks themselves, treating them as an extension of unit‑test‑style verification but at the integration level.
• QA may review and add sanity checks for high‑risk areas that developers overlook (e.g., side‑effects on billing when changing a UI flow).
• Release managers sometimes request a sanity sign‑off before allowing a hot‑fix to bypass the full regression cycle.

---

3. Smoke vs Sanity: Side‑by‑Side Comparison

---

4. Where Smoke and Sanity Live in the CI/CD Pipeline

4.1 Typical Pipeline Stages

1. Commit – developer pushes code.
2. Unit‑test stage – runs fast, isolated tests.
3. Build & package – compiles APK/AAB or bundles web assets.
4. Smoke gate – executes the smoke suite on a fresh emulator/device or a containerized browser.
5. If smoke passes → proceed to integration / component tests.
6. If smoke fails → block pipeline, notify owner, and halt further stages.
7. After integration tests → optional sanity window for hot‑fixes or small changes.
8. Regression suite (nightly or on demand) – broader validation.
9. Acceptance / UAT – business‑oriented validation.
10. Production deploy – final release.

4.2 Timing Details

• Smoke runs on every commit (or every scheduled build) because its cost is low and its value high: catching a broken build early saves hours of wasted debugging.
• Sanity runs on demand: after a developer marks a ticket as “Ready for QA” or when a release‑candidate branch is cut. Some teams also run a sanity check as part of a pre‑merge step for branches that only touch high‑risk code (e.g., payment gateway).
• In continuous deployment pipelines that promote directly to staging after a successful smoke, a sanity step may be inserted before the promotion to production, acting as a final sanity check on the release candidate.

4.3 Example CI Configuration (GitHub Actions)

name: CI Pipeline

on:
  push:
    branches: [ main ]
  pull_request:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up JDK
        uses: actions/setup-java@v3
        with:
          distribution: temurin
          java-version: '17'
      - name: Build Android APK
        run: ./gradlew assembleDebug

  smoke:
    needs: build
    runs-on: macos-latest   # uses Android emulator
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v3
      - name: Install Node & Appium
        run: |
          npm install -g appium
          appium driver install uiautomator2
      - name: Start Emulator
        run: |
          emulator -avd Pixel_4_API_33 -no-window -no-audio &
          # wait for device
          adb wait-for-device
          adb shell getprop sys.boot_completed
      - name: Run Smoke Suite (Appium + Python)
        run: |
          pip install -r requirements.txt
          python -m pytest tests/smoke/ -v --junitxml=smoke-results.xml
      - name: Publish Results
        if: always()
          uses: actions/upload-artifact@v3
          with:
            name: smoke-report
            path: smoke-results.xml

  sanity:
    # triggered manually via workflow_dispatch or by a label on PR
    if: github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'needs-sanity')
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install Playwright
        run: npm i -D @playwright/test
      - name: Run Sanity Checks (Playwright)
        run: npx playwright test tests/sanity/ --reporter=json --output=sanity-results.json
      - name: Upload Sanity Artifact
        if: always()
          uses: actions/upload-artifact@v3
          with:
            name: sanity-report
            path: sanity-results.json

*The smoke job runs on every push; the sanity job is optional and can be triggered manually or via a label.*

---

5. Building a Fast, Reliable Smoke Suite

5.1 Selecting the Right Tests

A smoke suite should be stable, fast, and representative of real user entry points. Start by mapping the user journey matrix: list all major entry points (home, login, search, settings) and the core actions reachable from each (view product, add to cart, submit form). For each entry point, pick one verification that confirms the path is alive.

If a feature is behind a feature flag, the smoke test should respect the flag (i.e., test both on and off states if the flag can be toggled via API).

5.2 Test Data Strategies

• Use static, deterministic data that exists in every test environment (e.g., a pre‑seeded user with known credentials).
• Avoid reliance on data that changes frequently (e.g., live product catalog) unless you can mock or stub the backend.
• For mobile, push a known‑state database onto the device before each smoke run using adb push or a test‑only content provider.

5.3 Automation Frameworks

5.4 Parallel Execution & Resource Management

• Device farms (Firebase Test Lab, AWS Device Farm, or an in‑house pool) allow you to run the same smoke script on multiple screen sizes, OS versions, and locales simultaneously.
• For web, Playwright’s test sharding (--shard=1/3) splits the suite across CI containers.
• Keep each test under 30 seconds; longer tests defeat the purpose of a quick gate. Use explicit waits only for unavoidable asynchronous events (e.g., network response).

5.5 Flaky‑Test Mitigation

Flaky smoke tests erode trust in the gate. Apply these tactics:

1. Deterministic start‑state – wipe app data or reset the emulator before each run.
2. Retry wrapper – allow a single retry only for known intermittent issues (e.g., occasional emulator boot delay).
3. Log and metrics – capture start‑time, end‑time, device logs, and screenshot on failure; feed them into a dashboard to spot trends.
4. Isolate external dependencies – stub or mock third‑party services (payment gateway, analytics) using tools like WireMock or MockServer; only call the real endpoint in a separate “canary” job that runs less frequently.

5.6 Using SUSA for Autonomous Smoke

SUSA can ingest an APK or a web URL and automatically generate a smoke‑like exploration: it launches the app, taps primary navigation elements, attempts login with a set of test credentials, and records any crashes or ANRs. Because SUSA builds its own behavior models (curious, impatient, novice, etc.), it can surface issues that a scripted smoke suite might miss—such as a dialog that blocks the main thread only under a specific locale.

To incorporate SUSA into the pipeline:

# Install the agent
pip install susatest-agent

# Run a smoke exploration against a locally built APK
susatest run \
  --app ./build/outputs/apk/debug/app-debug.apk \
  --personas curious impatient \
  --timeout 180 \
  --output susa-smoke-report.json

The JSON report includes a PASS/FAIL flag based on whether any critical flow terminated unexpectedly. You can fail the CI step if the report contains a FAIL.

---

6. Designing Effective Sanity Checks

6.1 Risk‑Based Selection

When a developer fixes a bug, ask:

• What component changed?
• Which integrations does it touch?
• What are the failure modes most likely to regress?

Create a sanity checklist that answers those questions. For a bug in the “apply coupon” function, the checklist might be:

1. Load the cart with at least one item.
2. Apply a valid coupon code → discount appears, total updates.
3. Apply an expired coupon → appropriate error message shows.
4. Apply a coupon that exceeds cart total → system blocks and shows warning.
5. Navigate away and back → coupon remains applied (state persistence).

6.2 Leveraging Existing Test Cases

Often, a subset of an existing regression test already covers the needed ground. Instead of writing from scratch, tag those regression tests with a label like sanity‑candidate. Your CI can then run only the tagged tests when a sanity trigger fires.

6.3 Example: Mobile Sanity Script (Appium + Python)

import time
from appium import webdriver
from appium.webdriver.common.appiumby import AppiumBy

def test_apply_coupon(driver):
    # 1. Ensure we are on the cart page
    cart = driver.find_element(AppiumBy.ID, "cart_icon")
    cart.click()
    assert driver.find_element(AppiumBy.ID, "empty_cart_msg") is None

    # 2. Add a product if cart empty
    add_btn = driver.find_element(AppiumBy.ID, "add_to_cart_button")
    add_btn.click()
    time.sleep(1)  # allow UI update

    # 3. Apply valid coupon
    coupon_input = driver.find_element(AppiumBy.ID, "coupon_input")
    coupon_input.send_keys("SAVE10")
    apply_btn = driver.find_element(AppiumBy.ID, "apply_button")
    apply_btn.click()

    # 4. Verify discount
    discount_text = driver.find_element(AppiumBy.ID, "discount_amount").text
    assert "$2.00" in discount_text  # assuming $20 item, 10% off

    # 5. Verify total
    total_text = driver.find_element(AppiumBy.ID, "total_amount").text
    assert "$18.00" in total_text

    # 6. Apply expired coupon – expect error
    coupon_input.clear()
    coupon_input.send_keys("OLD20")
    apply_btn.click()
    error_msg = driver.find_element(AppiumBy.ID, "coupon_error").text
    assert "expired" in error_msg.lower()

# Pytest hook to get driver fixture

This script runs in under 15 seconds on a typical emulator and validates the exact behavior changed by the ticket.

6.4 Using SUSA for Targeted Sanity

SUSA’s persona‑based exploration can be constrained to a specific screen or flow via the --start-activity or --start-url flag. For a sanity check on a modified checkout screen, you might run:

susatest run \
  --app ./app-debug.apk \
  --start-activity com.example.app.CheckoutActivity \
  --personas novice power_user \
  --max-steps 30 \
  --output susa-sanity-checkout.json

The resulting report tells you whether any of the selected personas encountered a crash, ANR, or accessibility violation while navigating the checkout flow. If the report is clean, you have a lightweight sanity signal that complements the developer’s scripted check.

---

7. Automation Strategies and Tooling Details

7.1 Test Execution Orchestration

• Local dev – run smoke/sanity via IDE or a simple script (./gradlew connectedAndroidTest).
• CI – wrap the execution in a Docker image that contains the device emulator, browsers, and all dependencies. This guarantees reproducibility across branches.
• Scheduled runs – in addition to per‑commit smoke, run a full‑device‑matrix smoke nightly to catch OS‑specific regressions that may not appear on the default emulator image.

7.2 Reporting and Metrics

A useful smoke/sanity report includes:

Many CI systems (GitHub Actions, GitLab CI, Jenkins) can publish JUnit XML or TestResults.json; plug those into a dashboard like Grafana or Allure for historical view.

7.3 Dependency Management

• Pin versions of Appium, Playwright, and device images in a requirements.txt or package.json.
• Use a container image that bundles the Android SDK, emulator system images, and Node.js; tag it with the version of your test framework.
• Cache the emulator snapshots between CI runs to shave off boot time (most CI providers allow saving a directory as a cache).

7.4 Handling Permissions and Dialogs

Mobile apps often present runtime permission dialogs (location, camera, notifications). In a smoke run you want to grant them automatically so they don’t block the flow.

• With Appium:

  driver.manage().settings().setSetting(Settings.SETTING_WAIT_FOR_IDLE_TIMEOUT, 0);
  driver.resetPermissions(); // grants all declared permissions in manifest

• With Playwright on Android via Chromium: use --use-fake-ui-for-media-stream and --disable-features=VizDisplayCompositor to bypass prompts.

7.5 Security and Smoke

A smoke suite should not store real credentials. Use environment variables injected at runtime (TEST_USER, TEST_PASS) and never commit them. For OAuth flows, consider using a mock identity provider that returns a predetermined token.

---

8. Edge Cases That Only Appear in Production

Even the most thorough smoke/sanity can miss issues that surface only under real‑world load or specific user conditions. Below are common categories and mitigations.

8.1 Simulating Network Conditions

*Android*:

adb shell tc qdisc add dev wlan0 root netem delay 200ms loss 5%

*Playwright*:

context = browser.new_context()
await context.route("**/*", lambda route: route.continue_(headers={}))
await context.set_offline(False)
await context.set_network_conditions(
    latency=200,
    download_throughput=500*1024,   # 500 Kbps
    upload_throughput=500*1024,
)

8.2 Automated Accessibility Checks

*Android*: Use the Accessibility Test Framework (ATF) bundled with Espresso, or run adb shell uiautomator runtest AccessibilityTest.jar.

*Web*: Playwright integrates with the axe-core library:

import pytest
from playwright.sync_api import expect
from axe_playwright_python import Axe

def test_home_page_accessibility(page):
    page.goto("https://example.com")
    axe = Axe(page)
    axe.inject()
    results = axe.run()
    assert len(results["violations"]) == 0, f"Accessibility violations: {results['violations']}"

Add this as a sanity step for any UI‑affecting change.

---

9. Checklist for Smoke and Sanity

9.1 Smoke‑Test Checklist (Run on Every Build)

9.2 Sanity‑Test Checklist (Run After a Change)

You can turn these checklists into Markdown TODO items in your test repository, enabling quick visual tracking.

---

10. Real‑World Examples

10.1 Smoke Suite for an E‑Commerce Android App

Context: A mid‑size retailer releases new builds twice a day. The app features a home carousel, product catalog, search, cart, and checkout.

Smoke Test Set (implemented with Appium + Java, runs on a Firebase Test Lab matrix of 4 devices):

Each test is independent and takes ≤ 8 seconds on average. The entire suite runs in ~1 minute on four devices in parallel, comfortably fitting into a CI gate.

Outcome: In the last quarter, this smoke suite caught three regressions: a missing permission declaration that caused a crash on Android 13, a layout bug that hid the search icon on tablets, and a race condition in the cart‑add endpoint that produced duplicate entries under high latency.

10.2 Sanity Check for a Banking Web App’s Transfer Flow

Context: A hot‑fix was applied to the “Enter amount” field to prevent users from submitting transfers with more than two decimal places.

Sanity Test Set (Playwright + TypeScript, runs against a staging URL):

The suite runs in ~12 seconds on a single Chrome instance. It caught a regression where the OTP screen erroneously displayed the raw amount with four decimal places due to a formatting pipe that was not updated after the fix.

---

11. Closing Takeaways

• Smoke testing is a build‑gate: it answers “is the system usable at all?” and must be fast, reliable, and run on every commit.
• Sanity testing is a change‑gate: it answers “did we fix what we intended and not break anything nearby?” and is scoped to the risk introduced by a recent modification.
• Both techniques complement each other and sit below full regression and acceptance testing in the testing pyramid.
• Automation is essential: choose a framework that matches your platform (Appium for mobile, Playwright/WebDriver for web), keep each test under 30 seconds, and run them in parallel on a device/browser matrix.
• Flakiness is the enemy of a smoke gate; mitigate it with deterministic state, limited retries, and rich logging.
• Leverage autonomous tools like SUSA for exploratory smoke or targeted sanity runs—they can surface issues that scripted checks miss, especially around edge