How to Automate Accessibility Settings Testing (Step-by-Step)

How to Automate Accessibility Settings Testing (Step-by-Step) begins with understanding why automating these checks matters for both compliance and user experience. Accessibility settings—such as font

How to Automate Accessibility Settings Testing (Step-by-Step) begins with understanding why automating these checks matters for both compliance and user experience. Accessibility settings—such as font scaling, high‑contrast mode, screen reader gestures, and touch‑target adjustments—are often toggled by users who rely on them daily. Manual verification of every setting across multiple device configurations is time‑consuming and error‑prone. Automating the process lets you catch regressions early, ensure consistent behavior across personas, and free QA engineers to focus on exploratory testing. This guide walks you through a complete, repeatable workflow: from environment setup and framework selection to locator strategy, flaky‑test mitigation, CI integration, reporting, and how autonomous exploration can bootstrap the effort without writing a single script up front.

How to Automate Accessibility Settings Testing (Step-by-Step): Preparing Your Environment

Before writing any test, you need a stable, reproducible test harness that can launch the application under the accessibility states you intend to validate. The foundation consists of three layers: device or emulator management, test runner configuration, and accessibility‑specific tooling.

Device and emulator provisioning

For mobile apps, Android emulators or real devices connected via ADB provide the most faithful representation of system‑level accessibility services. Use the Android SDK’s avdmanager to create a baseline emulator with Google Play services, then clone it for each accessibility profile you plan to test (e.g., default, large text, high contrast, TalkBack enabled). A simple Bash script can automate the creation:

# create-base-avd.sh
avdmanager create avd -n accessibility_base -k "system-images;android-33;google_apis;x86_64"
# clone for each profile
for profile in default large_text high_contrast talkback; do
  avdmanager create avd -n "accessibility_${profile}" -s accessibility_base
done

On iOS, Xcode’s simctl lets you bootstrap simulators with specific accessibility settings via defaults write. For web applications, you can launch Chrome or Firefox with command‑line flags that force high contrast (--force-high-contrast-support) or install extensions like axe‑core programmatically.

Test runner and language bindings

Choose a test runner that integrates well with your CI pipeline and offers rich assertions for UI properties. Popular options include:

If your team already uses a behavior‑driven framework like Cucumber, you can wrap any of the runners in step definitions that read feature files describing accessibility scenarios.

Accessibility‑specific libraries

To avoid reinventing the wheel, incorporate open‑source validators that expose WCAG rules as programmable APIs:

• axe‑core (JavaScript) – run via page.evaluate in Playwright or Selenium; returns violations with impact levels.
• Android Accessibility Test Framework (AATF) – provides Espresso‑compatible matchers for content descriptions, labeling, and touch target size.
• iOS XCTest + AccessibilityInspector – use XCUIElement properties like label, value, and traits.
• pa11y – CLI tool for web pages; can be invoked from Node scripts and output JSON for CI consumption.

Install these as dev dependencies. For a Python‑based mobile test suite, a typical requirements.txt might look like:

Appium-Python-Client==2.8.0
selenium==4.15.0
axe-selenium-python==1.0.0
pytest==8.2.0
pytest-html==4.1.1

With the environment ready, you can now move to selecting the framework that best matches your product’s technology stack and the accessibility settings you need to exercise.

How to Automate Accessibility Settings Testing (Step-by-Step): Choosing the Right Framework

The framework decision hinges on three factors: the platforms you support, the depth of accessibility interaction required, and the existing skill set of your team. Below is a decision matrix that maps common scenarios to recommended stacks.

Decision matrix

If your product spans more than one scenario, consider a polyglot approach: keep a thin orchestration layer (e.g., a Node script) that launches the appropriate runner based on an environment variable (TEST_PLATFORM=android|ios|web). This keeps test code isolated while giving you a unified CI entry point.

Evaluating flakiness propensity

Frameworks differ in how they handle asynchronous UI updates. Appium’s default implicit wait can hide timing issues, while Playwright’s auto‑waiting mechanism reduces flake but may mask underlying performance regressions. To compare, run a small sanity suite (e.g., open Settings → Accessibility → Font size → select “Largest”) on each framework and record the success rate over 20 runs. The table below shows illustrative results from a mid‑size Android app:

These numbers are not universal but illustrate why you should benchmark your own context before committing.

Skill‑set and maintenance considerations

• Language familiarity: If your team writes most unit tests in Python, staying within that language reduces context switching.
• Community support: Appium has a large community but slower release cadence; Playwright releases monthly with strong documentation.
• Debugging tooling: Playwright offers built‑in trace viewers and video capture; Appium relies on external tools like adb logcat or Android Studio profiler.
• License: All mentioned frameworks are open‑source (MIT/Apache), suitable for commercial use.

After you settle on a framework, the next step is to define what you will actually test—this is where a test matrix becomes invaluable.

How to Automate Accessibility Settings Testing (Step-by-Step): Designing a Test Matrix

A test matrix captures the combinatorial space of accessibility settings, user personas, and critical user flows. By enumerating these dimensions up front, you avoid ad‑hoc testing and ensure coverage of edge cases that only manifest under specific configurations.

Defining dimensions

1. Accessibility Settings – binary or multi‑state toggles you control via the OS or app:

• Font size (Default, Large, Largest)
• Display size (Default, Larger, Largest)
• High contrast mode (On/Off)
• Color inversion (On/Off)
• TalkBack/VoiceOver (Enabled/Disabled)
• Switch Control (Enabled/Disabled)
• Reduced motion (On/Off)
• Captioning (On/Off)
• Audio balance (Left/Center/Right)
• Mono audio (On/Off)

1. User Personas – behavioral profiles that affect interaction patterns (e.g., how fast a user taps, tolerance for delays). SUSA’s platform ships with predefined personas, but you can emulate them via parameterized test data:

• Curious – explores every setting, reads descriptions.
• Impatient – skips long dialogs, expects instant feedback.
• Novice – relies heavily on labels and hints.
• Adversarial – tries invalid inputs, rapid back‑button presses.
• Elderly – prefers larger touch targets, slower gestures.
• Accessibility – uses screen reader, high contrast, large text.
• Power user – uses shortcuts, expects deep customization.
• Vestibular – avoids motion, needs reduced‑motion compliance.

1. Critical Flows – end‑to‑end user journeys that, if broken, cause business impact:

• Login / Signup
• Product search → Add to cart → Checkout
• Profile edit → Save changes
• Help center → Submit ticket
• Settings → Accessibility toggle → Verify persistence

Building the matrix

A practical way to generate the matrix is to use a CSV file where each row represents a unique combination of settings and persona, and columns indicate which flows to execute. Below is a truncated example (full matrix may contain dozens of rows):

You can generate this CSV programmatically with a short Python script:

import itertools, csv

font_sizes = ["Default", "Large", "Largest"]
display_sizes = ["Default", "Larger", "Largest"]
contrasts = ["Off", "On"]
talkbacks = ["Off", "On"]
personas = ["Curious", "Impatient", "Novice", "Adversarial", "Elderly", "Accessibility", "Power user", "Vestibular"]
flows = ["Login", "Checkout", "ProfileEdit", "HelpSubmit", "SettingsAccess"]

rows = []
for fs, ds, cb, tb in itertools.product(font_sizes, display_sizes, contrasts, talkbacks):
    for p in personas:
        for f in flows:
            rows.append([fs, ds, cb, tb, p, f, ""])

with open("accessibility_matrix.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["FontSize","DisplaySize","HighContrast","TalkBack","Persona","Flow","ExpectedOutcome"])
    writer.writerows(rows)

Mapping matrix to test cases

Each row becomes a parameterized test case. In pytest, you can load the CSV and use @pytest.mark.parametrize:

import pytest, csv

def load_matrix():
    with open("accessibility_matrix.csv") as f:
        reader = csv.DictReader(f)
        return [(row["FontSize"], row["DisplaySize"], row["HighContrast"],
                 row["TalkBack"], row["Persona"], row["Flow"], row["ExpectedOutcome"])
                for row in reader]

@pytest.mark.parametrize("font,display,contrast,talkback,persona,flow,expected", load_matrix())
def test_accessibility_flow(font, display, contrast, talkback, persona, flow, expected, driver):
    # 1. Apply OS‑level settings via adb or UI
    apply_settings(driver, font, display, contrast, talkback)
    # 2. Set persona‑specific behavior (e.g., tap speed, gesture length)
    set_persona(driver, persona)
    # 3. Execute the flow
    outcome = execute_flow(driver, flow)
    # 4. Validate against expected
    assert outcome == expected, f"Flow {flow} failed under {font}/{display}/{contrast}/{talkback} for {persona}"

The matrix guarantees that every combination of a setting and a persona is exercised at least once for each flow. This systematic approach surfaces issues such as a button that becomes unreachable when font size is increased *and* TalkBack is enabled, a scenario that rarely appears in manual exploratory testing.

How to Automate Accessibility Settings Testing (Step-by-Step): Writing Stable Locators and Handling Waits

Flaky tests often stem from brittle locators or improper synchronization. In accessibility testing, the UI can shift dramatically when font size or display scaling changes, making static XPath or resource‑id selectors unreliable. Adopt a locator strategy that survives these transformations and pair it with explicit waits that respect animation durations.

Prefer accessibility‑focused attributes

Both Android and iOS expose accessibility‑specific properties that are less likely to change with visual scaling:

• Android: content-desc (content description), hint, label, tooltipText.
• iOS: label, value, hint.
• Web: aria-label, aria-labelledby, role, title.

When you need to interact with a control, first query by its accessibility attribute. For example, in Appium with Python:

def find_by_content_desc(driver, desc):
    return driver.find_element(AppiumBy.ACCESSIBILITY_ID, desc)

If the element lacks a content description (a common accessibility bug), the test will fail fast, surfacing the missing label—a desirable outcome.

Combining attributes for resilience

Sometimes a single attribute is not unique (e.g., multiple buttons share the same hint). Combine two attributes using a compound selector. In Appium’s Android UIAutomator2 you can chain selectors:

from appium.webdriver.common.mobileby import MobileBy

def find_unique_button(driver, desc, resource_id):
    return driver.find_element(
        MobileBy.ANDROID_UIAUTOMATOR,
        f'new UiSelector().description("{desc}").resourceId("{resource_id}")'
    )

On the web with Playwright, you can use CSS selectors that include ARIA attributes:

button = page.locator('button[aria-label="Submit"][data-test-id="pay-button"]')

Dealing with dynamic IDs

If your app generates random resource IDs (common in some cross‑platform frameworks), avoid them entirely. Instead, rely on text content (when it is static and translated) or hierarchy. For example, locate a “Settings” item by its position relative to a static header:

# Android: find the TextView with text "Settings", then click the sibling Switch
settings_label = driver.find_element(AppiumBy.ANDROID_UIAUTOMATOR,
                                    'new UiSelector().text("Settings")')
switch = settings_label.find_element(
    AppiumBy.XPATH,
    "./following-sibling::android.widget.Switch"
)

Explicit waits that respect animation

When you change a system accessibility setting (e.g., enable TalkBack), the OS may animate the transition or delay the broadcast of an accessibility event. Use waits that poll for a specific condition rather than a fixed time.

Appium (Python) example with WebDriverWait:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def wait_for_talkback_enabled(driver, timeout=15):
    def _talkback_on(d):
        # Query the accessibility service state via settings API
        state = d.execute_script(
            "return Settings.Secure.getInt(getContentResolver(), "
            "'accessibility_enabled', 0);"
        )
        return state == 1
    return WebDriverWait(driver, timeout).until(_talkback_on)

Playwright example for waiting until contrast change takes effect:

await page.wait_for_function(
    """() => {
        const computed = getComputedStyle(document.body);
        return computed.getPropertyValue('--bg-color') === '#000';
    }""",
    timeout=10000
)

Handling toast or snack‑bar messages

Accessibility changes often produce transient feedback (e.g., a toast announcing “Font size increased”). These can interfere with element detection if they appear over the UI under test. Dismiss them explicitly or wait for them to disappear:

# Android: wait for toast to vanish
WebDriverWait(driver, 10).until_not(
    EC.visibility_of_element_located((AppiumBy.ANDROID_UIAUTOMATOR,
                                      'new UiSelector().className("android.widget.Toast")'))
)

Locator maintenance checklist

• [ ] Prefer content-desc, aria-label, role, hint over resource-id or xpath based on positional indexes.
• [ ] If you must use indexes, wrap them in a helper that logs a warning when the count > expected.
• [ ] After each OS‑setting change, run a quick “sanity locator” test (e.g., find the app’s main toolbar title) to confirm the UI hierarchy is still reachable.
• [ ] Store locators in a separate constants file or data‑driven JSON so a UI rename only requires one edit.
• [ ] Run the test suite with the --tb=short flag to see which locator failures are due to missing accessibility attributes.

With robust locators and waits in place, you can focus on the actual accessibility assertions—checking contrast, label presence, touch target size, and screen‑reader announcements.

How to Automate Accessibility Settings Testing (Step-by-Step): Data Setup, Teardown, and Flaky Test Mitigation

Tests that modify system settings must leave the device or browser in a clean state for the next iteration. Poor teardown leads to cascading failures where a test runs with TalkBack still enabled, causing unexpected behavior in subsequent cases. Additionally, accessibility tests can be sensitive to device performance, leading to occasional flakes unrelated to the product under test. This section details strategies for reliable setup/teardown and techniques to detect and reduce flakiness.

Isolating system‑level changes

Most mobile test frameworks provide a way to issue ADB commands directly from the test code. Use these to save the current state before a test, apply the desired accessibility configuration, and restore it afterward.

Python helper for Android settings:

import subprocess, json, time

def get_current_settings():
    """Return a dict of relevant accessibility settings."""
    out = subprocess.check_output(
        ["adb", "shell", "settings", "list", "secure"]
    ).decode()
    settings = {}
    for line in out.splitlines():
        if "=" in line:
            key, val = line.split("=", 1)
            settings[key] = val
    return settings

def apply_settings(settings_dict):
    """Apply a dict of settings via ADB."""
    for key, val in settings_dict.items():
        subprocess.run(["adb", "shell", "settings", "put", "secure", key, val],
                       check=True)

def restore_settings(original):
    """Restore settings saved earlier."""
    apply_settings(original)

In a pytest fixture:

import pytest

@pytest.fixture(autouse=True)
def accessibility_state(driver):
    original = get_current_settings()
    yield  # test runs here
    # teardown: restore original state
    restore_settings(original)

For web browsers, you can manipulate Chromium flags via Chrome DevTools Protocol (CDP) or launch a fresh context with the desired flags for each test:

# Playwright Python
async def new_context_with_contrast(page):
    await page.context.new_context(
        viewport={"width": 1280, "height": 720},
        color_scheme="dark",  # simulates high contrast prefer‑dark
        forced_colors="active"
    )
    return await page.context.new_page()

Resetting the app state

Beyond system settings, the app itself may retain state (e.g., logged‑in user, cached data). Use a combination of:

• App reset: adb shell pm clear for a full wipe, or driver.reset() in Appium to stop and relaunch the app.
• API‑based cleanup: Call backend endpoints to delete test‑generated data before each test.
• Local storage clear: For web, await page.context.clear_cookies() and await page.context.storage_state(path=None).

Combine these in a fixture that runs after each test:

@pytest.fixture(autouse=True)
def clean_app_state(driver):
    yield
    driver.reset()  # Appium
    # optional: call REST API to wipe test user
    requests.delete(f"{API_BASE}/test-users/{test_user_id}", headers=auth_header)

Detecting flakiness with retry analysis

Even with solid setup/teardown, occasional flakes can appear due to device load, GC pauses, or network jitter. Implement a retry mechanism *only* for analysis, not to mask real failures. Pytest‑rerunfailures is a lightweight plugin:

pip install pytest-rerunfailures

Then run:

pytest --reruns 2 --reruns-delay 5

The plugin records which attempts succeeded. After a test run, generate a flakiness report:

import pytest

def pytest_terminal_summary(terminalreporter, exitstatus, config):
    if hasattr(terminalreporter, "_rerunfailures"):
        for nodeid, info in terminalreporter._rerunfailures.items():
            print(f"Flaky candidate: {nodeid} succeeded on attempt {info['attempt']}/{info['max_reruns']+1}")

Alternatively, integrate with a test‑management tool that tracks test outcome history (e.g., TestRail, Zephyr). Tag any test that shows >10 % variance over the last 20 runs as “flaky‑watch”.

Mitigating common flakiness sources

Teardown verification

After restoring settings, assert that the device is truly back to baseline. A simple verification step prevents drift:

def assert_settings_restored(expected):
    current = get_current_settings()
    for key, val in expected.items():
        assert current.get(key) == val, f"Setting {key} mismatch: expected {val}, got {current.get(key)}"

Call this at the end of the teardown fixture. If it fails, the test framework will mark the test as error, alerting you to a teardown bug before it corrupts subsequent runs.

By combining deterministic setup/teardown, targeted retries for analysis only, and continuous flakiness monitoring, you build a trustworthy accessibility test suite that can run nightly or on every pull request without becoming a maintenance burden.

How to Automate Accessibility Settings Testing (Step-by-Step): Integrating with CI/CD and Reporting

Automated accessibility tests provide the most value when they run continuously, giving immediate feedback on regressions. Integrating them into your CI pipeline requires consideration of execution time, artifact collection, and actionable reporting. This section outlines a practical CI configuration, shows how to publish results in formats that both developers and accessibility specialists can consume, and describes gating strategies that prevent merges when critical violations appear.

CI pipeline design

A typical pipeline for a mobile app might look like:

1. Checkout code.
2. Build APK/AAB (or web bundle).
3. Upload artifact to device farm or attach emulator.
4. Run accessibility test suite (parallelized by device/persona).
5. Collect logs, screenshots, videos, and accessibility violation reports.
6. Publish results as PR comments or dashboard updates.
7. Gate merge based on severity thresholds.

Below is a simplified GitHub Actions workflow that runs the Android matrix on Firebase Test Lab:

name: Accessibility CI

on:
  pull_request:
    branches: [ main ]

jobs:
  accessibility-test:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - name: Set up JDK
        uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: '17'
      - name: Build APK
        run: ./gradlew assembleDebug
      - name: Install Firebase CLI
        run: |
          curl -sL https://firebase.tools | bash
          export PATH="$PATH:$HOME/.firebase"
      - name: Authenticate to Firebase
        env:
          FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
        run: firebase login:ci --no-localhost
      - name: Run Test Lab matrix
        env:
          TEST_LAB_RESULTS_BUCKET: gs://my-testlab-results
        run: |
          gcloud firebase test android run \
            --type instrumentation \
            --app app/build/outputs/apk/debug/app-debug.apk \
            --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
            --device model=Pixel3,version=33,locale=en,orientation=portrait \
            --device model=Pixel3XL,version=33,locale=es,orientation=landscape \
            --environment-variables coverage=true,flag.screenreader=enabled \
            --timeout 90s \
            --results-bucket=$TEST_LAB_RESULTS_BUCKET \
            --results-dir=accessibility_run
      - name: Download results
        run: |
          gsutil -m cp -r $TEST_LAB_RESULTS_BUCKET/accessibility_run/* ./artifacts/
      - name: Process axe results
        run: |
          python scripts/process_axe.py ./artifacts/**/axe-results.json
      - name: Comment on PR
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const report = fs.readFileSync('./artifacts/accessibility-summary.txt', 'utf8');
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `## Accessibility Test Results\n\`\`\n${report}\n\`\`\``
            })

Key points:

• Parallelization: The --device flag can be repeated to test multiple configurations simultaneously. Pair each device with a different set of environment variables (e.g., flag.screenreader=enabled, flag.fontsize=largest) to emulate your matrix without duplicating test code.
• Artifact collection: Test Lab automatically captures logs, video, and screenshots. Pull them down for further processing.
• Result processing: Convert raw axe‑core JSON into a human‑readable summary and a SARIF file for GitHub’s code scanning UI.
• PR feedback: Posting a concise summary directly on the pull request gives developers immediate visibility.

Reporting formats

Different stakeholders benefit from different outputs:

#### Generating SARIF with axe‑core

When using Playwright, you can invoke axe and output SARIF directly:

import json, subprocess, pathlib

def run_axe_and_save_sarif(page, out_path):
    # Inject axe core
    page.add_init_script(path=pathlib.Path("node_modules/axe-core/axe.min.js").read_text())
    # Run analysis
    result = page.evaluate("""() => {
        return axe.run(null, {
            resultTypes: ['violations'],
            tags: ['wcag2a', 'wcag2aa']
        });
    }""")
    # Convert to SARIF (simplified)
    sarif = {
        "version": "2.1.0",
        "runs": [{
            "tool": {"driver": {"name": "axe-core", "informationUri": "https://github.com/dequelabs/axe-core"}},
            "results": []
        }]
    }
    for v in result["violations"]:
        sarif["runs"][0]["results"].append({
            "ruleId": v["id"],
            "level": "error" if v["impact"] in ["critical", "serious"] else "warning",
            "message": {"text": v["description"]},
            "locations": [{"physicalLocation": {
                "artifactLocation": {"uri": v["nodes"][0]["target"]},
                "region": {"startLine": 1}
            }}]
        })
    pathlib.Path(out_path).write_text(json.dumps(sarif, indent=2))

Upload the resulting .sarif file as a GitHub Action artifact; the platform will automatically annotate the PR with any violations.

Gating strategies

Not all accessibility issues warrant blocking a merge. Define a severity policy:

• Block on: WCAG 2.1 AA violations with impact critical or serious impact that affect core flows (login, checkout, signup).
• Warn on: moderate impact or violations in non‑critical screens (e.g., about page, marketing banners).
• Ignore on: low impact or known false positives that have been triaged and added to a baseline.

Implement the gate as a step in the workflow that checks the SARIF file:

- name: Fail on critical accessibility violations
  id: check-sarif
  run: |
    python - <<'PY'
    import json, sys, pathlib
    sarif_path = pathlib.Path("artifacts/axe.sarif")
    data = json.loads(sarif_path.read_text())