How to Test Onboarding and First-Run Experience

New users form an opinion about an application within the first few minutes of interaction. The onboarding flow is the digital equivalent of a first impression; it decides whether the app will be expl

Motivation

New users form an opinion about an application within the first few minutes of interaction. The onboarding flow is the digital equivalent of a first impression; it decides whether the app will be explored further or abandoned. From permission dialogs that ask for camera access to the first tutorial slide, every touchpoint shapes the user’s perception of ease of use, trustworthiness, and value.

When a product team launches a new feature or a redesigned onboarding sequence, the stakes are high. A poorly handled permission request can trigger immediate uninstalls, while a missing tooltip can leave users stuck on an empty state. The challenge is to verify that each step works as intended across a variety of device configurations, network conditions, and user personas.

From a testing perspective, onboarding is a moving target. It involves multiple disciplines: UI verification, state management, deep‑link handling, and user‑flow instrumentation. The process must cover both the happy path and the edge cases that only surface when the app is killed, resumed, or accessed via an external link. A comprehensive testing strategy therefore blends manual exploratory testing with automated scripts that can be re‑run in CI pipelines.

The goal of this guide is to provide engineers and QA professionals with a concrete, actionable roadmap for validating onboarding and first‑run experiences. It outlines a test matrix, describes manual and automated techniques, highlights production‑only edge cases, and ends with a quick checklist and takeaways that can be bookmarked for future releases.

---

Defining the Onboarding Test Matrix

A test matrix serves as a living document that maps each onboarding scenario to its verification method, priority, and ownership. The matrix below aggregates the most common onboarding dimensions: permission priming and denial, empty states, coach marks, progress checklists, deferred signup, deep‑link entry, resume‑after‑kill, and activation instrumentation.

How to use the matrix

• Priority reflects impact on retention and compliance. High‑priority items are mandatory for each release.
• Verification Method indicates whether the check is manual, automated, or a combination.
• Owner ensures clear responsibility; cross‑functional handoffs are documented in a shared spreadsheet that feeds the matrix.

When a new onboarding variant is introduced, the matrix is updated before any test execution. This prevents gaps and ensures that regression testing covers the same scenarios as the baseline.

---

Manual Testing Techniques

Manual testing remains indispensable for uncovering subtle UX friction that scripts cannot predict. The following techniques are proven to surface issues early, especially when combined with a novice persona mindset.

1. Persona‑Driven Exploration

Create a test plan that mirrors the behavior of a “novice” user: hesitant clicks, accidental gestures, and a tendency to read every label. Walk through the onboarding flow on a real device, noting moments where the user pauses, asks “what is this?”, or abandons the flow.

• Tip: Record the session with a screen‑recorder (e.g., adb shell screenrecord). The footage becomes a reference for developers to reproduce UI timing bugs.

2. Permission Dialog Scripting

Manually trigger each permission scenario: Grant, Deny (first time), Deny (later via Settings), and “Never ask again”. Verify that the app’s UI adapts gracefully in each case.

• Checklist:
• Dialog title matches platform conventions.
• Rationale text is clear and concise.
• “Deny” button leads to a helpful fallback or settings deep‑link.
• Analytics event permission_response captures the choice.

3. Empty‑State Validation

When permissions are denied or data is unavailable, the empty state must be informative and actionable. Manually test:

• Presence of an icon that matches the app’s design system.
• Descriptive text that explains why the content is empty.
• Primary CTA that is accessible (color contrast, touch target size).
• Tooltip or inline help that appears on first interaction with the empty view.

4. Coach Mark Interaction

Simulate a user who never reads tooltips. Tap the “next” button on a coach mark and ensure the tutorial advances. Then tap “skip” and verify that the user lands on the home screen without completing the tutorial (if skip is allowed).

5. Progress Checklist Verification

During manual walkthroughs, tick off each step and watch the progress bar animation. Verify that the checkmarks persist after app backgrounding and that the UI does not flicker or lose state.

6. Deferred Signup Flow

Select “Start Later” at the welcome screen and navigate back to the home screen. Confirm that no authentication token is present and that the user can still interact with non‑auth features.

7. Deep‑Link Injection

Use a browser to open myapp://onboard?step=3 and observe the app launch. If the deep‑link fails, manually inspect the intent filter in the AndroidManifest and the URI scheme handling.

8. Resume‑After‑Kill

Kill the app via the recent tasks manager, relaunch, and confirm that the last viewed step is restored. Document any data loss or UI glitches; these often stem from improper onSaveInstanceState handling.

9. Accessibility Audit

Run VoiceOver/TalkBack and navigate each onboarding screen. Ensure that each interactive element has a descriptive label, that focus order follows a logical progression, and that button sizes meet WCAG 2.2 AA guidelines.

10. Power‑User Shortcut

Log in with an existing account and open a deep‑link that should bypass onboarding. Verify that the app respects the user’s session and does not present onboarding UI.

All manual steps should be logged in a shared test case management tool (e.g., TestRail). Attach screenshots, videos, and notes to each entry. This creates a knowledge base that helps new team members understand common pitfalls.

---

Automated Testing Strategies

Automation accelerates regression coverage and provides a safety net for changes to the onboarding flow. The following strategies combine UI testing frameworks, API checks, and native tools.

1. UI Automation with Playwright (Web)

For web‑based onboarding (e.g., progressive web apps), Playwright offers reliable cross‑browser control. A typical script verifies the tutorial carousel and permission dialogs.

# Install dependencies
npm init -y
npm install @playwright/test
npx playwright install

Create tests/onboarding.spec.ts:

import { test, expect } from '@playwright/test';

test.describe('Onboarding flow', () => {
  test.beforeEach(async ({ page }) => {
    await page.goto('/onboarding');
  });

  test('should display first slide and allow skip', async ({ page }) => {
    const slide = page.locator('[data-testid="slide-0"]');
    await expect(slide).toBeVisible();
    await page.click('[data-testid="skip-button"]');
    await expect(page).toHaveURL('/');
  });

  test('permission dialog appears on camera request', async ({ page }) => {
    // Trigger camera permission by clicking the button that requests it
    await page.click('[data-testid="camera-button"]');
    const dialog = page.locator('[role="dialog"]');
    await expect(dialog).toBeVisible();
    await expect(dialog).toContainText('Camera access');
  });
});

Run the suite in CI:

npx playwright test --config playwright.config.ts

2. Native Mobile Automation with Appium

For Android and iOS, Appium scripts simulate taps, swipes, and text entry. Below is a Python snippet that validates the permission priming flow.

from appium import webdriver
from appium.options.android import UiAutomator2Options
import os

desired_caps = {
    "platformName": "Android",
    "deviceName": "emulator-5554",
    "appPackage": "com.example.app",
    "appActivity": "com.example.app.SplashActivity",
    "automationName": "UiAutomator2",
}
options = UiAutomator2Options().load_capabilities(desired_caps)
driver = webdriver.Remote("http://localhost:4723", options=options)

# Wait for permission dialog
from appium.webdriver.common.app_wait import AppWait
wait = AppWait(driver)
wait.until(lambda d: d.find_element_by_id("com.example.app:id/permission_dialog"))

# Verify dialog title
dialog_title = driver.find_element_by_id("com.example.app:id/permission_title").text
assert "Camera" in dialog_title

driver.quit()

3. SUSA (SUSATest) for Scriptless Exploration

SUSA provides an autonomous QA platform that uploads an APK or points to a web URL and explores the app without the need for handcrafted scripts. It executes a range of user personas, including a “novice” persona that mimics the confusion real users experience.

# Install the agent
pip install susatest-agent

# Run a full onboarding scan
susatest-agent run --apk app-debug.apk --persona novice --output-dir ./susa-results

The output includes a summary of crashes, ANRs, dead buttons, and accessibility violations discovered during the exploration. SUSA can also auto‑generate regression scripts (Appium + Playwright) from the discovered flows, making it a powerful complement to existing test suites.

4. API‑Level Validation

Onboarding often triggers backend events (e.g., user_onboarding_completed). Use a tool like Postman or Insomnia to assert that the correct payload is sent after each step.

# Example using curl
curl -X POST https://api.example.com/events \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"event":"user_onboarding_completed","step":"3","timestamp":"$(date -u +%Y-%m-%dT%H:%M:%SZ)"}'

5. Visual Regression Testing

Changes to onboarding UI can be unintentionally broken by design updates. Tools such as BackstopJS or Percy capture screenshots and compare them against baselines.

// backstop.json snippet
{
  "scenarios": [
    {
      "label": "Onboarding welcome slide",
      "url": "/onboarding",
      "viewportSizes": ["Desktop", "Mobile"]
    }
  ]
}

Run the suite:

npm install -g backstopjs
backstop test

6. Cross‑Session Learning

When using SUSA, note its cross‑session learning capability. The platform retains knowledge of explored screens and dead ends, making subsequent runs faster and more targeted. This is especially valuable for iterative onboarding redesigns where each iteration builds on the previous one.

---

Permission Priming and Denial Paths

Permission priming refers to the way an app cues users before requesting access to device features (camera, location, contacts). The priming message, timing, and subsequent denial handling directly affect user trust and retention.

1. Priming Message Content

• Best practice: Align the priming text with platform conventions. On Android, use the phrasing suggested by Google’s Material Design: “This app needs camera access to scan QR codes.”
• Verification: Automated accessibility checks ensure that the priming text is readable by screen readers. Manually verify that the tone is friendly, not demanding.

2. Contextual Timing

If the permission is required for a feature that is not immediately visible, delay the request until the user attempts to use that feature. This reduces permission fatigue.

• Test scenario: Launch the app, navigate to the “Scan” screen, and click the scan button. Verify that the permission dialog appears only after the button click.

3. Denial Flow

When a user denies a permission, the app must provide a clear path to Settings and explain why the permission is needed.

• Deep‑link to Settings: On Android, use Intent.ACTION_APPLICATION_SETTINGS with the package name. On iOS, open Settings.app via UIApplication.openSettingsURLString.

// iOS Swift example
if let settingsUrl = URL(string: UIApplication.openSettingsURLString) {
    UIApplication.shared.open(settingsUrl)
}

• Analytics logging: Record onboarding_permission_denied with the permission name and timestamp.

4. “Never Ask Again” Handling

If the user selects “Never ask again”, the app should gracefully degrade functionality and present an alternative workflow.

• Fallback UI: Show an empty state with a message like “Camera access is required for scanning. Please enable it in Settings.”
• CTA: Include a button that deep‑links directly to the permission settings screen.

5. Permission State Persistence

After a user grants a permission, the app must remember the choice across app kills and device reboots. Verify by killing the app (adb shell am force-stop com.example.app), waiting, and relaunching. The permission‑dependent feature should be usable without another prompt.

---

Empty States, Tooltips, and Coach Marks

Empty states occur when there is no data to display, often because of missing permissions or a new user’s first visit. Tooltips and coach marks guide users through non‑intuitive interactions.

1. Designing Effective Empty States

• Iconography: Use a universally understood icon (e.g., an image placeholder) that matches the app’s brand.
• Copy: Keep it concise—“You haven’t added any items yet.” Add a secondary line if needed: “Tap ‘Add’ to get started.”
• CTA: Ensure the primary button is large enough for thumb reach (minimum 44 × 44 dp on mobile).

2. Tooltips vs. Coach Marks

• Tooltips: Appear on demand, often triggered by a long‑press or help icon. They should disappear after a few seconds or user interaction.
• Coach Marks: Full‑screen walkthroughs that highlight key UI elements, typically used for onboarding. They should be skippable and respect the device’s notch area.

3. Implementation Validation

#### Android (XML) Example

<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/coach_container">
    <TextView
        android:id="@+id/coach_text"
        android:text="Drag the slider to adjust the value"
        android:visibility="gone" />
    <com.example.app.CoachMarkView
        android:id="@+id/coach_mark"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        app:target="@+id/slider"
        app:message="@string/coach_drag" />
</FrameLayout>

#### iOS (Swift) Example

let coachMark = CoachMark(
    target: slider,
    message: "Slide to set your preference",
    arrowPosition: .bottom
)
coachMark.show()

4. Automated Checks

• Visibility: Use UI automators to assert that the tooltip or coach mark is displayed when expected.
• Dismissal: Simulate a tap outside the tooltip and verify that it disappears.
• Accessibility: Run axslint or ios-accessibility-checker to ensure that the message is announced correctly.

5. Edge Cases

• Orientation changes: Coach marks should reposition correctly when the device rotates.
• Keyboard appearance: If a tooltip is placed near an input field, ensure it does not get obscured.
• Network latency: Simulate a slow connection and verify that tooltips still appear after the UI thread is ready.

---

Progress Checklists and Deferred Signup

Progress checklists give users a visual cue of how much of the onboarding journey remains. Deferred signup allows users to postpone account creation while still exploring core features.

1. Checklist Implementation

• Data storage: Store checklist state in a lightweight key‑value store (e.g., SharedPreferences, AsyncStorage) to survive app backgrounding.
• Atomic updates: When a step is completed, update the checklist and emit an analytics event (onboarding_step_completed).

#### Example (React Native)

const markStepComplete = async (stepId) => {
  await AsyncStorage.setItem(`step_${stepId}_completed`, 'true');
  await analytics.track('onboarding_step_completed', { stepId });
};

2. Visual Feedback

• Progress bar: Width should animate smoothly; abrupt jumps cause user confusion.
• Checkmark animation: Use a subtle bounce or fade‑in to confirm completion.

3. Deferred Signup Flow

• Trigger: “Start Later” button on the welcome screen or after a certain number of steps.
• State: User is not assigned a user ID, but the onboarding progress is saved under an anonymous identifier (e.g., anon_12345).

#### Backend API Example

// POST /onboarding/start-deferred
{
  "anonymousId": "anon_12345",
  "skippedAtStep": 2,
  "createdAt": "2024-09-01T12:34:56Z"
}

• Resumption: When the user later creates an account, merge the anonymous progress with the authenticated session. This requires a backend endpoint (/onboarding/merge) that updates the user’s onboarding step based on stored anonymous data.

4. Automated Validation

• Checklist persistence: Write an automation test that completes step 1, kills the app, and reopens to verify that step 1 is still marked as completed.
• Deferred signup: Simulate a user clicking “Start Later”, then log in with credentials and assert that the user lands on the home screen with no pending onboarding steps.

5. Persona Considerations

• Novice user: May not understand that skipping creates an anonymous profile. Provide a clear explanation: “You can finish later when you’re ready to create an account.”
• Power user: Expect a seamless merge without extra prompts. Ensure the backend merge operation is silent and does not trigger UI interruptions.

---

Deep‑Link Entry, Resume‑After‑Kill, and Activation Instrumentation

Deep‑linking enables external contexts (e.g., email links, advertisements) to land users directly into a specific onboarding step. Resume‑after‑kill ensures that the app respects its saved state after being terminated. Activation instrumentation captures the “aha moment” when a user transitions from onboarding to core product usage.

1. Deep‑Link Architecture

#### AndroidManifest

<activity
    android:name=".OnboardingActivity"
    android:exported="true">
    <intent-filter>
        <action android:name="android.intent.action.VIEW" />
        <category android:name="android.intent.category.DEFAULT" />
        <data android:scheme="myapp" android:host="onboard" />
    </intent-filter>
</activity>

#### iOS Info.plist (Swift)

let scene = UIWindowSceneSession.role(.windowScene)
let url = URL(string: "myapp://onboard?step=2")
if let url = url {
    UIApplication.shared.open(url, options: [:])
}

2. Parsing Deep‑Link Parameters

When the app receives a deep‑link, extract query parameters (step, ref, utm_source). Validate that the requested step exists; otherwise, default to the first step.

fun handleDeepLink(uri: Uri) {
    val stepParam = uri.getQueryParameter("step")
    val step = stepParam?.toIntOrNull() ?: 1
    navigateToStep(step)
}

3. Resume‑After‑Kill Logic

#### Android (onCreate)

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_onboarding)

    val savedStep = savedInstanceState?.getInt(STEP_KEY) ?: intent?.let {
        handleDeepLink(it.dataString ?: "")
    } ?: 1

    displayStep(savedStep)
}

#### iOS (viewWillAppear)

override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    let step = savedStep ?? deepLinkStep()
    showStep(step)
}

4. Activation (Aha) Instrumentation

Activation is measured by a combination of behavioral and technical signals. Typical triggers include completing the onboarding checklist, making a first API call, or viewing a key feature tutorial.

#### Event Payload Example

{
  "event_type": "activation_completed",
  "user_id": "12345",
  "timestamp": "2024-09-01T12:34:56Z",
  "context": {
    "step_completed": 5,
    "time_to_complete_seconds": 78,
    "device_os": "android",
    "app_version": "3.2.1"
  }
}

• Backend acceptance: Ensure the analytics endpoint validates the schema and stores the event for downstream reporting.

5. Automated Tests for Deep‑Link and Resume

Write a test that launches the app via an intent and asserts the correct step is displayed. Below is a Python/Appium snippet.

from appium import webdriver
from appium.options.android import UiAutomator2Options

caps = {
    "platformName": "Android",
    "deviceName": "emulator-5554",
    "appPackage": "com.example.app",
    "appActivity": "android.intent.action.VIEW",
    "automationName": "UiAutomator2",
}
options = UiAutomator2Options().load_capabilities(caps)
driver = webdriver.Remote("http://localhost:4723", options=options)

# Simulate deep-link
driver.start_activity("com.example.app", "com.example.app.OnboardingActivity")
driver.execute_script('mobile: deepLink', {
    "url": "myapp://onboard?step=3",
    "package": "com.example.app"
})

# Verify step 3 is shown
step_label = driver.find_element_by_xpath("//android.widget.TextView[@text='Step 3']")
assert step_label.is_displayed()
driver.quit()

6. Persona‑Specific Deep‑Link Paths

• Curious user: May click a link from a social share. Expect the app to skip the welcome screen and start at the step highlighted by the link.
• Impatient user: May use a short link that bypasses onboarding entirely (myapp://home). The app must detect a valid session and avoid showing onboarding UI.
• Adversarial user: May attempt malformed deep‑links (myapp://onboard?step=999). The app should gracefully fall back to the first step and log a warning.

7. SUSA Integration (Optional)

When using SUSA, enable the deep‑link exploration mode to automatically test external entry points. This ensures that deep‑links work from third‑party sources and that the app does not crash under unexpected URI formats.

susatest-agent run --apk app-debug.apk --deep-link true --persona curious

---

Edge Cases That Appear Only in Production

Even with comprehensive test coverage, certain onboarding issues surface only when the app lives in the real world. These edge cases often involve network interruptions, OS-level behaviors, and device-specific quirks.

1. Network Partition During Permission Grant

A user may grant camera permission, but the network request to upload the image fails due to poor connectivity. The app should cache the image locally and retry on reconnection, showing a subtle loading indicator.

• Testing tip: Use Charles Proxy or Burp Suite to simulate dropped packets while automating a permission grant and image capture.

2. OS Permission Revocation

Android Oreo and later allow users to revoke permissions from the app settings while the app is in the background. If the app resumes and attempts to use the revoked permission, it must handle the SecurityException gracefully and present a recovery flow.

try {
    val bitmap = cameraManager.takePicture()
    // process image
} catch (e: SecurityException) {
    showPermissionRecoveryDialog()
}

3. Device Shutdown While Onboarding

If the device powers off mid‑onboarding, the app’s saved instance may be incomplete. Verify that after reboot, the UI does not show corrupted state (e.g., half‑filled input fields). This requires checking onRestoreInstanceState and clearing invalid data.

4. Accessibility Service Conflicts

Some users install third‑party accessibility services that interfere with the onboarding coach marks. The app should detect when a screen reader is active and adapt coach marks to a non‑visual guidance method (e.g., sound cues).

• Automated check: Use uiautomator to query AccessibilityService status and assert that the onboarding flow still progresses.

5. Memory Pressure and ANR

During onboarding, the app may allocate heavy resources (e.g., loading large tutorial images). Under memory pressure, the UI thread can stall, causing an ANR.

• Instrumentation: Enable strict mode (adb shell setprop debug.strictmode true) and run the onboarding flow to capture any UI freezes.

6. Time‑Zone and Date‑Formatting Edge Cases

If onboarding includes a date picker, ensure that the displayed format respects the device’s locale and does not break after daylight‑saving transitions.

7. Back‑Button Handling in Coach Marks

Some devices have a hardware back button that triggers the skip action unintentionally. Implement a confirmation dialog for skip to prevent accidental exits.

8. Multilingual Onboarding

Non‑Latin scripts may cause text truncation in UI elements. Automated layout tests should verify that text overflow is handled (e.g., ellipsize="end").

9. Biometric Authentication Prompt

If the onboarding flow includes a biometric setup (e.g., fingerprint enrollment), the system may present the prompt at an unexpected moment (e.g., during a tutorial). Ensure the prompt respects the onboarding step and does not disrupt the flow.

10. Crash Reporting Integration

Enable crashlytics or similar SDK and verify that any onboarding crash is captured with sufficient context (e.g., step number, user persona). This requires instrumentation of custom exception handlers.

---

Quick Checklist for Engineers

Use this checklist as a sprint‑level artifact. Mark each item as *Pass*, *Fail*, or *Not Tested*. Items that fail should be logged with a reproducible bug ID and assigned to the appropriate owner.

---

Closing Takeaways

Onboarding is more than a series of screens; it is a critical user‑journey that determines whether a new user becomes an active, retained customer. The complexity arises from the need to handle permissions, guide users through non‑intuitive actions, preserve state across device events, and trigger meaningful activation moments.

A robust testing strategy combines manual exploratory testing—which uncovers subtle UX friction that automated tools miss—with automated validation that ensures consistency across releases. The test matrix serves as the backbone, aligning every scenario with a clear verification method and ownership.

When implementing permission priming, remember that the user’s decision is influenced by timing, clarity, and the ease of recovery if they choose to deny. Empty states must be both informative and actionable, supported by tooltips or coach marks that respect accessibility and device constraints.

Progress checklists and deferred signup flows add layers of flexibility, but they also introduce state‑management challenges that must be tested under real‑world conditions (network loss, OS permission revocation, memory pressure). Deep‑link entry and resume‑after‑kill scenarios demand careful handling of intents and saved instance states, while activation instrumentation provides the quantitative signal that the onboarding succeeded.

Edge cases that only surface in production—such as OS permission revocation, biometric interruptions, or multilingual layout issues—require ongoing monitoring and a feedback loop between analytics, crash reporting, and the QA team. Tools like SUSA can accelerate this loop by autonomously exploring the app from the perspective of a novice persona, generating regression scripts, and highlighting accessibility violations.

By internalizing the checklist and embedding the test matrix into the development workflow, engineering teams can maintain a high confidence level that onboarding works as intended across all user personas and device configurations. The result is a smoother first‑run experience, higher conversion rates, and a stronger foundation for long‑term user retention.

---

*End of article.*