qa-testing-android

QA Testing (Android)

Android testing automation with Espresso, UI Automator, Compose Testing, screenshot tests, and adaptive UI validation.

Core References: Android Testing Docs, Build-Managed Devices, Compose Testing, UI Automator, Screenshot Testing, Accessibility Checking

Quick Reference

Platform Versions (July 2026)

Google Play Target API Policy (verify before every release cutover)

As of the 2026-08-31 deadline: new apps and app updates submitted to Google Play must target API level 36 (Android 16) or higher — submissions targeting lower are rejected in Play Console. Apps not updated at all must still target at least API level 35 (Android 15) or they become invisible/uninstallable for new users on newer OS versions. A one-time extension to 2026-11-01 is available by request. This is a moving deadline — re-check https://developer.android.com/google/play/requirements/target-sdk before treating any specific API number as "the" requirement, since Google raises it roughly once a year.

Quick Start

• Prefer build-managed devices (GMD) for CI, using ATD images where the suite and supported API levels allow it; use connectedAndroidTest for local ad-hoc runs.
• ATD images only exist for API 30. For API 35/36 suites, use google or aosp images.
• Enable Android Test Orchestrator and clearPackageData for instrumented suites that need strong isolation.
• Disable animations in Gradle, not with ad-hoc ADB steps inside the test flow.
• Treat screenshot tests as a separate fast regression layer: Compose Preview Screenshot Testing for @Preview coverage, Paparazzi or Roborazzi for JVM rendering, device-based snapshots only when hardware fidelity matters.
• Cover adaptive layouts explicitly: compact and expanded layouts, tablet or foldable paths when supported, and orientation changes where the product depends on them.
• Add accessibility checks to existing UI tests early instead of treating accessibility as a manual-only phase.

Recommended Gradle defaults for stable instrumented tests:

android {
    testOptions {
        animationsDisabled = true
        execution = "ANDROIDX_TEST_ORCHESTRATOR"

        emulatorSnapshots {
            enableForTestFailures = true
            maxSnapshotsForTestFailures = 2
        }
    }
}

dependencies {
    androidTestUtil(libs.androidx.test.orchestrator)
}

If you rely on test isolation between instrumented tests, also set runner args such as clearPackageData=true in your Gradle or CI wiring.

When to Use

• Debug or stabilize flaky Android UI tests
• Add Espresso tests for View-based UIs
• Add Compose UI tests for composables
• Add UI Automator tests for system UI, cross-app flows, or macrobenchmark drivers
• Add screenshot tests for visual regression
• Validate adaptive layouts across screen sizes, window states, or foldable postures
• Add an Android test gate in CI

Inputs to Gather

• UI stack: Views, Compose, or mixed
• Test layer: unit, Robolectric, instrumented UI, screenshot, UI Automator, macrobenchmark
• CI target: PR gate vs nightly vs release; managed emulator vs device farm
• Device matrix: min supported API, target API, screen classes, locales, dark mode, form factors
• Flake symptoms: timeouts, missing nodes, idling or sync issues, renderer-only diffs, device-only failures
• Accessibility requirements: touch targets, labels, contrast, TalkBack-critical flows
• App seams: DI hooks for fakes, feature flags, test accounts, deterministic clocks, image loaders

Testing Layers

Core Principles (Stability)

Device Matrix

• Default: emulators for PR gates; real devices or device farms for release-critical journeys.
• Cover at least min supported API and target API, then add tablet or foldable coverage if the product exposes expanded layouts.
• Model adaptive coverage by layout behavior, not by device marketing names alone.

Flake Control

• Prefer testOptions { animationsDisabled = true } for instrumented tests.
• Use Android Test Orchestrator when shared app state or process crashes cause cross-test leakage.
• Use IdlingResources, Compose synchronization, waitUntil, or UI Automator conditions instead of sleeps.
• Mock network with MockWebServer or DI fakes; avoid live backends in CI.
• Keep screenshot tests deterministic: fixed fonts, locale, clocks, network images, and seeded data.
• Use test-failure snapshots or screenshots in CI so failures have artifacts, not just stack traces.

Accessibility And Adaptive Coverage

• Run accessibility checks in the same suite as high-value Espresso or Compose journeys.
• Validate compact and expanded layout states intentionally; do not assume a phone-only test matrix covers responsive layouts.
• Prefer stable selectors: withId() for Views, testTag for Compose, and resource-id or content descriptions for UI Automator.

Expert Judgment (What A Checklist Misses)

Emulator vs. Physical Device — the actual decision

A checklist says "use emulators for CI, real devices for release." The judgment call is which real-device signals are worth paying for:

• Emulators (ideally build-managed, ATD where hardware fidelity is not required) are correct for PR gates because they are deterministic, disposable, and free of thermal/battery/OEM noise that would otherwise get misattributed to app bugs.
• Physical devices or a device farm earn their cost only for journeys where OEM or hardware behavior is the actual risk: camera capture, biometric auth, NFC/BLE, background-work reliability under real Doze/App Standby, foldable hinge/posture transitions, and payment SDKs that behave differently under real Play Integrity checks. If none of those are in the flow, a device farm run is theater, not signal.
• A common anti-pattern is running the entire regression suite nightly on a device farm "for confidence." This burns budget without changing the failure mode you'd actually catch — most of what fails there also fails, faster and cheaper, on a managed emulator. Reserve device-farm minutes for the handful of hardware-coupled journeys above and gate them separately from the general regression suite.
• Emulator system-image choice is itself a judgment call, not just "google vs aosp": Google Play Services-dependent flows (Play Billing, FCM, Play Integrity, Maps) need google images; anything asserting on hardware-rendered pixels needs a non-ATD image; everything else should default to ATD for speed.

Android Flake Taxonomy (diagnose by category, not by symptom)

Generic "flaky test" triage wastes time re-running instead of classifying. Android UI-test flakiness clusters into a small number of root-cause families — identify which one you're looking at before reaching for retries:

1. Synchronization gaps — an async operation Espresso/Compose does not know how to wait for. Signature: passes on a fast local machine, fails under CI load; NoMatchingViewException or an assertion firing against a stale loading state. Root cause is almost always an unregistered IdlingResource (Espresso) or a missing waitUntil/synchronized TestDispatcher (Compose/coroutines) — see references/espresso-patterns.md for the registration trap.
2. Animation/transition timing — window or property animations still running when Espresso samples the view tree. Fix at the Gradle level (animationsDisabled = true), never per-test with sleeps; ad-hoc adb shell settings put global *_scale 0 inside a test body is a workaround that silently stops working if the runner changes.
3. Semantics-tree merging (Compose-specific) — assertExists()/onNodeWithText() finds nothing because Material components merge child semantics into the parent node by default. Not a timing issue at all; do not "fix" it with a wait loop. See the merged-tree trap in references/compose-testing.md.
4. State leakage across tests — a prior test left SharedPreferences, a DB row, or a singleton in a state the next test doesn't expect. Symptom is order-dependent failures that disappear when the failing test is run alone. Fix with Test Orchestrator + clearPackageData, not by reordering tests.
5. Doze / App Standby / background restrictions — tests that rely on background work (WorkManager, FCM, foreground services) can behave differently under device power-management states than in a freshly booted emulator. This shows up almost exclusively on physical devices or long-running CI machines, rarely on ephemeral managed devices — if a background-work test is flaky only in one environment, suspect power-management state before suspecting the test.
6. Renderer/fidelity mismatches — screenshot diffs that are real but meaningless: ATD disabling hardware rendering, GPU driver differences between CI and local, or font substitution differences across OS images. Distinguish this from an actual visual regression by re-running the same golden generation pipeline on the same image type used to record it.
7. Non-deterministic test data or clocks — live network calls, real System.currentTimeMillis(), or unseeded random data leaking into assertions or screenshot goldens. Always mockable; the fact that it's still happening usually means a fake wasn't reused when a new screen was added.

When a test is flaky, name which of the seven it is before touching the test — the fix for #1 (register/await) will not touch #4 (isolate/clear state), and applying #2's fix (disable animations) to a #5 issue (Doze) does nothing.

Test Pyramid Reality For Compose Apps

The classic pyramid (many unit tests, some integration, few E2E) still holds, but Compose changes where the middle layer sits:

• Compose lets you unit-test a composable's rendered semantics tree via createComposeRule() without an Activity or device at all — this is functionally a unit test even though it "looks like" a UI test. Prefer it over createAndroidComposeRule<Activity>() whenever the composable doesn't need real navigation, DI graph, or activity lifecycle.
• This shifts a large share of what used to be slow instrumented Espresso coverage down into fast, parallelizable JVM-adjacent tests — but only if ViewModels are tested independently (Turbine/MockK against StateFlow, not through the UI). A Compose app that only has "instrumented Compose tests that also exercise the ViewModel" has recreated the old inverted pyramid with new tools.
• Screenshot tests (Compose Preview Screenshot Testing / Paparazzi / Roborazzi) form a separate, parallel layer, not a replacement for either unit or instrumented behavioral tests — they catch visual regressions that behavioral assertions cannot see (spacing, overlap, dark-mode contrast) but say nothing about correctness of state transitions.
• Reserve UI Automator and Maestro-style E2E flows for the top of the pyramid: a handful of true cross-app or full-stack smoke journeys (onboarding, checkout, deep-link entry), run on nightly/release pipelines, not PR gates.

Device / API Fragmentation Strategy

• Always cover the app's actual min supported API and its current target API — these two are non-negotiable regardless of market share.
• Beyond that pair, prioritize by behavioral risk, not by device popularity: API levels that changed permission models, background-execution limits, or storage scoping (e.g., scoped storage, notification runtime permission, foreground-service type declarations) deserve dedicated coverage on the OS version that introduced the change, even if that OS version is not the most common one in your install base.
• Treat any specific device/API market-share number as unverified unless sourced from your own analytics (Play Console's own device catalog and Android Vitals for your app) — public aggregate stats go stale within months and vary enormously by app category and geography. Do not hardcode "X% of users are on API N" into test-planning docs; pull it fresh from the app's own Play Console dashboard each planning cycle.
• Foldable/tablet/expanded-layout coverage is now a fragmentation axis in its own right, independent of API level — a phone-only test matrix increasingly under-covers real usage on large-screen and foldable devices even when API coverage looks complete.

When To Invest In Screenshot Testing

Screenshot testing has real setup and maintenance cost (goldens go stale, false positives from font/renderer drift, review burden on every intentional UI change). It is worth that cost when:

• The team ships UI changes frequently enough that manual visual review does not scale (multiple PRs/day touching shared components).
• The app has a design system or component library where a single regression fans out across many screens — screenshot tests catch that fan-out cheaply; behavioral tests do not.
• Dark mode, large font scale, or RTL layouts are supported and have historically regressed silently.

It is a poor early investment for a small team still iterating rapidly on visual design — churn in intentional goldens will dominate signal from real regressions. Start with host-side tools (Compose Preview Screenshot Testing, Paparazzi, Roborazzi) for cheap iteration; only add device-based snapshot testing (Shot, or ATD-excluded device runs) once a specific rendering-sensitive surface (WebView, Maps, camera preview, custom Canvas/GPU work) has already caused a shipped visual bug that host-side rendering could not have caught.

Writing Tests

• Espresso (Views): open references/espresso-patterns.md
• Compose: open references/compose-testing.md
• UI Automator (system or cross-app): open references/uiautomator.md
• Screenshot tests: open references/screenshot-testing.md
• Adaptive screen-size testing: open references/adaptive-screen-testing.md
• Accessibility checks: open references/accessibility-checks.md

Workflow

Add a New UI Test (Instrumented)

• Pick the narrowest framework that can observe the behavior: Espresso or Compose first, UI Automator at the system boundary.
• Add stable selectors: View id, Compose Modifier.testTag, system resource-id or content description.
• Control externals: fake network, deterministic data, fixed time and locale, and stable image sources.
• Add waits through framework synchronization or explicit conditions; avoid Thread.sleep().
• Run locally with connectedAndroidTest or a single managed-device task before widening the matrix.

Add Screenshot Coverage

• Use Compose Preview Screenshot Testing for @Preview-driven Compose UI states.
• Use Paparazzi or Roborazzi when you need JVM-fast coverage across Compose and legacy Views.
• Keep goldens small, reviewable, and tied to stable UI states instead of whole-screen snapshots of volatile content.
• Capture HTML or diff artifacts in CI on every failure.

Diagnose a Flaky Instrumented Test

• Reproduce locally or on one managed device before widening the matrix.
• Remove nondeterminism: network, locale, clock, feature flags, image loading, renderer differences.
• Replace sleeps with idling, waitUntil, onElement, or watcher-based synchronization.
• Capture logcat, screenshot, screen recording, and any managed-device failure snapshot.
• If still flaky, isolate app state further with Orchestrator or runner args and bisect the interaction steps.

Add A CI Gate (Preferred: Build-Managed Devices)

• Configure build-managed devices plus ATD images when hardware rendering is not required and the API level is supported by the image.
• Add the GitHub Actions GPU flag when your runner lacks hardware rendering support.
• Keep PR gates small and deterministic; expand device groups or sharding on nightly and release pipelines.
• Upload reports from build/reports/androidTests/, screenshot reports, diff images, logcat, and managed-device outputs.

ADB Commands (Triage)

# Screenshot
adb exec-out screencap -p > screenshot.png

# Screen recording
adb shell screenrecord /sdcard/demo.mp4

# Pull managed-device test artifacts after a local failure
adb pull /sdcard/Android/media ./device-artifacts

CI Integration

Preferred: build-managed devices. See references/gradle-managed-devices.md and references/android-ci-optimization.md.

# .github/workflows/android.yml
name: Android CI
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6  # or later — verify at actions/checkout releases
      - uses: actions/setup-java@v5  # or later
        with:
          java-version: '17'  # or later
          distribution: 'temurin'
      - uses: gradle/actions/setup-gradle@v5  # or later
      - run: ./gradlew testDebugUnitTest <device><api>DebugAndroidTest -Pandroid.testoptions.manageddevices.emulator.gpu=swiftshader_indirect

ASCII Flow

Android testing request
  -> Classify layer: unit, Robolectric, Compose, Espresso, UI Automator, screenshot
  -> Choose device/API matrix from risk, analytics, and adaptive UI needs
  -> Stabilize state, idling, selectors, permissions, and test data
  -> Run local targeted tests before managed-device or connected-device gates
  -> Capture artifacts: logs, screenshots, videos, and test reports
  -> Deflake root cause before expanding CI matrix or retries

Navigation

The reference guides are intentionally large; search within them instead of loading everything:

• rg -n "^## " frameworks/shared-skills/skills/qa-testing-android/references/compose-testing.md
• rg -n "Idling|waitUntil|Synchronization" frameworks/shared-skills/skills/qa-testing-android/references/compose-testing.md
• rg -n "DisplaySizeRule|DeviceConfigurationOverride|fold" frameworks/shared-skills/skills/qa-testing-android/references/adaptive-screen-testing.md
• rg -n "PreviewTest|Paparazzi|Roborazzi|Shot" frameworks/shared-skills/skills/qa-testing-android/references/screenshot-testing.md

Do / Avoid

Do

• Prefer build-managed devices plus ATD images for stable CI
• Use Android Test Orchestrator only where isolation solves a real state-leak or crash problem
• Add screenshot and accessibility checks to high-value UI flows
• Run a small matrix on PRs and widen via groups or shards on nightly or release pipelines
• Use a MainDispatcherRule + TestDispatcher (runTest { }, advanceUntilIdle(), kotlinx-coroutines-test) for every ViewModel and StateFlow test — Dispatchers.setMain(UnconfinedTestDispatcher()) hides off-main crashes that only surface in production
• Run a release-variant smoke test in CI that exercises at least one @Serializable endpoint per data class, so R8 full-mode stripping of kotlinx-serialization $serializer classes is caught before release (see software-android-runtime-debugging/references/proguard-r8-triage.md)
• For Compose tests, assert on semantic state — onNodeWithTag(...).assertTextEquals(...) — not on reference equality of UI state objects, because Strong Skipping Mode means the UI may or may not receive the same instance across emissions

Avoid

• Thread.sleep() for synchronization
• Tests that depend on live backends or time-sensitive external content
• Flaky selectors such as localized text or positional-only nodes
• Device-wide screenshot assertions on ATD when hardware-rendered fidelity is required
• Asserting StateFlow<UiState> reference equality across emissions (assertThat(state).isSameInstanceAs(previous)) — new data class instances from copy() have different references but structurally equal content; test semantic equality, not identity
• Running serialization-dependent tests only on debug builds — R8 is off in debug, so debug-only suites cannot catch serialization keep-rule failures

Resources

Templates

Related Skills

Fact-Checking

• Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
• Use web search or web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.