diff --git a/.github/skills/android-e2e-tester/SKILL.md b/.github/skills/android-e2e-tester/SKILL.md new file mode 100644 index 00000000..e25eee11 --- /dev/null +++ b/.github/skills/android-e2e-tester/SKILL.md @@ -0,0 +1,571 @@ +--- +name: android-e2e-tester +description: "Execute and iterate end-to-end (E2E) tests for an in-development Android Auth feature (MSAL, Broker, Common, ADAL, Authenticator) on an Android emulator or a connected real device. Builds a device pool from emulators and any adb-connected hardware, finds or creates a suitable AVD (or reuses a running emulator / real device), leases the device so concurrent tests don't collide, builds/installs the right test app, drives the UI, and verifies via logcat. Delegates the actual run to a sub-agent (same model as the parent) and waits for its verdict; when several test cases and several devices are available, splits the cases across devices and runs them in parallel (one sub-agent per case), then writes a per-case report plus an overall run summary. Only use this skill when the user explicitly asks to run an end-to-end test — e.g. 'run the E2E test', 'test this feature end to end', 'verify the feature on a device or emulator', 'does this work on a device', 'run this ADO test case', or 'try the sign-in flow'. Do NOT auto-start it after a feature finishes development, when a coding-agent PR lands, or as an implicit step of any other workflow; a human has to ask for an E2E run. Discovers what to test from user-provided steps, an ADO Test Case work item, a known test-steps file, the session's design spec, or the implementation diff — feature-specific steps live in the test case, not the skill. Auto-performs inputs the AI can handle (typing lab test credentials, tapping buttons, granting permissions, simulating a fingerprint, provisioning lab accounts via the LAB API), mocks unavailable dependencies and sets feature flags via temporary code changes when needed, and asks the user when the intent is unclear or a step is a real blocker (push MFA, hardware, missing credentials, implementation gaps). Checks logs to decide pass/fail, drives a fix-and-retest loop until it passes, and — for ADO test cases — always writes an HTML + Markdown test report at the end. Prefers an emulator when a step needs an injectable fingerprint/biometric (App Lock, number-match with biometric gate)." +--- + +# Android E2E Tester + +Take an in-development Android Auth feature and prove it works on a real device or an emulator: provision +the device, deploy the app, run the scenario (auto-handling every input the AI reasonably can), read the +logs to decide pass/fail, and loop fix-and-retest until it passes — or stop and ask when genuinely blocked. +For a test case driven from Azure DevOps, always finish by writing an HTML + Markdown test report. + +## When this runs + +- **Only when the user explicitly asks to run an E2E test** — e.g. "run the E2E test", "test this feature + end to end", "verify the feature on a device/emulator", or "run this ADO test case". This skill does **not** + auto-start after a feature finishes development, when a coding-agent PR lands, or as an implicit step of any + other workflow (including an orchestrated build→test flow) — a human has to ask for an E2E run. Once asked, + run Phase 1 to confirm *what* to test before touching the device. + +## Run folder + +Put all artifacts (log snapshots, screenshots, per-iteration notes) **outside the repo** so nothing is +committed: +``` +$env:USERPROFILE\android-e2e-runs\-\ +``` +Pass this as `-Out` to `authlogs.ps1`/`deviceui.ps1`. Reuse the resolved emulator `-Serial` on every +script call so commands target the right device. + +## Scripts + +All under `scripts/` (PowerShell — the team's cross-platform shell). Run `-?` mentally via the +`.SYNOPSIS` in each file. They self-resolve the SDK; no hardcoded paths. + +| Script | Purpose | Key commands | +|---|---|---| +| `emulator.ps1` | Device pool: emulators **and** real devices | `ensure`, `status`, `list`, `pool`, `list-images`, `create`, `start` | +| `devicelease.ps1` | Lease a device so concurrent tests don't collide | `acquire`, `heartbeat`, `release`, `list`, `reap` | +| `appcontrol.ps1` | App build/install/state | `build`, `install`, `launch`, `clear`, `uninstall`, `is-installed`, `grant`, `list-apks` | +| `deviceui.ps1` | AI-driven UI I/O | `dump`, `wait-text`, `tap-text`, `tap-desc`, `input-text` (`-Clear`, `-CharByChar`, `-SecretRef`), `unlock`, `key`, `finger`, `finger-status`, `finger-enroll`, `screenshot`, `current-app` | +| `authlogs.ps1` | Log capture + verdict | `clear`, `scan`, `snapshot`, `grep`, `watch` | +| `labapi.ps1` | Provision/reset LAB test accounts (EasyAuth via WAM SSO); fetch tenant passwords from Key Vault | `create-user`, `reset`, `enable-policy`, `disable-policy`, `delete-device`, `open`, `fetch-password` | +| `report.ps1` | Render the HTML + Markdown test report (mandatory for ADO test cases); `summary` = overall multi-case run report | `render` (a per-case run JSON), `summary` (a batch/run folder) | +| `secrets.ps1` | Encrypted (DPAPI) store so passwords/PINs never hit the chat | `set`, `set-device-pin`, `list`, `test`, `get-masked`, `remove`, `path` | + +## Execution model — run inside a sub-agent, and supervise it + +An E2E run is long and chatty (boot, install, dozens of UI dumps/taps, log scans). **Delegate the actual +device-driving to a sub-agent** so the parent's context stays clean and the parent supervises rather than +performs. + +1. **Launch one sub-agent to own the run, on the SAME model as the parent.** If the parent is running on + Claude Opus 4.8, request the sub-agent on Claude Opus 4.8 (match whatever model the parent is on). + Give it a **self-contained** prompt: the scenario + explicit success criterion (Phase 1 output), the + leased `$serial` (or tell it to lease one with **its own agent id** as owner), the app/module + + package, credential-handling rules (type on device, never print), the run-folder path, any + mocks/flights to apply, and an instruction to **finish with a `PASS` / `FAIL` / `BLOCKED` verdict plus + evidence** (success signal + correlation_id, or the exact blocker). +2. **The parent WAITS for the sub-agent to finish.** Start it in the background, then block on its + completion (read its result with wait=true). **Do not end your turn, and do not report to the user, + until the sub-agent's verdict is in.** This is the #1 real failure mode we hit: the parent returned + early, the sub-agent's result and analysis were never surfaced to the user. +3. **Supervise with a watchdog while waiting:** + - **Hang → restart.** The sub-agent should emit a progress line and refresh its device lease + (`devicelease.ps1 heartbeat`) at each phase. If there's no progress within a timeout (e.g. ~10–15 min + for an install-heavy flow) **and the test isn't done**, treat it as hung: nudge once; if still stuck, + **restart** it (fresh sub-agent, resume from the last known state) rather than waiting forever. + - **Result in → stop.** If the verdict/evidence is already available, **stop waiting / terminate** the + sub-agent and move to reporting — don't let a finished run linger. + - **Cap restarts** (e.g. 2). If still unresolved, escalate to the user with the full evidence trail. +4. **Only the parent reports to the user.** Collect the sub-agent's verdict + artifact links and present + the Phase 7 report yourself. + +If the harness can't spawn a sub-agent (or you're already the deepest agent), run the phases inline — but +keep the same discipline: **don't declare done until the verdict is in, and don't loop forever on a hang.** + +## Running multiple test cases (batch) — split across devices, run in parallel + +When asked to run **more than one** test case (a suite, a list of test points, several ids), treat it as a +**batch**, not a serial slog: + +1. **Enumerate the work and the devices.** Resolve each case's scenario (Phase 1 per case) and **expand each + case into its test points** (Phase 1 → "Test points and configurations") so you know how many points each + case has and which build each uses (`Local\` for a `LocalFlights` config, `ECS\` otherwise). The **unit of + work is the whole case** — all of a case's test points run together and land in **one consolidated report** + (a case is one lane; its points run back-to-back within it). List every free device in the pool + (`emulator.ps1 pool` / `devicelease.ps1 list`) and count how many you can drive in parallel (respect the + lease pool cap, `-MaxPoolSize`). +2. **Fan out one sub-agent per test case, capped at the device count.** Give each case its own sub-agent + (same model as the parent) with a **self-contained** prompt: that case's scenario + success criterion, + the app/module + which APKs to install, credential rules (type on device, never print), its run-folder + subpath, and "finish with a PASS/FAIL/BLOCKED verdict + evidence". The sub-agent **runs every test point + for its case** (back-to-back on its leased device) and writes a **single consolidated report** for the + case. **Each sub-agent leases its OWN device** with its own agent id as owner + (`devicelease.ps1 acquire -Owner `) so no two lanes touch the same device. If there are **more + cases than devices**, each device is a **lane** that runs its queue of cases one after another; the parent + keeps every lane busy until the queue drains. +3. **One shared batch folder, one folder per case (all its test points inside).** Use a single + `…\android-e2e-runs\-\`. Give **each case** one folder `tc\` holding a single + case-level `run.json` and a single `TestReport.html/.md`. When a case has more than one test point, record + the points as a **`testPoints[]` array** in that one `run.json` (each entry carries its own + `ado.testPointId` + `configuration` + `buildSource`, steps, evidence, verdict) and put each point's + screenshots in a point-scoped subfolder `tc\\iter\` so paths stay relative. `report.ps1 + render` then produces **one report per case** with a section per test point + a single shared "Proposed + test steps". The suite summary still scans this folder and expands each case's points into their own rows + in the **Config** column. +4. **The parent supervises all lanes** with the same watchdog discipline as a single run (progress + + `heartbeat` per phase; restart a hung lane; stop a finished one; cap restarts). **Don't end the turn until + every case has a verdict.** +5. **Aggregate into an overall report.** After all cases finish, render the **suite summary** over the batch + folder (Phase 7): `./scripts/report.ps1 summary -In -Title ""` → `SUMMARY.html` + + `SUMMARY.md` (per-case verdict table linked to each report, plus overall counts). Present the overall + verdict and the per-case breakdown — not just the last case. + +**Only one device available?** Run the cases **serially** in a single lane (still clean-state per case, still +a per-case report each, still a final suite summary). **Independence is mandatory:** every case starts from a +clean state (Phase 3) and must not rely on another case's leftover accounts/registrations. + +## Workflow + +Execute in order. Announce a one-line status at each phase. + +### Phase 1 — Determine what to test + +Synthesize a concrete E2E scenario from the session context, in priority order: + +1. **User-provided test steps** in the session (explicit steps or a scenario the user described). +2. **An ADO Test Case work item** — the primary home for feature-specific steps. Fetch and parse it (see + "Sourcing test-case-specific steps" below); the `test-planner` skill can author/export these. +3. **A known test-steps file** the session points to (an exported plan, a markdown checklist, or a path + the user names). +4. **The design spec** (`design-docs/`) — its acceptance criteria and flows. +5. **The implementation diff** — `git -C diff` across changed repos to see what actually + changed and which app/module it affects. + +Produce: the **feature summary**, the **target app/module** (see +[references/app-and-module-map.md](references/app-and-module-map.md)), and a **step-by-step scenario** +with an explicit, observable **success criterion** (e.g. "AcquireTokenSilent returns a token; logs show +`executed successfully` with a correlation_id; no crash"). + +**Ask the user if** the scenario is ambiguous, multiple flows could be meant, or you cannot tell what +"working" looks like. Do not guess a scenario when the intent is unclear. + +**Keep specifics out of the skill.** This skill is generic. Anything specific to one feature — which app +configuration / client id to select, which lab account, which broker to pair, the exact tap sequence, the +expected success markers — belongs in the **test case**, not here. Read those specifics from the sources +above at run time; never hardcode a particular feature's steps into the skill or its scripts. + +#### Sourcing test-case-specific steps + +- **From ADO** (preferred for anything repeatable): a **Test Case** work item stores its steps in the + `Microsoft.VSTS.TCM.Steps` field (HTML/XML — each `` has an action and an expected result). Fetch it: + ```powershell + az boards work-item show --id --org ` + --fields "System.Title,Microsoft.VSTS.TCM.Steps" --output json + ``` + Parse the steps into an ordered action/expected list, drive them in Phase 4, and use each step's + **expected result** as its success criterion. A linked **Shared Steps** work item is fetched the same + way. (The `test-planner` skill authors and pushes these test cases, so the loop is: + test-planner → ADO → this skill.) +- **From a known file**: if the session provides a test-steps file (an exported plan, a markdown + checklist, or a path the user names), read the steps from there and treat them like ADO steps. +- If neither exists and the scenario isn't otherwise clear, **ask the user** rather than inventing + feature-specific steps. + +#### Test points and configurations (which build to run) + +A single ADO **Test Case** can appear in a suite as **one or two *test points*, each carrying a +*configuration*** (visible as the "Configuration" column on the test point). In the Android **Broker** suite +the two configurations are `RC MSAL - RC Broker` and `RC MSAL - RC Broker (LocalFlights)`, and the +configuration decides **which staged build folder you install from**: + +> ⚠️ **Counter-intuitive mapping — get this right:** +> - configuration name **contains `LocalFlights`** → install from the **`Local\`** folder +> - **any other** configuration (plain, no `LocalFlights`) → install from the **`ECS\`** folder +> +> `LocalFlights` does **not** mean the ECS folder. See +> [references/app-and-module-map.md → ECS vs Local builds](references/app-and-module-map.md#ecs-vs-local-builds-test-point-configuration). + +**Run every test point of the case, once each** — a 2-point case runs **twice** (once from `Local\` for the +LocalFlights point, once from `ECS\` for the plain point); a 1-point case runs once. The **only** exceptions +are when the **case body itself** says it is ECS-only or Local-only — then run just that one build. + +**Enumerate a case's test points + configurations from the Test Plan** (this is the source of truth — a case +in isolation doesn't tell you its points). Verified recipe: +```powershell +$org='https://dev.azure.com/identitydivision'; $project='Engineering' +$plan=; $suite= +$tok = az account get-access-token --resource '499b84ac-1321-427f-aa17-267ca6975798' --query accessToken -o tsv +$h = @{ Authorization = "Bearer $tok" } +$u = "$org/$project/_apis/testplan/Plans/$plan/Suites/$suite/TestPoint?api-version=7.1-preview.2" +$pts = (Invoke-RestMethod -Headers $h -Uri $u -Method GET).value +$pts | ForEach-Object { + [pscustomobject]@{ + testPointId = $_.id + caseId = $_.testCaseReference.id + config = $_.configuration.name + build = if ($_.configuration.name -match 'LocalFlights') { 'Local' } else { 'ECS' } + } +} | Where-Object caseId -eq | Format-Table +``` +`499b84ac-1321-427f-aa17-267ca6975798` is the Azure DevOps AAD resource GUID (`az` is already signed in for +identitydivision). Each returned row is **one run** for this case: feed its `build` into Phase 3 (which folder +to install from) and record `configuration` + `buildSource` in that run's `run.json` (Phase 7) so the report +and suite summary show a **Config** column. When you're driving a **batch**, each `(case, test point)` pair is +its own lane/run (see "Running multiple test cases"). + +### Phase 2 — Provision the device (emulator or real device) + +The **device pool** is every usable target: emulators **and** any real device connected over adb. +`emulator.ps1 pool` lists them. Derive requirements from the feature (min API, Google Play services for +broker/push flows) — see "Emulator / real-device requirements per feature" in the app-and-module map — +then one command finds-or-creates, reuses-or-starts, and waits for boot: + +```powershell +./scripts/emulator.ps1 ensure -RequireGoogleApis -ApiLevel 30 -Wait +# prefer a connected real device if one meets the requirements: +./scripts/emulator.ps1 ensure -RequireGoogleApis -PreferPhysical -Wait +# pin an exact device (emulator serial or real-device serial): +./scripts/emulator.ps1 ensure -Serial 39121FDJH003ZK +# or target a specific AVD: ./scripts/emulator.ps1 ensure -Avd Pixel_7 -Wait +# stick to emulators only: add -NoPhysical +``` +A connected, booted real device that satisfies the requirements is a valid target (real devices are +never created/booted — only used if already connected). By default `ensure` reuses a running emulator +first, then a matching real device, then boots/creates an AVD; `-PreferPhysical` flips it to try real +devices first. Capture the printed `SERIAL=` and use it as `-Serial` everywhere. **When multiple tests +may run concurrently, lease the device first** (see Phase 2a) so two runs don't collide. If provisioning +fails, see [references/troubleshooting.md](references/troubleshooting.md) (missing images, hypervisor, +boot hang). + +> **Speed check (do this if the run feels slow).** Run `./scripts/emulator.ps1 resolve-sdk` — it prints the +> host GPU/perf profile. On a **Cloud PC / VM / RDP host there is no GPU**, so the emulator uses slow +> **software rendering** (SwiftShader) and its NAT makes Play Store downloads crawl — nothing makes an +> emulator fast there. **Prefer a connected physical device** (the skill auto-prefers one on a GPU-less +> host; force with `-PreferPhysical`). AVDs the skill creates already appear in **Android Studio's Device +> Manager** (shared `~/.android/avd`); to see one in Studio's **Running Devices**, start it from Studio and +> let the skill reuse it. Details: [references/emulator-performance.md](references/emulator-performance.md). + +### Phase 2a — Lease the device (avoid collisions with other tests) + +Multiple E2E tests can run at once (different agents/sessions). To stop two runs from driving the same +device, **lease it** before use and **release it** when done: + +```powershell +# Acquire (reaps abandoned leases, picks a free device, or boots one if the pool cap allows). +# Use YOUR agent/session id as the owner so a dead agent's lease can be reclaimed. +$serial = (./scripts/devicelease.ps1 acquire -Owner $AgentId -Feature ` + -RequireGoogleApis -ApiLevel 30 -Wait | + Select-String '^SERIAL=(.+)$').Matches[0].Groups[1].Value +``` +- **Owner = your agent/session id** (or set `$env:E2E_AGENT_ID`). Liveness is heartbeat-based: refresh + during the run so a long test isn't reclaimed out from under you: + `./scripts/devicelease.ps1 heartbeat -Owner $AgentId -Serial $serial` (call it at each phase). +- **Pool cap:** `-MaxPoolSize N` (default 4) limits concurrent devices. If the pool is full and every + device is held by a **live** owner, acquire fails with guidance (wait, raise the cap, or connect a + device). If a holder's agent/session is **dead** (heartbeat older than `-StaleMinutes`, default 30), + its device is reclaimed automatically on the next `acquire`/`reap`. +- `acquire` will **boot/create an emulator** to add a device when the pool has no free one and the cap + allows — so it subsumes Phase 2's `emulator.ps1 ensure` (pass the same requirement flags). Use + `emulator.ps1 ensure` directly only when you deliberately don't want leasing. +- Do the whole run against the leased `$serial` (pass it as `-Serial` everywhere). + +### Phase 3 — Deploy the app-under-test + +1. Decide the test surface and whether a **broker** must also be installed (brokered flows need a + calling app + a broker) — see the app-and-module map's pairing rules. +2. Prefer an APK Android Studio already built (`appcontrol.ps1 list-apks`); otherwise build + (`appcontrol.ps1 build -Module <:module>`). Gradle builds need the repo's Maven creds — if they fail + with 401, that's an environment blocker for the user. **When the case is staged with both an `ECS\` and a + `Local\` folder, the folder to install from is dictated by the test point's configuration** — `Local\` + for a `LocalFlights` config, `ECS\` otherwise (Phase 1 → "Test points and configurations"; + [map](references/app-and-module-map.md#ecs-vs-local-builds-test-point-configuration)). Run the case once + per test point, each from its own folder. +3. **Start every test case from a clean state (unless told otherwise): uninstall, then freshly install all + test apps.** A `pm clear` alone is **not** a clean slate — it leaves AccountManager work-account entries + and broker/WPJ registration behind; only **uninstalling** the app removes those. So for each app the case + needs (app-under-test + any broker/companion), uninstall any existing copy first, then install the + provided/built APK: + ```powershell + ./scripts/appcontrol.ps1 uninstall -Package -Serial # ok if "not installed" + ./scripts/appcontrol.ps1 install -Module :msalTestApp -Serial # or -Apk for a provided APK + ./scripts/appcontrol.ps1 launch -Package -Serial + ``` + Verify the exact package/activity at runtime (the map shows how) — don't trust hardcoded IDs. Known + provided-APK filenames (e.g. `app-production-universal-release-signed` = the **Authenticator** under test; + `com.microsoft.windowsintune.companyportal-signed` = **Company Portal**) are in + [references/app-and-module-map.md](references/app-and-module-map.md#known-provided-apks). **Do not tear + down accounts/registrations at the *end* of a case** — leaving them is intended (the *next* run's + uninstall clears them, or a tester handles them); see Phase 7. + +### Phase 4 — Execute the scenario (auto-handle inputs) + +Clear the log buffer first, then drive the flow: + +```powershell +./scripts/authlogs.ps1 clear -Serial +``` + +Loop per screen using [references/ui-interaction.md](references/ui-interaction.md), following the +**fast path** (see [references/run-speed.md](references/run-speed.md)): `dump` **once** per screen and +compute every target from that one XML → act → verify by the **next** screen's anchor with +`tap-text -Then ""` (tap + wait in one call) or `wait-text`, **never a fixed `Start-Sleep`**. +Batch the dump→tap(s) for a screen into a **single** shell call so process/adb startup is paid once per +screen, not once per tap. Re-`dump` only when you must read genuinely new state; verify a navigation by its +anchor, not a reflexive re-dump after every tap. Save a `screenshot` only at **milestones** and only on +screens that actually render (skip FLAG_SECURE screens — they come back black; capture a `uiautomator dump` +as evidence instead). + +**Auto-handle** everything the AI reasonably can: type lab test credentials, tap Next/Sign in/Accept/ +Allow/Yes, pick an account, grant permissions, simulate a fingerprint (`finger`), enter a TOTP if the +seed is known, set/enter a device PIN. **Do not** print or commit credentials — when the user gives you +a password or device PIN, take it via the encrypted store and reference it by name (see +[references/secrets-and-files.md](references/secrets-and-files.md)), never inline. + +**Typing into eSTS/WebView credential fields (hard-won).** The email/password pages are a WebView and +Chrome's autofill/passkey overlay can silently swallow a bulk `input text` (the value lands in the wrong +field or is dropped, producing "Enter a valid email"). **Try bulk first** (it's fast); only fall back to +`-CharByChar` if verification shows the value didn't land: +```powershell +./scripts/deviceui.ps1 input-text -Text $upn -Clear -Serial # bulk first (fast) +./scripts/deviceui.ps1 input-text -Text $upn -Clear -CharByChar -Serial # fallback if it didn't land +./scripts/deviceui.ps1 input-text -SecretRef labpw -Clear -CharByChar -Serial # password from the store (never echoed) +``` +`-Clear` empties the field first, `-CharByChar` defeats the overlay. For the **password**, prefer +`-SecretRef ` (resolves from the encrypted store and implies `-Secret`) so the value never lands +in a tool call or the transcript — see [references/secrets-and-files.md](references/secrets-and-files.md); +`-Text $pw -Secret` still works if you already hold the value out-of-band. For **shared lab-tenant +passwords**, don't ask the user to paste — pull it from Key Vault into the store first with +`./scripts/labapi.ps1 fetch-password -TestTenant ID4SLAB2 -IntoSecret labpw` (needs `az login` + +`TM-MSIDLABS-DevKV`), then reference `-SecretRef labpw`. Dismiss a passkey/"Save +password" bottom sheet with `key ESCAPE` before typing if one appears. +See [references/common-blockers.md](references/common-blockers.md). + +**Unlocking the lock screen on a physical device.** If the device sleeps/relocks mid-run and a step needs a +PIN, seed it once with `secrets.ps1 set-device-pin` (done by the human — it picks the device, or shows a +numbered menu when several are attached, and stores the PIN as `devicepin_`), then let the tool enter ++ **verify** it: `deviceui.ps1 unlock -Serial ` **auto-resolves** that per-device PIN (no `-SecretRef` +needed). It confirms the keyguard actually cleared and +**stops after 3 attempts** (`-MaxAttempts`, default 3, exits 3 if it gives up) so a wrong PIN never trips +Android's Gatekeeper lockout — do **not** loop it yourself. With several devices attached (even the same +model) each has a unique adb serial; because the PIN is stored per serial, `unlock -Serial ` always +uses the right device's PIN. A PIN only substitutes for biometric where a "Use PIN" path is offered; a +fingerprint-only step needs an emulator (`finger`) or a human. See +[references/secrets-and-files.md](references/secrets-and-files.md) and +[references/common-blockers.md](references/common-blockers.md). + +**Provision / repair the lab account with the LAB API** when the scenario needs a fresh account or the +account state is stuck (e.g. MFA already registered from a previous run, a CA policy blocking the step): +```powershell +./scripts/labapi.ps1 create-user -UserType GlobalMFA # temp user, auto-deletes in 60 min +./scripts/labapi.ps1 reset -Upn $upn -Operation mfa # clear stale MFA registration (temp users only) +./scripts/labapi.ps1 disable-policy -Upn $upn -Policy GlobalMFA # unblock a CA-gated segment +./scripts/labapi.ps1 fetch-password -TestTenant ID4SLAB2 -IntoSecret labpw # pull tenant pw from Key Vault (no paste) +``` +**Don't reset a password just because sign-in rejected it.** If the device shows *"Your account or password is +incorrect"* / *"password has expired"* / *"that Microsoft account doesn't exist"*, do **not** run +`reset -Operation password` (or any password change) unless the **test case explicitly tells you to**. A wrong +password almost always means you have the wrong *value* or the wrong *account*, not that the account needs +changing — so instead: re-pull the shared value with `fetch-password` (it may have rotated in Key Vault), confirm +the UPN/tenant matches what the case named, and check the account isn't `Locked_…` from a prior lockout. Resetting +is especially dangerous for **shared durable accounts** (e.g. `AndroidTBUser1@id4slab2`) that other tests reuse — +changing their password breaks every other case. If the value is genuinely right and it still fails, mark the run +**BLOCKED** with the exact on-screen error rather than mutating the account. + +**Account policy — prefer fresh ID4SLAB2 temp users, one per case.** Provision a **new temp user for each +test case** (`create-user` makes an ID4SLab2 user that auto-deletes in ~60 min) instead of reusing one across +cases — **even when a test case names a fixed MSIDLAB4 account** (that lab is being deprecated): create the +matching `-UserType` instead (e.g. `MAMCA`, `MDMCA`, `Basic`, `GlobalMFA`). Reuse a specific named/durable +account only when the case genuinely requires that exact identity, or the user tells you to. + +**Freshness gate — poll ≤ 3 min, then recreate or reuse a < 30-min-old user.** After `create-user`, a brand-new +temp user can lag ESTS replication and show *"This username may be incorrect"* at sign-in. Don't keep waiting: +if the new user isn't **consistently** sign-in-able within **3 minutes** of polite polling, **create another** +temp user, **or reuse a previously created temp user still under ~30 minutes old** (inside its 60-min TTL and +already propagated — one that already signed in once is safest). Note the swap in the report; never "fix" the lag +with a password reset. See +[common-blockers.md → Fresh temp user not sign-in-able yet](references/common-blockers.md#fresh-temp-user-not-sign-in-able-yet-ests-propagation-lag). + +See [references/lab-api.md](references/lab-api.md) for all endpoints, usertypes, the account policy, and the auth workaround. + +**Set flags & mock what's missing (don't fake a pass).** If the scenario needs a feature flag on, or a +step depends on data/a dependency you can't produce naturally (a server API not deployed yet, a +collaborator app you can't drive), set the flag and mock the missing piece — including **temporary code +changes** that you revert afterward. If a middle piece genuinely can't be mocked, test the flow in +**segments**. See [references/mocking-flights-and-segments.md](references/mocking-flights-and-segments.md). + +**Do system-settings / on-device actions yourself before calling something blocked.** Some steps have no +adb command (advance the device clock, delete a user certificate, toggle a system setting, change the +language). Don't jump straight to BLOCKED — **open the relevant Settings screen and drive it with +`deviceui.ps1` (`dump` → `tap-text`/`tap-desc` → `input-text`) exactly like a human tester would.** E.g. to +advance the clock: Settings → *Date & time* (Samsung: *General management → Date and time*) → turn **off** +automatic date/time → set it manually. Only fall back to a blocker when the surface itself is genuinely +undrivable — a **secure/native system dialog** (Knox cert install, keyguard credential prompt, biometric +sensor) uiautomator can't read/act on, or an action that truly needs **root** (e.g. expiring an app's +*internal* monotonic cached-token timer on a non-rooted retail device — moving the wall clock won't affect +it). See [references/common-blockers.md](references/common-blockers.md#doing-it-yourself-in-system-settings). + +**Stop and ask the user** only for genuine blockers (see the consolidated list below). + +### Phase 5 — Verify from logs + +```powershell +./scripts/authlogs.ps1 scan -Package -Serial -Out \iter1\logcat.txt +``` +`scan` prints a heuristic verdict plus evidence (crash stacks, failure lines, success lines, AADSTS +codes, correlation IDs). **Reason over the evidence** — don't blindly trust PASS/FAIL. Confirm the +scenario's specific success criterion is met (a positive success signal, not just absence of errors), +per [references/log-signals.md](references/log-signals.md). For automation-test mode, also read the +`build/reports/androidTests/` results. + +Classify the outcome: +- **PASS** — success criterion met, no crash → go to Phase 7. +- **FAIL (real defect)** — crash in changed code, `AADSTS50011/700016`, regressed silent auth, broken + IPC → go to Phase 6. +- **Environment/harness problem** (missing image/creds, signature mismatch, stale snapshot, offline + adb) → fix the setup per troubleshooting, then re-run from Phase 2/3. Do **not** send these to the fix loop. +- **INCONCLUSIVE** — wrong package filter, action never executed, logging off → re-run the scenario + cleanly before deciding. + +### Phase 6 — Iterate: fix and retest + +For a real defect: + +1. **Root-cause** from the evidence: the failing signal, correlation_id, stack, and the code in the + diff it points to. State the hypothesis with the log line that supports it. +2. **Get it fixed:** + - If the code is in a checked-out sub-repo the agent can edit, **fix it directly** (Kotlin-first per + repo conventions), then rebuild. + - If the work is managed via a coding-agent PR, hand the diagnosis to that agent (e.g. `@copilot` + on the PR, or the `pbi-dispatcher` skill) with the exact evidence and root cause. +3. **Reinstall → clear → re-run the scenario → re-scan** (Phases 3–5). +4. Repeat, **max 3–5 iterations**. If still failing, **stop and escalate** to the user with the full + evidence trail (what was tried, current hypothesis, why it's stuck). Do not loop forever. + +Track iterations in the run folder (one subfolder per attempt) so regressions/progress are diffable. + +### Phase 7 — Report + +Summarize: scenario tested, target app/emulator, final verdict, iterations taken, key evidence +(success signal + correlation_id, or the blocker), and links to the run-folder artifacts (logs, +screenshots). If blocked, state exactly what you need from the user to proceed. + +**For an ADO Test Case, a written test report is MANDATORY — always generate it, on every outcome +(PASS / FAIL / BLOCKED / PARTIAL), not only on success.** Render both an HTML and a Markdown report into +the run folder with `report.ps1`: +```powershell +# Build a run JSON (verdict, device/app/account metadata, per-step action/expected/result, evidence, +# blockers, artifact paths — UPN only, never a password), then render: +./scripts/report.ps1 render -In \run.json # writes TestReport.html + TestReport.md next to it +``` +The step list should mirror the ADO test case's steps, and each step's **result** should be judged +against that step's **expected result**. Include the ADO ids (`testCaseId`/`planId`/`suiteId`) and a link +so the report ties back to the test plan. **When the run came from a specific test point, also record +`ado.testPointId`, `ado.configuration` (the config name), and `ado.buildSource` (`ECS` or `Local`)** so the +report header and the suite summary's **Config** column show which build this run exercised. + +**If a case has more than one test point, put them all in ONE report** — don't write a separate report per +point. Give the case one `run.json` with a **`testPoints[]` array**; each entry carries its own point-level +fields (`ado.testPointId`/`configuration`/`buildSource`, `device`, `app`, `account`, `steps`, `evidence`, +`blockers`, `verdict`) and references its screenshots by a point-scoped relative path (e.g. +`ecs/iter1/07_token.png`, `local/iter1/07_token.png`). `render` emits one section per test point and derives +the overall verdict from the points. Any **"Proposed test steps"** you add live at **case level** and must be +**generic across every test point** (don't mention ECS/Local or a specific point); give each proposed step an +optional `attachment` (a screenshot path/URL) to fill the report's **Attachments** column. See +[references/test-reporting.md](references/test-reporting.md) +for the JSON schema and a worked example. (For an automation-test run, also attach the +`build/reports/androidTests/` output.) Present the verdict and the report paths to the user; optionally +publish the outcome back to the ADO Test Run. + +**When you ran a batch (multiple ADO cases), also generate the overall run summary** — in addition to each +per-case report — so the user gets one roll-up verdict: +```powershell +./scripts/report.ps1 summary -In -Title "" # SUMMARY.html + SUMMARY.md: per-case table + counts +``` + +**Do not tear down accounts or registrations after a case (unless told otherwise).** Leave temp users, +device/WPJ registrations, and installed apps as they are — the *next* run's clean-state step (Phase 3 +uninstall+reinstall) removes them, or a tester handles them manually. This keeps a failed/blocked case's +state available for inspection and avoids racing another lane (temp lab users self-destruct in ~60 min +anyway). The only required end-of-case teardown is **releasing the device lease** (below) and **reverting any +temporary code/flag/mocks** (Guardrails). + +**Release the device lease** so the next test can use it (do this even on failure/blocked): +```powershell +./scripts/devicelease.ps1 release -Owner $AgentId -Serial $serial +``` + +## When to ask the user + +Ask (don't guess) when: +- The **scenario/intent is unclear**, or several flows could be meant, or success can't be defined. +- A **blocker** needs a human — but only *after* you've tried the on-device/Settings path yourself (Phase 4): + real push-notification MFA on another device, SMS/phone OTP, a hardware key/NFC/QR/camera, CAPTCHA, a + **secure native system dialog** (Knox cert install, keyguard/biometric prompt), a credential the AI doesn't + have, or a tenant/CA policy it can't provision. +- The **implementation is incomplete** for the path under test (stubs, `TODO`, missing wiring, feature + flag off with no way to enable) — **first try** to set the flag (temp code change) and mock/segment + around a missing middle piece (see + [mocking-flights-and-segments.md](references/mocking-flights-and-segments.md)); report the gap and ask + only if the feature genuinely can't be exercised even in segments. +- **Environment setup** is missing and only the user can fix it (Maven creds/PAT, no sub-repo checkout, + no system image, no lab account). + +When you can complete a manual step *with* the user live (e.g. they approve a push), pause, let them do +it, then resume the automated flow. + +## References + +Load these as needed (don't preload all): + +| File | Read it when | +|---|---| +| [references/app-and-module-map.md](references/app-and-module-map.md) | Choosing/deploying the test app, package discovery, broker pairing, credentials, emulator requirements | +| [references/lab-api.md](references/lab-api.md) | Provisioning/resetting a lab test account, the account policy (prefer fresh ID4SLAB2 temp users over MSIDLAB4), LAB API endpoints/usertypes/policies, the EasyAuth auth workaround, fetching tenant passwords from Key Vault, `labapi.ps1` | +| [references/common-blockers.md](references/common-blockers.md) | Recurring hiccups & when to switch to an emulator (fingerprint/App-Lock, number-match MFA, session timeouts, FLAG_SECURE, autofill/passkey overlay, screenshot corruption); driving System Settings yourself before declaring a blocker; clean-state between runs | +| [references/test-reporting.md](references/test-reporting.md) | Writing the mandatory ADO test report — run-JSON schema, `report.ps1`, worked example | +| [references/run-speed.md](references/run-speed.md) | The run feels slow; understanding per-step latency sources and how to shorten them | +| [references/emulator-performance.md](references/emulator-performance.md) | The run is slow, you're on a Cloud PC/VM/RDP (software rendering), or you want the emulator to show in Android Studio's Device Manager / Running Devices | +| [references/log-signals.md](references/log-signals.md) | Interpreting logcat, success/failure patterns, AADSTS codes, per-flow pass criteria, eSTS correlation | +| [references/ui-interaction.md](references/ui-interaction.md) | Driving auth screens, AI-vs-human inputs, FLAG_SECURE gotcha, selector strategy | +| [references/mocking-flights-and-segments.md](references/mocking-flights-and-segments.md) | A flag must be set, a dependency/server data is unavailable, or the flow can't run fully E2E (mock it or test in segments) | +| [references/secrets-and-files.md](references/secrets-and-files.md) | The user must hand you a password / device PIN / keystore password, or an APK / test file — keep secrets out of the chat (`secrets.ps1`, `labapi.ps1 fetch-password`, `-SecretRef`, `unlock`) and use a file drop-folder | +| [references/troubleshooting.md](references/troubleshooting.md) | Emulator/build/install/uiautomator/broker failures; env-vs-defect triage | + +## Guardrails + +- **Never** print, log, or commit real or lab credentials, tokens, or secrets. Type them onto the device + only — use `input-text -SecretRef ` (or `-Text ... -Secret`) so the value never hits the + transcript. When the user needs to hand you a password or device PIN, take it via the encrypted store + (`secrets.ps1 set`) and reference it by name; for **shared lab-tenant passwords** pull them straight + from Key Vault with `labapi.ps1 fetch-password` (no paste needed); for APKs/files use a drop-folder path. + See [references/secrets-and-files.md](references/secrets-and-files.md). +- **Artifacts stay outside the repo** (the run folder). Never commit logs/screenshots/reports. +- **For an ADO test case, always generate the HTML + Markdown report** (`report.ps1`) on every outcome — + PASS, FAIL, BLOCKED, or PARTIAL. A run driven from a test case is not "done" until the report exists. +- **Run every test point of a case, and match the build to its configuration.** A Broker-suite case may have + two test points; run each once. Install from **`Local\`** when the point's configuration name contains + `LocalFlights`, from **`ECS\`** otherwise (⚠️ not the other way round). Skip a point only if the case body + says it's ECS-only or Local-only. Record `configuration`/`buildSource` in each run's report. +- **A batch of ADO cases also needs the overall `report.ps1 summary`** over the batch folder, not just the + per-case reports. +- **Split a multi-case batch across available devices and run in parallel** (one sub-agent per case, each + leasing its own device); fall back to serial only when a single device is free. See "Running multiple test + cases". +- **Start each case clean; don't clean up after.** Unless told otherwise, uninstall+reinstall the test apps + at the **start** of every case (a `pm clear` doesn't remove work accounts/registrations), and **do not** + tear down accounts/registrations at the **end** — leave them for the next run or a tester. +- **Drive System Settings / on-device UI yourself before declaring a blocker** (advance the clock, delete a + cert, toggle a setting) — only block on a genuinely secure/native dialog or a true root requirement. +- **Prefer fresh ID4SLAB2 temp accounts, one per case** — use `create-user -UserType ` even when a + case names a deprecated MSIDLAB4 account, unless a specific durable account is required. +- **Don't reset/change a password on an auth failure unless the test case says so.** "Incorrect password" or + "password expired" on the sign-in screen means re-fetch the value (`fetch-password` — it may have rotated) and + re-check the UPN/tenant, **not** `reset -Operation password`. Never change the password of a shared durable + account (e.g. `AndroidTBUser1@id4slab2`) — it breaks other tests; if the value is right and it still fails, + mark the run **BLOCKED** with the exact error. +- **Prefer an emulator when a step needs an injectable fingerprint/biometric** (App Lock, biometric-gated + number-match). `adb emu finger touch` works only on emulators; a physical device needs a human at the + sensor. See [references/common-blockers.md](references/common-blockers.md). +- **When delegating to a sub-agent, wait for its verdict** before reporting or ending the turn; restart it + if it hangs and the test isn't done, terminate it once the result is in. Never surface "done" without the + sub-agent's PASS/FAIL/BLOCKED evidence. +- **Lease the device before use and release it after** (even on failure) so concurrent runs don't collide. +- **Temp changes for mocks/flags stay uncommitted and get reverted** — never commit/push a flag flip or a + mock; leave a `TODO: REVERT` marker and confirm the tree is clean before finishing. +- **Environment problems are not code defects** — fix setup and re-run; only hand real defects to the fix loop. +- **Cap the loop** (3–5 iterations) and escalate with evidence rather than looping indefinitely. +- **Require a positive success signal** matching the scenario before declaring PASS. +- Follow repo conventions when fixing code (Kotlin for new code, the `Logger` class, minimal changes). diff --git a/.github/skills/android-e2e-tester/references/app-and-module-map.md b/.github/skills/android-e2e-tester/references/app-and-module-map.md new file mode 100644 index 00000000..08a619c8 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/app-and-module-map.md @@ -0,0 +1,170 @@ +# App & Module Map — What to Build, Install, and Test + +Table of contents: +- [Pick the test surface from the feature](#pick-the-test-surface-from-the-feature) +- [Module → path → typical package](#module--path--typical-package) +- [Known provided APKs (staged test builds)](#known-provided-apks) +- [Discover the exact package & launch activity at runtime](#discover-the-exact-package--launch-activity-at-runtime) +- [Two E2E execution modes](#two-e2e-execution-modes) +- [Broker pairing rules](#broker-pairing-rules) +- [Test credentials (Lab API) and when they block](#test-credentials-lab-api-and-when-they-block) +- [Emulator / real-device requirements per feature](#emulator--real-device-requirements-per-feature) + +The repo root (`android-complete`) aggregates sub-repos cloned via `git droidSetup`: +`msal/`, `common/`, `broker/`, `adal/`, and optionally `authenticator/`. Gradle modules are +declared in `settings.gradle`. If a sub-repo folder is missing, the checkout hasn't been set up — +that is a blocker to raise with the user. + +## Pick the test surface from the feature + +Map the changed repo(s) (from `git diff` / the design) to the app you drive: + +| Feature lives in | App to install & drive | Also install (dependency) | +|---|---|---| +| **msal** (SDK) | `:msalTestApp` (interactive) or `:msalautomationapp` (automated) | A broker for brokered flows (see pairing) | +| **broker / broker4j** | `:brokerHost` test host + a broker (`:AADAuthenticator`) | `:msalTestApp` as the calling client | +| **common / common4j** (IPC, cache, telemetry) | Both sides: `:msalTestApp` **and** a broker | — | +| **adal** (legacy) | `:adalTestApp` | A broker for brokered ADAL flows | +| **authenticator** (opt-in) | `:MSAuthenticator` | — | + +Isolated broker testing without real Authenticator/Company Portal can use the **mock brokers**: +`:mockauthapp` (mock Authenticator), `:mockcp` (mock Company Portal), `:mockltw` (mock Link-to-Windows). + +## Module → path → typical package + +Package IDs below are **typical hints** — always verify at runtime (next section), because they +vary by build variant (e.g. `localDebug` often adds a `.local` suffix). + +| Gradle module | Project path | Typical applicationId | +|---|---|---| +| `:msalTestApp` | `msal/testapps/testapp` | `com.msft.identity.client.sample.local` | +| `:msalautomationapp` | `msal/msalautomationapp` | `com.microsoft.identity.client.msalautomationapp` | +| `:brokerHost` | `broker/userapp` | `com.microsoft.identity.testuserapp` | +| `:brokerautomationapp` | `broker/brokerautomationapp` | `com.microsoft.identity.client.brokerautomationapp` | +| `:AADAuthenticator` | `broker/AADAuthenticator` | `com.azure.authenticator` | +| `:adalTestApp` | `adal/userappwithbroker` | `com.microsoft.aad.adal.testapp` | +| `:mockcp` | `broker/mockbrokers/mockcp` | mock Company Portal | +| `:mockauthapp` | `broker/mockbrokers/mockauthapp` | mock Authenticator | +| `:MSAuthenticator` | `authenticator/PhoneFactor/app` | `com.azure.authenticator` | + +Real production brokers used in some E2E: Microsoft Authenticator `com.azure.authenticator`, +Intune Company Portal `com.microsoft.windowsintune.companyportal`, Link to Windows `com.microsoft.appmanager`. + +## Known provided APKs + + +When the user hands you a folder of APKs (commonly `…\Downloads\ECS` and `…\Downloads\Local`, or +`…\Downloads\APKs\…`), map them by **filename** so you install the right thing without re-deriving it each +run. `-signed` = a production-signed release build; `-dist-debug` / `-local-debug` = the test/automation +harness builds. Always confirm the real `applicationId` after install (next section) — filenames are a +convenience, not a contract. + +| Provided APK filename (with/without `.apk`) | App | Package (verify at runtime) | Role in a test | +|---|---|---|---| +| `app-production-universal-release-signed` | **Microsoft Authenticator (app under test)** | `com.azure.authenticator` | The build being tested; **universal** ABI — safe on emulators too. | +| `app-production-arm64-v8a-release-signed` | Microsoft Authenticator (arm64 only) | `com.azure.authenticator` | Same app, arm64-only — use on a **physical** arm64 device; can fail dexopt on an x86_64 emulator (prefer the universal one there). | +| `com.microsoft.windowsintune.companyportal-signed` | **Intune Company Portal** | `com.microsoft.windowsintune.companyportal` | Broker / MDM companion for WPJ / device-registration / MAM-CA flows. | +| `company-portal` | Intune Company Portal | `com.microsoft.windowsintune.companyportal` | Same app, alternate filename. | +| `teams` | Microsoft Teams | `com.microsoft.teams` | 1P OneAuth client (some brokered/SSO scenarios). | +| `msalTestApp-dist-debug` | MSAL test app | `com.msft.identity.client.sample(.local)` | Calling client for MSAL / brokered flows (Mode B). | +| `adalTestApp-dist-debug` | ADAL test app | `com.microsoft.aad.adal.testapp` | Calling client for legacy ADAL flows. | +| `brokerHost-local-debug` | Broker Host test app | `com.microsoft.identity.testuserapp` | Drives broker APIs directly (Get Accounts / device state). | + +## ECS vs Local builds (test-point configuration) + +When a case is staged with both an `ECS\` and a `Local\` folder, **which folder you install the app-under-test +from is decided by the ADO test point's *configuration*, not by a fixed default:** + +| Test-point configuration name | Build folder to install from | +|---|---| +| contains **`LocalFlights`** (e.g. `RC MSAL - RC Broker (LocalFlights)`) | **`Local\`** | +| anything else (e.g. plain `RC MSAL - RC Broker`) | **`ECS\`** | + +> ⚠️ **Counter-intuitive — read twice.** The `LocalFlights` configuration maps to the **`Local\`** folder, and +> the *plain* configuration maps to the **`ECS\`** folder. Do **not** assume "LocalFlights → ECS". A case with +> two test points is therefore installed **from `Local\` for the LocalFlights point and from `ECS\` for the +> plain point** — i.e. you run that case **twice**, once per folder. + +Install the **app-under-test** from the folder the point's configuration selects; pull any *other* required app +(a broker the case needs but didn't stage, Teams, Chrome) from that **same** folder if present, else the Play +Store. How to enumerate a case's test points and read each one's configuration is in the SKILL +([Phase 1 → Test points and configurations](../SKILL.md#test-points-and-configurations-which-build-to-run)). + +## Discover the exact package & launch activity at runtime + +Do not trust the hints above blindly. Resolve the real values: + +```powershell +# applicationId from the module's build.gradle (accounts for variant suffixes): +Select-String -Path msal/testapps/testapp/build.gradle -Pattern 'applicationId' + +# After install, confirm the installed package and find the LAUNCHER activity: +./scripts/appcontrol.ps1 is-installed -Package com.msft.identity.client.sample.local +adb shell cmd package resolve-activity --brief -c android.intent.category.LAUNCHER + +# Or read a built APK directly: +& "$env:ANDROID_HOME\build-tools\\aapt.exe" dump badging | Select-String 'package:|launchable-activity' +``` + +Then launch with the resolved values: +```powershell +./scripts/appcontrol.ps1 launch -Package -Activity +``` + +## Two E2E execution modes + +**Mode A — Automated instrumented tests (preferred when a matching test exists).** +The automation apps drive full sign-in flows and fetch lab accounts automatically via the Lab API. +Run the connected test task for the variant: +```powershell +./gradlew :msalautomationapp:connectedLocalDebugAndroidTest --tests "**" +./gradlew :brokerautomationapp:connectedLocalDebugAndroidTest --tests "**" +``` +Results land in `build/reports/androidTests/connected/` and `build/outputs/androidTest-results/`. +Read those reports plus `./scripts/authlogs.ps1 scan` to judge pass/fail. + +**Mode B — Interactive UI-driven (for new features with no automated test yet).** +Install the test app, launch it, and drive the UI with `deviceui.ps1` (tap the acquire-token +button, sign in, handle prompts), then verify with `authlogs.ps1 scan`. Use this when the feature +is brand-new or the scenario is exploratory. See [ui-interaction.md](ui-interaction.md). + +Choose Mode A if an automation test already covers the flow; otherwise Mode B. + +## Broker pairing rules + +Brokered auth requires **a calling app + a broker app**, both installed: + +- The broker must be a build the client trusts (signature/redirect-URI must match). Local `localDebug` + test apps are built to trust the `localDebug` broker signatures — keep both on the same variant. +- Signature mismatch on install shows `INSTALL_FAILED_UPDATE_INCOMPATIBLE` → uninstall the old one first + (`./scripts/appcontrol.ps1 uninstall -Package `). +- Only one active broker is used at a time. If both Authenticator and Company Portal are present, the + broker-selection logic picks one; for deterministic tests install a single broker (or a mock). +- After installing/replacing a broker, **clear the client app state** (`appcontrol.ps1 clear`) so it + re-discovers the broker cleanly. + +## Test credentials (Lab API) and when they block + +- Automated tests (Mode A) obtain lab accounts through `common/labapi` / the Lab API automatically — + no manual credential handling. +- Interactive tests (Mode B) need a test account. Preferred sources, in order: + 1. A lab account the user provides in the session. + 2. A credential the automation harness/Lab API exposes. +- **Provision or repair accounts on demand with `scripts/labapi.ps1`** (`create-user`, `reset`, + `enable-policy`/`disable-policy`, `delete-device`). Full endpoint list, usertypes/policies, the EasyAuth + auth workaround, and entitlements are in **[lab-api.md](lab-api.md)**. +- **Blocker to raise with the user** if none is available, or if the scenario needs a real MSA/consumer + account, a specific tenant/CA policy, or a federated account that the AI cannot provision (try + `labapi.ps1 disable-policy` first if it's a lab CA policy). +- **Never** hardcode, print, or commit real credentials. Type them into the device only (`input-text + -Secret`); never echo them into logs or the transcript. + +## Emulator / real-device requirements per feature + +Pass these to `emulator.ps1 ensure` (they gate both AVD selection and real-device eligibility): + +- **Broker / push / GMS-dependent flows** → `-RequireGoogleApis` (needs Google Play services for FCM, + account manager). Prefer a `google_apis` or `google_apis_playstore` image. +- **Min OS behavior** (e.g. scoped storage, notification permission on API 33+) → `-ApiLevel `. +- **Play Store app installs** (rare in tests) → `-RequirePlayStore`. +- Most auth E2E works on a standard `google_apis` image, API 30+. diff --git a/.github/skills/android-e2e-tester/references/common-blockers.md b/.github/skills/android-e2e-tester/references/common-blockers.md new file mode 100644 index 00000000..65054c6a --- /dev/null +++ b/.github/skills/android-e2e-tester/references/common-blockers.md @@ -0,0 +1,327 @@ +# Common Scenarios, Hiccups & Blockers + +Recurring friction from real E2E runs, what causes it, and how to get past it — plus **when to switch +from a physical device to an emulator**. Read this before driving an auth flow so you don't lose time +rediscovering a known trap. See also [ui-interaction.md](ui-interaction.md) (screen-by-screen driving) +and [troubleshooting.md](troubleshooting.md) (env/build/install). + +Table of contents: +- [Decision: emulator vs physical device](#decision-emulator-vs-physical-device) +- [Steps that need a fingerprint / biometric / App Lock](#steps-that-need-a-fingerprint--biometric--app-lock) +- [Number-match MFA](#number-match-mfa) +- [Session timeouts & SSO resets mid-flow](#session-timeouts--sso-resets-mid-flow) +- [Chrome autofill / passkey overlay steals input](#chrome-autofill--passkey-overlay-steals-input) +- [FLAG_SECURE black screenshots](#flag_secure-black-screenshots) +- [Screenshot corruption via redirection](#screenshot-corruption-via-redirection) +- [Single-use pairing / setup links](#single-use-pairing--setup-links) +- [Stale account state between runs](#stale-account-state-between-runs) +- [Fresh temp user not sign-in-able yet (ESTS propagation lag)](#fresh-temp-user-not-sign-in-able-yet-ests-propagation-lag) +- [Password rejected at sign-in — don't reset it](#password-rejected-at-sign-in--dont-reset-it) +- [Doing it yourself in System Settings](#doing-it-yourself-in-system-settings) +- [Genuine blockers (stop and ask)](#genuine-blockers-stop-and-ask) +- [Quick reference table](#quick-reference-table) + +## Decision: emulator vs physical device + +The skill can run on either, but some steps are **only automatable on an emulator** because they need an +input you can inject programmatically. Choose up front from the test case's steps: + +**Prefer an emulator when the scenario includes any of:** +- A **fingerprint / biometric** prompt you must satisfy (enroll + inject a touch). +- **App Lock** in Microsoft Authenticator (it re-prompts for device credential/biometric). +- A **biometric-gated number-match** approval. +- Anything that needs `adb emu ...` (finger, sensors, GSM, battery) — emulator-only console commands. + +**A physical device is fine (or better) when:** +- No biometric is required (pure password + KMSI + token acquisition). +- You're on a **GPU-less host** (Cloud PC / VM / RDP) where the emulator is painfully slow — a physical + device is much faster there (see [emulator-performance.md](emulator-performance.md)). +- The feature needs real hardware (real FCM push, real SIM) — though real push MFA is still a human step. + +> Rule of thumb: **biometric/App-Lock → emulator; everything else → whatever is fastest** (usually a +> connected physical device on a Cloud PC). If you start on a physical device and hit an unavoidable +> fingerprint gate, that's the signal to re-run the biometric segment on an emulator. + +> **Caveat — the emulator isn't always available.** On some GPU-less hosts the emulator **won't stay up**: +> the Bluetooth HAL crash-loops and takes `system_server` down repeatedly, so a heavy app (e.g. +> Authenticator) can't be driven even after it boots. See the boot recipe and the instability signal in +> [troubleshooting.md](troubleshooting.md#emulator-wont-start-or-boot). If you've exhausted that recipe and +> it still won't stabilize, then a **biometric-gated step becomes a genuine blocker on this host** — do the +> non-biometric parts on the physical device, mark the biometric step BLOCKED with evidence, and ask the +> user (don't burn the whole run fighting the emulator). Also make sure you install the **right ABI**: use +> the **universal** APK on emulators — a large `arm64-v8a`-only APK can crash install-time dexopt on an +> `x86_64` image (see [Install failures](troubleshooting.md#install-failures)). + +## Targeting the right device (multiple devices / same model) + +Every attached device gets a **unique adb serial**, independent of model — two identical phones still show up +as two different serials. List them with their model/transport so you can tell them apart: + +```powershell +adb devices -l +# emulator-5554 device product:sdk_gphone64_x86_64 model:sdk_gphone64_x86_64 transport_id:21 +# 39181FDH2000ABC device product:husky model:Pixel_8_Pro transport_id:9 +# 39181FDH2000XYZ device product:husky model:Pixel_8_Pro transport_id:12 <- same model, different serial +``` + +Then **always pass `-Serial `** to every `deviceui.ps1` call so the action lands on the intended +device. If you omit `-Serial` while more than one device is attached, adb **errors out** ("more than one +device/emulator") instead of guessing — a safe failure, not a wrong-device action, but it will stall the run. +When the human owns several devices, ask which serial to use (or which model), and store each device's PIN +with **`secrets.ps1 set-device-pin`** — it shows a numbered picker (serial + model, so you can tell same-model +units apart) and saves the PIN under that device's own name `devicepin_`. Then `unlock -Serial ` +**auto-resolves** the matching PIN, so it always targets the correct device without you re-typing a secret name. +Serials are stable for a physical device across reconnects; emulator serials (`emulator-55xx`) +depend on the console port and can change between boots, so re-check `adb devices -l` at the start of a run. + +## Steps that need a fingerprint / biometric / App Lock + +`adb emu finger touch ` injects a fingerprint **only on an emulator**. A physical device has no adb +path to press the sensor — a human must do it. Microsoft Authenticator makes this bite because it +**enables App Lock by default** right after you register an account, so the *next* action (re-opening the +app, approving a browser number-match) is gated behind the device PIN/biometric. + +Options, best first: +1. **Run on an emulator.** Enroll once, then inject touches: + ```powershell + ./scripts/deviceui.ps1 finger-enroll -Serial # sets a PIN + enrolls (idempotent) + ./scripts/deviceui.ps1 finger -Text 1 -Serial # inject a touch when the prompt appears + ./scripts/deviceui.ps1 finger-status -Serial # verify one is enrolled + ``` +2. **Turn App Lock off** so no biometric is required: Authenticator → Settings → toggle **App Lock** off + (drive it with `tap-text`), then proceed with password-only steps. Do this early if the scenario + doesn't specifically test App Lock. +3. **Fall back to a device PIN.** If a keyguard/biometric prompt also accepts a PIN, set a known PIN during + setup and enter it with `deviceui.ps1 unlock` (many biometric prompts have a "Use PIN" path). Seed the PIN + once into the encrypted store, then let the tool type + **verify** it — it checks the keyguard actually + cleared and **stops after 3 tries** (`-MaxAttempts`, default 3) so a wrong PIN can't drive a physical + device into an escalating lockout: + ```powershell + ./scripts/secrets.ps1 set-device-pin # picks the device, you paste the PIN (DPAPI-encrypted) + ./scripts/deviceui.ps1 unlock -Serial # auto-uses devicepin_; verified; gives up after 3 + ``` + On an **emulator** you can set one deterministically: `adb -s shell locksettings set-pin 1234`, then + satisfy biometric prompts with `adb -s emu finger touch 1`. +4. **Probing an unknown PIN on a physical device — don't brute-force.** `adb shell locksettings verify --old ` + tests a guess **non-destructively** (it doesn't change anything), but Android's **Gatekeeper throttles + after ~5 wrong tries** and too many failures can lock the user out of their own device. Try at most a + couple of obvious values, and if they miss, **stop and ask the user for the PIN** — never loop guesses on + hardware you don't own. +5. **Physical device + human.** If the scenario *must* run on hardware, pause and ask the user to press the + sensor, then resume — see [Genuine blockers](#genuine-blockers-stop-and-ask). + +## Number-match MFA + +Modern MFA shows a **number in the browser** that you must select/enter in Authenticator. It's +automatable **only if** getting into Authenticator isn't biometric-gated: +- If App Lock is on → satisfy the biometric first (see above) → then read the number from the browser + (`deviceui.ps1 dump` on the Chrome page — it's **not** FLAG_SECURE) and tap the matching tile in + Authenticator. +- If the number appears as a **push you must approve on a *separate* physical phone**, it's a genuine + blocker (no TOTP seed) — ask the user. + +## Session timeouts & SSO resets mid-flow + +Switching between the browser and Authenticator (or a slow manual segment) can **time out the eSTS web +session**. Symptoms: you return to the browser and it's back on the password page, or SSO silently +re-prompts. Mitigations: +- Keep the biometric/PIN setup done **before** you start the timed web segment so you don't stall on it. +- If you get bounced to sign-in, just re-drive the email/password screens (SSO often re-completes quickly). +- Don't leave the flow parked while doing long setup on the side — provision the account/emulator first, + then run the web segment in one go. + +## Chrome autofill / passkey overlay steals input + +On eSTS **WebView** email/password fields, Chrome's autofill / **passkey** overlay can intercept a bulk +`input text` — the value lands in the wrong field or is dropped, and eSTS shows "Enter a valid email". +Fix (baked into `deviceui.ps1`): +```powershell +./scripts/deviceui.ps1 input-text -Text $upn -Clear -CharByChar -Serial +./scripts/deviceui.ps1 input-text -Text $pw -Clear -CharByChar -Secret -Serial +``` +`-Clear` empties the field, `-CharByChar` types one character at a time (defeats the overlay), `-Secret` +keeps the value out of the transcript. If a "Save password / use passkey" bottom sheet pops, dismiss it +with `key ESCAPE` (not `BACK`, which can exit the app) before typing. **Verify by whether the page +advances**, not by reading the field's `text` — a WebView often doesn't reflect typed content back in the +accessibility tree (the password field `i0118` is an exception and does show a length). + +## FLAG_SECURE black screenshots + +eSTS login pages, some broker screens, and **Microsoft Authenticator** set `FLAG_SECURE`, so `screencap` +returns an all-black image and some nodes are hidden. Don't rely on screenshots to verify these: +- Verify state with `uiautomator dump` + XML parse instead (the account list, a specific `resource-id`, + the focused activity via `deviceui.ps1 current-app`). Save the **XML** as your evidence artifact. +- Chrome pages are **not** FLAG_SECURE and screenshot fine — capture those normally. +- Full detail: [ui-interaction.md](ui-interaction.md#the-flag_secure-gotcha). + +## Screenshot corruption via redirection + +`adb ... screencap -p > file.png` through PowerShell **corrupts the PNG** (newline translation on the +binary stream). Always capture on-device then pull — which is exactly what `deviceui.ps1 screenshot` does: +```powershell +adb shell screencap -p /sdcard/_sc.png ; adb pull /sdcard/_sc.png # never `screencap -p > file` +``` + +## Single-use pairing / setup links + +MFA-setup "pair your account to the app" deep-links (e.g. from `aka.ms/mfasetup`) and QR codes are often +**single-use** — a second attempt shows "code already used" or a stale QR. If a pairing step fails on a +retry, **re-generate** the link/QR from the setup page rather than reusing the old one, or provision a +fresh user (`labapi.ps1 create-user`). + +**Finding a link that isn't in the accessibility tree.** The pairing hyperlink often lives **inside a +WebView** and does **not** appear as a tappable node in `uiautomator dump`, so `tap-text` can't find it. +It's rendered in the usual link **blue**, though, so detect it visually and tap by pixel: +1. `deviceui.ps1 screenshot` the Chrome page (Chrome is **not** FLAG_SECURE, so it captures fine). +2. Scan the PNG for link-blue pixels (Python + PIL) and tap the centroid of the topmost blue run. A blue-ish + test that works well: `b > 120 and (b - r) > 50 and (b - g) > 30 and r < 120`. +```powershell +python -c "from PIL import Image; im=Image.open(r'').convert('RGB'); w,h=im.size; px=im.load(); \ +ys=[(x,y) for y in range(0,h) for x in range(0,w) if (lambda r,g,b: b>120 and b-r>50 and b-g>30 and r<120)(*px[x,y])]; \ +print(min(ys,key=lambda p:p[1]) if ys else 'none')" +# then: deviceui.ps1 tap -X -Y (coords are in device pixels) +``` +Remember screenshot coords are **device pixels**; if you scaled the image, scale the tap back up. Once you +know the link's rough y-band on a given page you can re-tap it directly without re-scanning. + +## Stale account state between runs + +A lab account that already completed first-time MFA setup will **skip** the very registration step you +want to test, making the run look like it "passed" without exercising anything. Reset the state: +```powershell +./scripts/labapi.ps1 reset -Upn $upn -Operation mfa # clear MFA registration +./scripts/labapi.ps1 create-user -UserType GlobalMFA # or just get a brand-new temp user +``` +See [lab-api.md](lab-api.md). + +**Clean state comes from a fresh install, at the START of a case — not teardown at the end.** `pm clear` +does **not** remove work-account entries from AccountManager or an existing broker/WPJ registration; only +**uninstalling** the app does. So each case's clean slate = **uninstall → reinstall** the app-under-test (and +any broker it uses) as the first step. Do **not** try to delete accounts/registrations after a case — leave +them in place for the next run's uninstall (or a human) to clear. Combined with a **fresh temp account per +case**, this keeps runs independent without brittle post-run cleanup. Device-clock or policy state you changed +mid-run should still be restored so it doesn't leak into the next case. + +## Fresh temp user not sign-in-able yet (ESTS propagation lag) + +A brand-new temp user from `CreateTempUserID4SLab2` (or any just-created lab account) can take **several +minutes to become consistently sign-in-able**. During that window the sign-in page shows *"This username may +be incorrect. Make sure you typed it correctly."* and the error often **flaps** — it clears for one attempt, +then returns — because the account exists in Graph (you got an `objectId`) but ESTS replication across +front-end nodes lags behind. Observed lag has exceeded **14 minutes**. This is **not** a typo or a real +product failure, so per [Password rejected at sign-in](#password-rejected-at-sign-in--dont-reset-it) do **not** +reset or mutate anything. Options, in order: + +1. **Reuse an already-propagated temp user** you created earlier in the same run (they live ~60 min). For flows + where the account is incidental — e.g. a *browser* SSO test where only the MSAL APK folder differs between + test points — reusing one propagated user across both points is legitimate and does not weaken the + assertion. Note the reuse in the report. +2. **Pre-warm**: create the temp user **early** (at the start of the case, before installing/clean-stating apps) + so replication finishes while you do other setup. Better still, create the *next* case's user before you need it. +3. **Poll politely**: retry sign-in every ~60–90 s (re-type the UPN each time) rather than hammering in a tight + loop; tight retries just re-hit the same stale node. + +**Freshness gate — poll ≤ 3 min, then recreate or reuse a < 30-min-old user.** Don't wait indefinitely on a +stuck new account. If a freshly created temp user is not **consistently** sign-in-able within **3 minutes** of +polite polling, stop waiting on it and either **create another** temp user, **or reuse a previously created temp +user that is still very fresh — under ~30 minutes old** (comfortably inside its ~60-min TTL, and old enough to +have finished replicating; one that already signed in once is the safest reuse). Record the substitution — +which user you abandoned and which you used instead — in the report. This is a time cap on option 1–3 above, +not a replacement for them. + +If a case genuinely needs a *distinct fresh* account and none is sign-in-able even after recreating, record the +lag as a **PASS-with-note** (if you completed via reuse) or **BLOCKED** with the exact on-screen error and the +`objectId` as evidence — not a password reset. See [lab-api.md](lab-api.md). + +## Password rejected at sign-in — don't reset it + +When eSTS shows *"Your account or password is incorrect"*, *"Your password has expired"*, or *"That Microsoft +account doesn't exist"*, the reflex to "fix" it by resetting the password is almost always **wrong**. Resetting +mutates a shared lab account and can mask a real product bug. Treat a rejected password as a *value/identity* +problem first: + +1. **Re-fetch the value** — the shared tenant password may have rotated in Key Vault since you cached it: + `./scripts/labapi.ps1 fetch-password -TestTenant -IntoSecret `, then re-type with + `deviceui.ps1 input-text -SecretRef -Secret`. +2. **Check the identity** — confirm the UPN and tenant match exactly what the test case named (an easy slip is + using an `@msidlab4` account with the `id4slab2` password, or vice-versa). +3. **Check for a lockout** — a `Locked_…` prefix or repeated failures may mean a prior run locked the account; + provision a **fresh temp user** instead of hammering it. +4. **Re-type carefully** — special characters can drop in `-CharByChar` mode; verify the field length matches. + +Only run `labapi.ps1 reset -Operation password` when the **test case itself** exercises a password-change / +expiry flow. **Never** reset a **shared durable account** (e.g. `AndroidTBUser1@id4slab2`) — other cases reuse +it. If the value is confirmed correct and sign-in still fails, mark the run **BLOCKED** with the exact on-screen +error rather than changing the account. See [lab-api.md](lab-api.md). + +## Doing it yourself in System Settings + +Some test steps have **no adb command** — advance the device clock, delete a user certificate, toggle a +system setting, switch language, add/remove an account. That does **not** make them blockers: a human tester +just opens **Settings** and taps through, and so should you. Drive the Settings app with `deviceui.ps1` the +same way as any other screen (`dump` → `tap-text`/`tap-desc` → `input-text`) **before** you ever mark a step +BLOCKED. + +Open Settings (or jump straight to a sub-screen via its intent action): +```powershell +adb -s shell am start -a android.settings.SETTINGS # top-level Settings +adb -s shell am start -a android.settings.DATE_SETTINGS # Date & time +adb -s shell am start -a android.settings.SECURITY_SETTINGS # Security (credential/cert mgmt is under here) +adb -s shell am start -a android.settings.LOCALE_SETTINGS # Language +# then: deviceui.ps1 dump / tap-text / tap-desc to drive the screen +``` + +**Worked example — advance the clock >1h** (a common token-expiry step). Moving the *device wall clock* needs +**no root** (only `adb shell date` does): +1. `am start -a android.settings.DATE_SETTINGS` (Samsung path: Settings → **General management → Date and time**). +2. `dump` to confirm state, then `tap-text "Automatic date and time"` to turn it **off**. +3. `tap-text "Set date"` / `tap-text "Set time"`, set a value >1h ahead, confirm with **Done/OK**. +4. Do the app action you were timing, then restore automatic time afterward if later steps need real time. + +**When it *is* still a blocker.** Only fall back to BLOCKED when the surface is genuinely undrivable: +- A **secure/native system dialog** uiautomator can't read or act on — Samsung **Knox** certificate install, + the **keyguard** credential prompt, a **biometric** sensor, a PIN pad on a FLAG_SECURE screen. +- An action that truly needs **root** — e.g. expiring an app's *internal* monotonic cached-token timer + (`SystemClock.elapsedRealtime`), which the wall clock doesn't affect; that belongs on a rooted device or + emulator, not a retail phone. (Advancing the *wall clock* above is **not** this case.) +- A step that would leave **partial/destructive state** on a device you don't own (e.g. a real MDM + enrollment) with no clean rollback — pause and ask. + +Capture the exact screen (XML dump / screenshot) as evidence and mark just that step BLOCKED, per +[Genuine blockers](#genuine-blockers-stop-and-ask). + +## Genuine blockers (stop and ask) + +These can't be produced by the AI — report them and ask the user (see SKILL "When to ask the user"): +- **Real push-notification MFA** approval on a *separate* physical device with no TOTP seed. +- **SMS / phone-call OTP** to a real number. +- **Hardware** FIDO2 key, NFC, real camera/QR the emulator can't provide. +- **CAPTCHA** / "prove you're human". +- A **fingerprint on a physical device** that the scenario insists must run on that hardware. +- A credential/tenant/CA policy the AI can't provision (try `labapi.ps1 disable-policy` first if it's a + lab CA policy). + +## Quick reference table + +| Scenario / step | Automatable? | Where | How / workaround | +|---|---|---|---| +| Fingerprint / biometric prompt | ✅ emulator · ❌ physical | **Emulator** | `finger-enroll` + `finger`; on physical, human presses sensor | +| Authenticator **App Lock** re-prompt | ✅ emulator · ⚠️ physical | **Emulator** | inject biometric, or turn App Lock **off**, or use device PIN | +| Number-match MFA (same device) | ✅ if not biometric-gated | Either | read number from Chrome dump → tap tile in Authenticator | +| Real push MFA on another phone | ❌ | — | **Blocker** — ask the user | +| eSTS password typing | ✅ | Either | `input-text -Clear -CharByChar -Secret` | +| Autofill/passkey overlay | ✅ | Either | char-by-char + `key ESCAPE` to dismiss sheet | +| Verify state on FLAG_SECURE screen | ✅ | Either | `uiautomator dump` + XML (screenshot is black) | +| Session timed out mid-flow | ✅ | Either | re-drive sign-in; set up biometric/PIN before timed segment | +| Single-use pairing link reused | ✅ | Either | re-generate the link / fresh temp user | +| Pairing link inside a WebView (not in uiautomator) | ✅ | Either | screenshot → PIL blue-pixel scan → `tap -X -Y` | +| Emulator won't stay up (BT-HAL / system_server loop) | ⚠️ host-dependent | — | try boot recipe; if unstable, biometric step is a **blocker** on this host | +| Wrong-ABI APK on emulator (arm64 on x86_64) | ✅ | Emulator | install the **universal** APK; keep arm64 APK for physical | +| Stale MFA already registered | ✅ | Either | `labapi.ps1 reset -Operation mfa` or new temp user | +| Fresh temp user "username may be incorrect" (just created) | ✅ | Either | ESTS propagation lag (can be >14 min) — **don't** reset; poll ≤3 min then **recreate** or **reuse a <30-min-old** temp user; pre-warm early | +| System-settings step (advance clock, delete cert, toggle) | ✅ drive Settings UI · ❌ only if secure/root | Either | `am start -a android.settings.*` → `dump`/`tap-text`; block only on Knox/keyguard/root | +| Clean slate for a new case | ✅ | Either | **uninstall+reinstall** app-under-test (`pm clear` keeps work accounts/registration); don't tear down after | +| SMS / phone / hardware-key / CAPTCHA | ❌ | — | **Blocker** — ask the user | +| Multiple devices / same model attached | ✅ | Either | unique adb serial each; always pass `-Serial`; `adb devices -l` to pick; `secrets.ps1 set-device-pin` to store per-device PIN | +| Unlock the lock screen with a PIN | ✅ | Either | `secrets.ps1 set-device-pin` once, then `unlock -Serial ` (auto-uses saved PIN) — verifies + **stops after 3** tries | +| "Password incorrect / expired" at sign-in | ✅ | Either | **don't** `reset -Operation password`; re-`fetch-password` (may have rotated), re-check UPN/tenant, use a fresh temp user; reset **only** if the case says so; never reset shared durable accounts | diff --git a/.github/skills/android-e2e-tester/references/emulator-performance.md b/.github/skills/android-e2e-tester/references/emulator-performance.md new file mode 100644 index 00000000..1583c347 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/emulator-performance.md @@ -0,0 +1,127 @@ +# Emulator Performance & Android Studio Integration + +Table of contents: +- [Why the emulator is slow (and the #1 fix)](#why-the-emulator-is-slow-and-the-1-fix) +- [Diagnose your host in one command](#diagnose-your-host-in-one-command) +- [Fast-path: use a physical device](#fast-path-use-a-physical-device) +- [Making the emulator as fast as it can be](#making-the-emulator-as-fast-as-it-can-be) +- [Slow Play Store / network downloads](#slow-play-store--network-downloads) +- [Android Studio: Device Manager & Running Devices](#android-studio-device-manager--running-devices) + +## Why the emulator is slow (and the #1 fix) + +The dominant factor is **GPU rendering mode**: + +- **With a real GPU** the emulator uses **host (hardware) rendering** — UI, `screencap`, and `uiautomator dump` are fast. +- **Without a usable GPU** — typically a **Cloud PC / VM / RDP session** (e.g. a `CPC-*` Windows 365 box, which reports only a "Microsoft Hyper-V Video" / "Remote Display" adapter) — the emulator falls back to **SwiftShader software rendering** (the CPU draws every frame). Everything graphical is slow, and there is **no flag that makes a GPU appear**; `-gpu host` can't help when the host has no GPU. + +CPU virtualization (WHPX/HAXM/AEHD) is separate and usually still works on a VM (nested virtualization), so the CPU isn't the bottleneck — the **GPU is**. + +**#1 fix: run the scenario on a physical device instead of the emulator.** The skill's device pool includes +connected real devices, and on a GPU-less host `emulator.ps1 ensure` / `devicelease.ps1 acquire` will +**auto-prefer a connected physical device** for you. A phone renders on its own GPU and downloads over real +Wi-Fi, so flows that crawl on a software-rendered emulator (installing Company Portal, driving WebViews) +run in seconds. + +## Diagnose your host in one command + +```powershell +./scripts/emulator.ps1 resolve-sdk +``` +It prints the host GPU/perf profile, e.g. on a Cloud PC: +``` +host GPU: NONE -> emulator uses SLOW software (SwiftShader) rendering +host type: VM (Virtual Machine), RDP session, Cloud PC +emu defaults: -gpu swiftshader_indirect -cores 6 -memory 6144 +TIP: use a physical device for a fast run (emulator.ps1 ensure -PreferPhysical). +``` +If `host GPU` says NONE, expect a slow emulator and prefer a real device. + +## Fast-path: use a physical device + +1. Connect the device over USB (enable **USB debugging**) or Wi-Fi (`adb connect :5555`), and accept the + RSA prompt. Confirm it's in the pool: `./scripts/emulator.ps1 pool`. +2. Acquire/target it — on a GPU-less host it's automatic, or force it explicitly: + ```powershell + ./scripts/devicelease.ps1 acquire -Owner $AgentId -PreferPhysical -Wait # lease a real device + # or, without leasing: + ./scripts/emulator.ps1 ensure -PreferPhysical -Wait + ``` +3. Drive everything against that `$serial` exactly as with an emulator. + +**Caveats for real devices:** brokered flows with a **production** broker (Company Portal / Authenticator) +won't release tokens to a **debug-signed** calling app (caller-trust) — that's a signing/environment limit, +not a defect (see [troubleshooting.md](troubleshooting.md#signing--redirect-uri-mismatch-aadsts50011)). +Biometric prompts need a **real enrolled fingerprint** (the emulator-only `emu finger` simulation doesn't +apply) — see [ui-interaction.md](ui-interaction.md). Never print credentials; type them on the device. + +## Making the emulator as fast as it can be + +When you must use the emulator (no device available), these help — but won't beat a GPU-less ceiling: + +- **More cores/RAM.** `emulator.ps1` now auto-sizes `-cores`/`-memory` from the host (e.g. 6 cores / 6144 MB + on a big Cloud PC, up from a stock AVD's 4/1536). Override with `-Cores` / `-Memory`. +- **Keep the warm snapshot.** Don't pass `-ColdBoot` unless a snapshot is corrupt — a warm boot from + snapshot is much faster than a cold boot. +- **Explicit GPU mode.** The script auto-picks `host` when a GPU exists, else `swiftshader_indirect`. + Force with `-Gpu host` on a machine that has a GPU but defaulted to software. +- **Headless where possible.** `-NoWindow` skips the Qt UI (uiautomator/screencap still work); saves some + overhead, especially over RDP. +- **Use x86_64 images on x86 hosts.** ARM images on an x86 host run under slow translation. (On a physical + ARM phone, native arm64 is fine — build the matching ABI.) +- **Install the matching APK ABI.** On an `x86_64` emulator, prefer a **universal** APK + (`app-production-universal-release-signed.apk`). A large **`arm64-v8a`-only** APK can pass the ABI check + (the image lists `x86_64,arm64-v8a`) yet **crash install-time dexopt** — a Watchdog kill / `Broken pipe` / + `Can't find service: package`. Keep the arm64 APK for **physical** devices. Details in + [troubleshooting.md](troubleshooting.md#install-failures). +- **Know when to abandon the emulator.** On some GPU-less hosts the emulator boots but stays **unstable** — + the Bluetooth HAL crash-loops (`hci_backend_aidl.cc:40 initializationComplete`) and repeatedly restarts + `system_server`, so a heavy app can't be driven. `-feature -Bluetooth` may not fix it. Once you've tried the + [cold-boot recipe](troubleshooting.md#emulator-wont-start-or-boot) and it still won't settle, **switch to a + physical device** rather than burning time — accept that a biometric-gated step then becomes a blocker on + that host (see [common-blockers.md](common-blockers.md#decision-emulator-vs-physical-device)). +- **Raise timeouts, don't fight slowness.** On a slow host, prefer longer `wait-text` / boot / install + timeouts and poll for completion (e.g. `is-installed` in a loop) instead of fixed sleeps, so a slow-but- + correct step isn't misread as a failure. + +## Slow Play Store / network downloads + +Large downloads (e.g. Company Portal ~30 MB) can take **minutes** on an emulator that finishes in **seconds** +on a phone. Causes and mitigations: + +- The emulator routes through a **user-mode NAT** with its own DNS; throughput is limited and unrelated to + your real link speed. `-netspeed full -netdelay none` (already set) removes artificial throttling but can't + make the NAT fast. +- Software rendering also slows the Play Store UI itself. +- **Mitigation:** poll for install completion by package presence rather than a fixed wait; mind any + **time-boxed** feature step (a slow download can outrun a TTL — re-trigger with the app already installed). + For download-heavy flows, **use a physical device** — this is the clearest win. + +## Android Studio: Device Manager & Running Devices + +Two different things, and the skill already lines up with the first: + +- **Device Manager (the AVD list).** The skill creates AVDs in the **default AVD home** (`~/.android/avd`, + unless you set `ANDROID_AVD_HOME`) — the **same** place Android Studio reads. So **AVDs the skill creates + already appear in Android Studio's Device Manager**, and you can start/stop/wipe/delete them there. (Verify + with `./scripts/emulator.ps1 list` vs. Device Manager — same names.) + +- **Running Devices (the embedded mirror window).** This tool window mirrors an emulator that Android Studio + **launched into it**. The skill starts the emulator as a **standalone process** (its own window / gRPC + endpoint), so it shows up in `adb devices` and Device Manager but **not automatically inside the Running + Devices window**. Two ways to get parity: + + 1. **Start from Studio, let the skill reuse it (recommended).** In Studio: *Settings → Tools → Emulator → + "Launch in the Running Devices tool window"* (enable), then start the AVD from **Device Manager** — it + opens inside **Running Devices**. Now run the skill **without** `-Avd`/`-Serial`; `ensure` **prefers an + already-running emulator**, so it drives the exact device you see mirrored in Studio. (Confirm the skill + picked it: its `SERIAL=` matches `adb devices`.) + 2. **Mirror the skill's emulator.** With that same setting enabled, recent Android Studio can also mirror an + externally-started emulator in Running Devices once it detects it (it may prompt). If it doesn't appear, + use option 1 — it's the reliable path. + + Either way the skill and Studio share one emulator, one AVD list, and one adb — no duplication. + +> Net: if you're on a Cloud PC/VM (no GPU), the fastest, least-friction setup is a **connected physical +> device** (the skill auto-prefers it). If you want the emulator visible in Studio's **Running Devices**, +> start it from Studio and let the skill reuse it. diff --git a/.github/skills/android-e2e-tester/references/lab-api.md b/.github/skills/android-e2e-tester/references/lab-api.md new file mode 100644 index 00000000..c645a249 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/lab-api.md @@ -0,0 +1,227 @@ +# LAB API — Provisioning and Managing Test Accounts + +The **MSID LAB user-manager API** (`https://labusermanagerapi.azurewebsites.net`) creates and manages +lab test accounts for E2E runs. Drive it with `scripts/labapi.ps1`. The canonical, always-current list of +endpoints/parameters is the LAB **URL generator** web app: +. + +Table of contents: +- [Authentication (why a normal token fails)](#authentication-why-a-normal-token-fails) +- [Endpoints](#endpoints) +- [Account policy (prefer fresh ID4SLAB2 temp users)](#account-policy-prefer-fresh-id4slab2-temp-users) +- [`labapi.ps1` usage](#labapips1-usage) +- [Response shape (create-user)](#response-shape-create-user) +- [When to use which endpoint](#when-to-use-which-endpoint) +- [Entitlements](#entitlements) + +## Authentication (why a normal token fails) + +The API sits behind **Azure App Service Authentication (EasyAuth)**. A service token from +`az account get-access-token` is rejected with `consent_required` — EasyAuth wants an **interactive** +Entra session, not a raw bearer token. The LAB URL-generator page works because your browser is already +signed in; the endpoint validates against that existing session. + +`labapi.ps1` reproduces that: it opens the endpoint in **headless Microsoft Edge** +(`--headless=new --dump-dom`), which reuses your Entra **WAM SSO** session, then parses the JSON the +endpoint returns. It caches an isolated Edge profile under `%TEMP%\labapi_edge_profile` so the first call +may do a silent WAM handshake and later calls are fast. If a call comes back needing sign-in, run it once +in a visible browser to seed the session: +```powershell +./scripts/labapi.ps1 open -Url "https://labusermanagerapi.azurewebsites.net/api/CreateTempUserID4SLab2?usertype=GlobalMFA" +``` + +> **Edge `--dump-dom` regression (Edge 150+).** Newer headless Edge can return **0 bytes** for +> `--dump-dom` (and even `--version` prints nothing), which silently broke provisioning. `labapi.ps1` +> now **auto-falls back to the DevTools protocol (CDP)**: it relaunches Edge with +> `--remote-debugging-port`, finds the real `http(s)` page target (skipping `edge://` dialogs), opens a +> `System.Net.WebSockets.ClientWebSocket`, and reads `document.body.innerText` via `Runtime.evaluate`. +> This needs no Node/npm and works on Windows PowerShell 5.1+ and PowerShell 7. You don't do anything — +> if `--dump-dom` is empty the script switches automatically. If you ever need to debug it, `-Raw` prints +> whatever raw text the (dump-dom **or** CDP) path captured. + +> This is a **harness auth workaround**, not a code path under test. It only works on a machine where the +> operator is interactively signed in to Edge with an entitled account. + +## Endpoints + +Base URL: `https://labusermanagerapi.azurewebsites.net/api/` + +| Endpoint | Params | What it does | +|---|---|---| +| `CreateTempUserID4SLab2` | `usertype` | Creates a **temp cloud user** in the ID4SLab2 lab. **Auto-deletes after ~60 min.** Returns UPN + metadata. | +| `ResetID4SLab2` | `upn`, `operation` | Resets `mfa` or `password` for a user. **Password reset = temp users only.** | +| `EnablePolicyID4SLab2` | `upn`, `policy` | Enables a CA/special policy for a locked user. | +| `DisablePolicyID4SLab2` | `upn`, `policy` | Disables a CA/special policy for a locked user. | +| `DeleteDeviceID4SLab2` | `upn`, `deviceid` | Removes a device registration from Entra ID. | +| `List of Test Accounts` | `team` | KeyVault **deep-link** to the pre-created test-account JSON secret (team: `Android`, `JS`, `iOS`, `OneAuth`). Opens in the Azure Portal — use `labapi.ps1 open`. | +| `Fetch Password for Tenant` | `testTenant` | KeyVault **deep-link** to a tenant's password secret (`ID4SLAB2`, `ID4SLAB1`, `ARLMSIDLAB1`, `MNCMSIDLAB1`, `MSIDLAB4`, `MSIDLAB3`, `MSIDLAB8`). Opens in the Azure Portal. | + +**`usertype` values:** `Basic`, `GlobalMFA`, `MAMCA`, `MDMCA`, `MFAONSPO`, `MFAONEXO`, `FIDOBasic`, +`FIDOMDM`, `AuthappLBAC`, `AuthappRichContext`. + +**`policy` values:** `GlobalMFA`, `MAMCA`, `MDMCA`, `MFAONSPO`, `MFAONEXO`, `AuthappLBAC`, +`AuthappRichContext`. + +**`operation` values:** `mfa`, `password`. + +The two KeyVault items return **portal deep-links to a secret**, not a JSON API — they need an interactive +browser (`labapi.ps1 open`), and reading the secret needs the DevKV entitlement below. + +## Account policy (prefer fresh ID4SLAB2 temp users) + +**Default to a brand-new ID4SLAB2 temp user for every case** — even when a test case names a specific +**MSIDLAB4** (or other legacy-lab) account. MSIDLAB4 is being deprecated, its durable accounts carry state +from prior runs, and reusing one account across cases is exactly what produces the "already registered, step +skipped" false pass. `create-user` (→ `CreateTempUserID4SLab2`) hands you a clean, isolated account that +self-deletes in ~60 min, so there is nothing to tear down. + +Map the **capability** the case wanted onto an ID4SLAB2 `usertype` rather than reusing the named account: + +| Case asks for… | Use instead | +|---|---| +| MSIDLAB4 MAM-CA account | `create-user -UserType MAMCA` | +| MSIDLAB4 MDM-CA account | `create-user -UserType MDMCA` | +| A plain/basic Entra user (e.g. old `…?usertype=Basic` endpoint) | `create-user -UserType Basic` | +| Global-MFA user | `create-user -UserType GlobalMFA` | +| FIDO / passkey user | `create-user -UserType FIDOBasic` / `FIDOMDM` | + +Provision it **just before** the run (the ~60-min TTL), and pair it with a fresh app install for a truly +independent case. Only fall back to a specific named durable account when the case genuinely needs *that +identity's* pre-provisioned data (a seeded mailbox, a long-lived group membership) a temp user can't have — +then read its password with `fetch-password`, and **don't** mutate or tear it down. + +**Freshness gate — poll ≤ 3 min, then recreate or reuse a < 30-min-old user.** A just-created temp user can lag +ESTS replication and show *"This username may be incorrect"* at sign-in. Don't wait indefinitely: if the new +user isn't **consistently** sign-in-able within **3 minutes** of polite polling, either **create another** temp +user, **or reuse a previously created temp user that is still under ~30 minutes old** (inside its 60-min TTL and +already propagated — one that already signed in once is safest). Note the swap in the report; never "fix" the lag +with a password reset. See +[common-blockers.md → Fresh temp user not sign-in-able yet](common-blockers.md#fresh-temp-user-not-sign-in-able-yet-ests-propagation-lag). + +## `labapi.ps1` usage + +```powershell +# Create a fresh temp user of a given type (prints UPN=... on success): +./scripts/labapi.ps1 create-user -UserType GlobalMFA + +# Clear a stale MFA registration so a first-time-setup flow can be re-run cleanly: +./scripts/labapi.ps1 reset -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Operation mfa + +# Reset the password of a temp user — ONLY when the test case explicitly asks for a password reset/change flow. +# A rejected password at sign-in is NOT a reason to do this (see "When to use which endpoint" below). +./scripts/labapi.ps1 reset -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Operation password + +# Temporarily disable a CA policy that blocks a segment you're not testing, then re-enable it after: +./scripts/labapi.ps1 disable-policy -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Policy GlobalMFA +./scripts/labapi.ps1 enable-policy -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Policy GlobalMFA + +# Remove a stale device registration at the START of a run if a leftover Entra device would interfere: +./scripts/labapi.ps1 delete-device -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -DeviceId + +# Open a KeyVault deep-link (test-account list / tenant password) in a visible browser: +./scripts/labapi.ps1 open -Url "https://labusermanagerapi.azurewebsites.net/api/WebApp" + +# Fetch a tenant's shared password straight from Key Vault into the local DPAPI store (no paste, no browser): +./scripts/labapi.ps1 fetch-password -TestTenant ID4SLAB2 -IntoSecret labpw +# ...then type it on-device without it ever appearing in chat: +./scripts/deviceui.ps1 input-text -SecretRef labpw -Secret + +# Debug: dump the raw DOM Edge returned instead of parsing JSON: +./scripts/labapi.ps1 create-user -UserType Basic -Raw +``` + +Flags: `-TimeoutSec ` (Edge virtual-time budget, default 30), `-Fresh` (throwaway Edge profile instead +of the cached one). + +## Response shape (create-user) + +`CreateTempUserID4SLab2` returns JSON like: +```json +{ + "title": "User Creation Successful", + "userType": "GlobalMFA", + "upn": "Locked_5b335908a3@ID4SLab2.onmicrosoft.com", + "passwordUri": "https://ms.portal.azure.com/#@.../Microsoft_Azure_KeyVault/Secret/https://msidlabs.vault.azure.net/secrets/ID4SLAB2", + "credentialVaultKeyName": "https://msidlabs.vault.azure.net:443/secrets/ID4SLab2", + "tenantId": "c7cef333-42af-492c-afb0-21f74a661133", + "tenantName": "ID4SLab2.onmicrosoft.com", + "labName": "ID4SLab2", + "authority": "https://login.microsoftonline.com/", + "objectId": "2007370f-74dc-4e03-b4b6-148838cd4323", + "userObject": { "UserPrincipalName": "Locked_5b335908a3@ID4SLab2.onmicrosoft.com", "...": "..." } +} +``` +`labapi.ps1 create-user` echoes the full JSON and a convenience `UPN=` line. The **password** for a +temp user is the shared lab password (kept in the KeyVault the `passwordUri` points to). You do **not** need +the user to paste it: pull it from Key Vault directly with `labapi.ps1 fetch-password` (see next section), +which caches it DPAPI-encrypted and types it via `deviceui.ps1 input-text -SecretRef -Secret`. +**Never print the password into the transcript.** + +## Fetch a tenant password from Key Vault directly + +The "Fetch Password for Tenant" generator link is just an Azure Portal deep-link to a secret in the +`msidlabs` vault (e.g. `https://msidlabs.vault.azure.net/secrets/ID4SLAB2`). If you are signed into the +**Azure CLI** (`az login`) with an account that holds **`TM-MSIDLABS-DevKV`** (or equivalent vault GET +access), you can read that secret straight from the vault — no browser, no manual paste: + +```powershell +# Cache the ID4SLab2 shared password into DPAPI secret 'labpw' (prints only a masked length): +./scripts/labapi.ps1 fetch-password -TestTenant ID4SLAB2 -IntoSecret labpw +# -> "Fetched password into DPAPI secret 'labpw' (20 chars). ..." + +# Or pass the exact secret URI create-user returned in credentialVaultKeyName / passwordUri: +./scripts/labapi.ps1 fetch-password -SecretId "https://msidlabs.vault.azure.net/secrets/ID4SLAB2" -IntoSecret labpw +``` + +`-TestTenant` accepts the KeyVault secret names: `ID4SLAB2`, `ID4SLAB1`, `ARLMSIDLAB1`, `MNCMSIDLAB1`, +`MSIDLAB4`, `MSIDLAB3`, `MSIDLAB8` (override the vault with `-Vault`, default `msidlabs`). The command: + +- reads the value with `az keyvault secret show --id --query value -o tsv` (no parentheses in the + JMESPath, so it survives the `az.cmd`/cmd.exe wrapper), +- writes it **only** to the local DPAPI store (`%USERPROFILE%\.android-e2e-secrets\.sec`), the + same on-disk format `secrets.ps1` uses, and +- prints **only** a masked confirmation (`... (NN chars).`) — the plaintext never touches the host, + pipeline, logs, or git. + +Then `deviceui.ps1 input-text -SecretRef labpw -Secret` resolves it in-process and types it on the device. + +If `fetch-password` errors with a vault/auth message, it's a **setup blocker**: run `az login` with an +entitled account (see Entitlements below), or fall back to `secrets.ps1 set -Name labpw` and paste once into +the hidden prompt. This uses **your** entitled CLI session — if the token expires or you sign out, access +stops. + +## When to use which endpoint + +- **Need a clean account** → `create-user`. Remember it self-destructs in ~60 min; provision it just + before the run, not at the start of a long setup. Prefer this over any named MSIDLAB4 account (see + [Account policy](#account-policy-prefer-fresh-id4slab2-temp-users)). +- **A first-time-registration flow already registered on a prior attempt** (so the app skips the very step + you want to test) → `reset -Operation mfa` to clear MFA, or provision a brand-new user. +- **Sign-in says the password is incorrect or expired** → do **NOT** `reset -Operation password`. That message + almost always means you're typing the wrong *value* or targeting the wrong *account*, not that the account + needs changing. First re-pull the shared value with `fetch-password` (it may have rotated in Key Vault) and + double-check the UPN/tenant the case named; also check the account isn't `Locked_…` from an earlier lockout. + Only run `reset -Operation password` when the **test case itself** describes a password-change/expiry flow. + **Never** reset the password of a **shared durable account** (e.g. `AndroidTBUser1@id4slab2`) — other cases + reuse it and a reset breaks them. If the value is confirmed correct and it still fails, mark the run + **BLOCKED** with the exact on-screen error instead of mutating the account. +- **A CA policy blocks a segment you're not testing** (e.g. you want to test token acquisition but MFA + keeps interrupting) → `disable-policy`, run the segment, then `enable-policy` to restore state. +- **A device-registration test left a stale device** and the *next* run needs a clean slate → optionally + `delete-device` **at the start of the next run**. Per the clean-state policy you do **not** tear down + registrations at the end of a case — a fresh app install + fresh account already isolates the next run; + use `delete-device` only if a lingering Entra device object would actually interfere. +- **You need the exact UPNs of the durable, pre-created accounts** (not temp) → open the **List of Test + Accounts** KeyVault link for your team; get the tenant password via `fetch-password` (CLI, no paste) or + the **Fetch Password for Tenant** deep-link. + +## Entitlements + +Request/manage at : + +- **`TM-MSIDLabs-Ext`** — required for **all** LAB APIs. Needs both **RO** and **RW**. +- **`TM-MSIDLABS-DevKV`** — required to read the **Mobile Build Vault** (`msidlabs`): the KeyVault + deep-links *and* `labapi.ps1 fetch-password` (via `az`). Needs both **RO** and **RW**. + +If a call returns a sign-in/consent page instead of data, you're either not signed into Edge with an +entitled account or you're missing one of the above — that's a **user/setup blocker**, not a defect. diff --git a/.github/skills/android-e2e-tester/references/log-signals.md b/.github/skills/android-e2e-tester/references/log-signals.md new file mode 100644 index 00000000..d6a59ee2 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/log-signals.md @@ -0,0 +1,135 @@ +# Log Signals — Judging E2E Pass/Fail from logcat + +Table of contents: +- [How auth logging works](#how-auth-logging-works) +- [Capture discipline](#capture-discipline) +- [Success signals](#success-signals) +- [Failure signals](#failure-signals) +- [Operation flow markers](#operation-flow-markers) +- [Common AADSTS / error codes](#common-aadsts--error-codes) +- [Judging success per flow](#judging-success-per-flow) +- [Correlating with eSTS](#correlating-with-ests) + +`authlogs.ps1 scan` automates most of this. Use this doc to interpret the evidence it prints and +to reason beyond the heuristic verdict. + +## How auth logging works + +MSAL, Common, Broker, and ADAL all log through the shared **`Logger`** class (not `android.util.Log` +directly). Lines carry a component tag and, for most auth operations, a **correlation_id** (a GUID) +that ties a client request to the broker and to the eSTS token service. Verbose/PII logging is off by +default; tokens and secrets are redacted. Do not expect to see token values — presence of a +`correlation_id` plus a success marker is the signal, not the token itself. + +Typical tags to filter on: `MSAL`, `Broker`, `Common`, `ADAL`, `OneAuth`, plus operation classes like +`BrokerMsalController`, `CommandDispatcher`, `SilentTokenCommand`, `InteractiveTokenCommand`. + +## Capture discipline + +1. `authlogs.ps1 clear` **immediately before** running the scenario — otherwise stale lines pollute the verdict. +2. Run the scenario. +3. `authlogs.ps1 scan -Package ` — always pass `-Package` so pass/fail signals are + scoped to auth-relevant + app lines (reduces false positives from system noise). +4. Save every run's snapshot into the run folder so failures can be diffed across iterations. + +## Success signals + +| Pattern | Meaning | +|---|---| +| `executed successfully` | A broker/command operation completed | +| `AcquireToken...success` / `Token ... acquired` | Token obtained (silent or interactive) | +| `TokenResult ... SUCCESS` | Result object reports success | +| `Retrieved ... token from cache` | Silent cache hit | +| `Saved ... token` | Token cached after acquisition | +| `PRT is already registered` / `Loading Workplace Join entry` | Healthy device/PRT state | + +## Failure signals + +| Pattern | Meaning / likely cause | +|---|---| +| `FATAL EXCEPTION`, `E AndroidRuntime` | App crash — always a FAIL; capture the stack | +| `AADSTS\d+` | eSTS-side error (see codes below) | +| `error_code=` / `errorCode:` | Broker/MSAL error surfaced to caller | +| `No PRT present` | Missing Primary Refresh Token — silent auth will fail | +| `INTERACTION_REQUIRED` | Silent failed; interactive needed (may be expected) | +| `invalid_grant` | Token/refresh token rejected | +| `BrokerCommunicationException` | Client↔broker IPC broke (bind/permission/signature) | +| `NullPointerException`, `NoSuchMethodError`, `ClassNotFoundException` | Code/wiring bug in the change under test | +| `CertPathValidatorException` | TLS/cert issue (often network/proxy) | + +## Operation flow markers + +Trace which operations ran (helps localize where a flow broke): + +| Log marker | Operation | +|---|---| +| `GetDeviceModeMsalBrokerOperation` | Check if Shared Device Mode is enabled | +| `AcquireTokenSilentMsalBrokerOperation` | Silent token acquisition | +| `AcquireTokenInteractiveMsalBrokerOperation` | Interactive auth | +| `GetCurrentAccountMsalBrokerOperation` | Fetch signed-in account | +| `SignOutFromSharedDeviceMsalBrokerOperation` | SDM sign-out | +| `RemoveAccount` / `removeAccount` | Account removal | + +A healthy interactive sign-in usually shows: command dispatched → interactive operation → eSTS round +trip (correlation_id) → token saved → `executed successfully`. + +## Common AADSTS / error codes + +| Code | Meaning | Usually means | +|---|---|---| +| `AADSTS50011` | Redirect URI mismatch | App registration / signature / redirect config wrong | +| `AADSTS65001` | Consent required | Grant consent in the flow (AI can tap Accept) | +| `AADSTS50076` / `50079` | MFA required | Interaction/MFA step needed | +| `AADSTS50126` | Invalid username/password | Wrong test credential | +| `AADSTS700016` | App not found in tenant | Wrong client id / tenant | +| `AADSTS50058` | Silent sign-in failed, no session | Expected before an interactive sign-in | +| `AADSTS530021` | App not approved-client-app (CA) | Wrong/unapproved app config — see troubleshooting | + +`AADSTS50011`/`700016` typically indicate a **real defect** in the change under test → root-cause and +hand to the fix loop. `AADSTS65001`/`50076`/`50058` are often **flow steps**, not defects. +`AADSTS530021` is an **environment/config** block (the signed-in app config isn't CA-approved) that stops +the flow *before* the step under test — switch to an approved app config; do not treat it as a defect. + +## Broker telemetry keys worth grepping + +Brokered flows emit per-request telemetry that is often the clearest pass/fail signal — grep the snapshot +for these keys (values are logged as `Key: , Value: <...>`): + +| Key | Meaning | +|---|---| +| `broker_app_used` | `true` = the request was actually serviced through the broker; `false` = it wasn't (e.g. broker ineligible / not found) | +| `request_eligible_for_broker` | `true` = the request passed broker-eligibility and was routed to the broker | +| `is_successful` | Overall request success as the SDK reports it | +| `api_status_code` | e.g. `RequiredBrokerMissing`, `UserCanceled`, a success status | +| `auth_flow` | `Broker` vs non-broker path taken | + +For a brokered scenario, `broker_app_used=true` + `request_eligible_for_broker=true` is a strong PASS +signal that the broker path executed; `broker_app_used=false` with a `RequiredBrokerMissing` status +points at broker discovery/eligibility rather than the token exchange. + +## Judging success per flow + +- **Silent token (AcquireTokenSilent):** PASS = token retrieved from cache/refresh with a success + marker and no `INTERACTION_REQUIRED`/`No PRT`. If it returns `INTERACTION_REQUIRED` when a valid + account exists, that is a regression. +- **Interactive sign-in (AcquireToken):** PASS = the flow reaches the account/consent, completes, and + logs token saved + `executed successfully`. A lingering login page or an `AADSTS` error is a FAIL. +- **Sign-out / account removal:** PASS = removal operation runs and a subsequent `GetCurrentAccount` + shows no account. +- **Crash at any point → FAIL**, regardless of other signals. + +Do not declare PASS on the absence of errors alone — require a positive success signal that matches the +scenario. Absent both → verdict is INCONCLUSIVE; investigate (wrong package filter, logging off, or the +action never executed). + +## Correlating with eSTS + +When on-device logs are ambiguous, take the `correlation_id` from the snapshot and correlate with eSTS +telemetry using the **kusto-analyst** or **incident-investigator** skill. Basic query shape: + +```kql +AllPerRequestTable +| where env_time >= ago(1d) +| where CorrelationId == "" +| project env_time, CorrelationId, Call, Result, ErrorCode +``` diff --git a/.github/skills/android-e2e-tester/references/mocking-flights-and-segments.md b/.github/skills/android-e2e-tester/references/mocking-flights-and-segments.md new file mode 100644 index 00000000..7f3425f0 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/mocking-flights-and-segments.md @@ -0,0 +1,93 @@ +# Mocking, Flights & Segment Testing — Getting an E2E Run Unblocked + +Table of contents: +- [Principle: make it testable, don't fake a pass](#principle-make-it-testable-dont-fake-a-pass) +- [Temporary code changes — the revert discipline](#temporary-code-changes--the-revert-discipline) +- [Setting feature flags / flights](#setting-feature-flags--flights) +- [Mocking unavailable data or dependencies](#mocking-unavailable-data-or-dependencies) +- [Segment testing when the flow can't run end to end](#segment-testing-when-the-flow-cant-run-end-to-end) +- [Reporting mocked / segmented runs](#reporting-mocked--segmented-runs) + +When a run is blocked by something that isn't a defect in the code under test — a feature flag that's +off, a server API that isn't deployed yet, a piece in the middle that isn't implemented — **don't stop at +"blocked" and don't fake a pass.** Get as much of the feature under real test as you can by setting the +flag, mocking the missing input, or testing the flow in segments. Temporary code changes are allowed for +this, provided they're reverted and never committed. + +## Principle: make it testable, don't fake a pass + +- Prefer the **most real** option available: real data > a faithful mock at the network boundary > + a stub at the client boundary > a hardcoded value. The closer to real, the more the test proves. +- A mock must match the **actual contract** (field names, types, status codes) you're standing in for — + a mock that doesn't match the agreed server/API shape proves nothing. +- Never let a mock silently turn a red flow green. State clearly (in the report) which parts were real + and which were mocked, and what that means for confidence. + +## Temporary code changes — the revert discipline + +Making a temp change (flip a flag default, stub a response, inject a value) is fine **only** with this +discipline: + +1. Keep the change **in the working tree only** — never `git commit`/`git push` it, and never stage it + into a feature commit. +2. Leave a searchable marker at the edit site, e.g. `// TODO: REVERT — E2E mock, do not commit`. +3. **Revert after the run** — `git restore ` (or `git stash` the change while you commit real + work, then drop it). Verify with `git status` that the tree is clean of test-only edits before you + finish. +4. If the change is in a **library** the app consumes as a dependency, remember you must re-publish + + rebuild for it to take effect — see + [troubleshooting.md → Testing a local library change](troubleshooting.md#testing-a-local-library-change-publish-to-mavenlocal). + +## Setting feature flags / flights + +Get the flag into the state the scenario needs, cheapest option first: + +1. **A runtime override the app already exposes** — a test-app UI toggle, a config/spinner, `adb shell + setprop`, or a flights provider you can set at startup. Use this if it exists; no code change needed. +2. **Flip the flag's source default (temp).** If the app never installs a flights provider so every + lookup returns the coded default (a real gap we hit — see + [troubleshooting.md → Feature flags / flights](troubleshooting.md#feature-flags--flights-not-taking-effect)), + the only lever is the **default** in the library: `MyFlight("Key", true)`. Flip it, re-publish + + rebuild, run, then **revert**. Never commit the flip; the shipped default must stay as designed. +3. **Multiple flags:** set every flag the path reads, and confirm from logs that the flag-gated branch + actually executes (don't assume the flip took — verify the code path ran). + +## Mocking unavailable data or dependencies + +When a step needs data you can't produce naturally (a server response for an API that isn't deployed, a +service you can't trigger from the device), mock it at the **highest-fidelity** point you can: + +| Situation | Mock approach | +|---|---| +| Server API / endpoint not deployed yet | Point the client at a **local mock server** returning the agreed JSON, or use the repo's existing `MockWebServer`/interceptor test infra if present. Match the real contract exactly. | +| A single response **field** the server will add later | Inject it at the client's response-parsing boundary (temp code), using the agreed field name/shape, so the downstream path runs. | +| A collaborating app/broker you can't drive | Use a **mock broker** (`:mockcp`, `:mockauthapp`, `:mockltw` — see the app-and-module map) instead of the real one. | +| Data normally fetched at runtime (account, config) | Feed it via **adb intent extras** or a test hook the app exposes, or preload cache/prefs the app reads. | +| A push/callback you can't originate | Simulate the inbound intent/broadcast with `adb shell am start`/`am broadcast` carrying the expected payload. | + +Rules: the mock must be faithful to the real contract; gate it behind the feature flag or a debug hook +where possible; mark and revert it (above); and record in the report that the segment ran against a mock. + +## Segment testing when the flow can't run end to end + +If a piece in the middle is genuinely missing and **can't** be mocked faithfully, don't declare the whole +feature blocked — **test the segments that can run**, each with its own success criterion: + +1. **Segment before the gap:** drive the flow up to the boundary and assert the correct outputs at that + boundary (the right request was made, the right state/telemetry was produced, the parked/queued item + exists). That proves everything up to the gap. +2. **Segment after the gap:** feed the boundary inputs the missing piece *would* have produced (via a + mock/fixture) and assert the rest of the flow proceeds correctly. That proves everything after the gap. +3. Together the two segments cover the feature minus the exact missing piece; name the gap precisely and + what still needs an end-to-end pass once it's available. + +This is strictly better than "blocked": you localize exactly what's untested (the gap) and prove the rest. + +## Reporting mocked / segmented runs + +In the Phase 7 report, always state: +- **What was real vs mocked** (which flag was flipped, which data/dependency was mocked, at which boundary). +- **Which segments passed** and the success signal for each (with correlation_ids where relevant). +- **The remaining gap** — the exact piece that still needs a true end-to-end pass, and what unblocks it + (server API deployed, feature implemented, real credential/policy). +- **Confirmation that all temp changes were reverted** and the tree is clean. diff --git a/.github/skills/android-e2e-tester/references/run-speed.md b/.github/skills/android-e2e-tester/references/run-speed.md new file mode 100644 index 00000000..3f41aed4 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/run-speed.md @@ -0,0 +1,120 @@ +# Run-Speed Analysis — why steps are slow and how to shorten them + +An analysis of where wall-clock time goes in a UI-driven run, using the AAD MFA sign-in run +(`20260721_184540`) as the reference, plus concrete speed-ups. This is guidance, not a hard gate — but the +default driving pattern should follow the "fast path" below. + +Table of contents: +- [Measured timeline](#measured-timeline) +- [Where the time actually goes](#where-the-time-actually-goes) +- [Root causes](#root-causes) +- [Speed-ups (what to do differently)](#speed-ups-what-to-do-differently) +- [Fast-path recipe](#fast-path-recipe) +- [Expected savings](#expected-savings) + +## Measured timeline + +Timestamps are screenshot capture times; the gap is the time that segment took. + +| Segment | From → To | Elapsed | What happened | +|---|---|---:|---| +| App first-run | — → 01_firstrun 18:48:16 | (setup) | launch + accept privacy | +| Add account → sign-in page | 01 → 02_chrome 18:53:30 | **5m14s** | navigate menus, load eSTS WebView, type UPN | +| Enter password → post-sign-in | 02 → 03_after_signin 18:57:35 | **4m05s** | type password (char-by-char + retries), eSTS round-trips | +| MFA wizard | 03 → 04_mfa_wizard 18:58:11 | 36s | "More info required" wizard | +| Pairing | 04 → 05_pairing 18:58:54 | 43s | pair account | +| MFA challenge | 05 → 07_mfa_challenge 19:03:33 | **4m39s** | switch to browser, re-auth, number-match appears | +| Number match | 07 → 08_number_match 19:04:55 | 1m22s | read number, attempt approval (blocked by App Lock) | + +**UI total ≈ 16m40s**, plus **≈3–5m** of setup (provision account, install/verify APK, device lease). +A human does the same flow in **~3–4 minutes**. The gap is almost entirely **harness overhead**, not the +device or the network. + +## Where the time actually goes + +The three fat segments (5m14s, 4m05s, 4m39s) share the same shape — they're dominated by the +**observe→act loop overhead**, not by anything the app is doing: + +1. **Per-action adb round-trips.** A single "tap the button labeled X" is really: `uiautomator dump` → + `exec-out cat` the XML → parse → compute center → `input tap x y` → **fixed `Start-Sleep`** → often a + re-`dump` to confirm. That's 4–6 adb invocations and a hard sleep for **one** logical action, and each + screen has several actions. +2. **Fixed sleeps instead of polling.** Waiting a flat `Start-Sleep -Seconds 3–5` after every tap "to be + safe" is the single biggest tax. Screens that were ready in 300 ms still cost the full sleep; multiply + by dozens of actions across a run. +3. **Fresh process per tool call.** Every `powershell` tool call is a brand-new process that re-resolves + `adb`, re-reads env, and re-establishes the adb client each time — hundreds of ms of pure startup, paid + on every micro-step because steps were issued as separate calls. +4. **Char-by-char typing.** The autofill/passkey overlay forces one-character-at-a-time input at ~55–60 ms + per character; a 20-char password + a UPN is a couple of seconds *just typing*, before any verification. +5. **Screenshot capture + pull + view.** `screencap` on-device → `adb pull` → open the PNG is ~1–2 s each; + taking one after every step to "see what happened" adds up fast (and is wasted on FLAG_SECURE screens + that come back black). +6. **Retries when input didn't land.** WebView typing that silently dropped (autofill) triggered re-typing + and re-verification — the 4m05s password segment is mostly this. +7. **Genuinely slow bits (unavoidable-ish).** eSTS WebView first paint, the switch to the browser for the + challenge, and any re-auth after a session timeout are real seconds — but they're the minority. + +## Root causes + +- **Chatty, one-action-per-call driving** with a fresh shell each time → startup + round-trip cost paid + hundreds of times. +- **Pessimistic fixed sleeps** substituting for readiness signals. +- **Verify-by-re-dump / verify-by-screenshot** after every action instead of only at decision points. +- **Overlay-forced char-by-char** typing on every field, even when bulk would have worked. +- **No reuse** of the uiautomator dump: we re-dump for the next action instead of reusing the XML we just + fetched. + +## Speed-ups (what to do differently) + +Ordered by payoff: + +1. **Keep one long-lived shell for a whole screen/segment.** Batch the dump→parse→tap(s) for a screen into + a *single* `powershell` call (an async session you reuse) so you pay process/adb startup once per + segment, not once per tap. This alone removes most of the fresh-process tax (root cause 3). +2. **Replace fixed sleeps with anchor polling — and fuse tap+wait.** `deviceui.ps1 wait-text -Text + ""` returns the instant the screen is ready instead of always waiting N seconds. + Better still, `tap-text "