E2e pipeline overhaul

[vadim-v]

A structural rework of .github/workflows/NativePipeline.yml and its Maestro test
harness. The goal is to make the native end-to-end suite fast, reproducible, and
honest — today it is slow and fails the majority of runs, mostly for infrastructure
reasons that mask the handful of real widget problems underneath.

This document walks reviewers through the most impactful changes, in rough order of
impact, with representative diffs. Per-flow failure-video recording (§9) earns its
keep: it's what turned an opaque "couldn't find element" failure into a visible diagnosis
of the trial-license session cap described in § What this surfaced.
It is no longer treated as optional.

Scope: 30+ commits, ~170 files. Most of the file count is one-line
assertScreenshot additions across 52 Maestro flows (see §3); the structural
surface area is small and concentrated in the workflow + a few composite actions.

---

TL;DR for reviewers

#	Change	Why it matters
1	Self-contained runtime via Portable App Distribution (PAD)	Deletes a fragile hand-rolled runtime (m2ee-tools clone + CDN download + fake health check). One supported artifact, real readiness check.
2	Native bundling via mxbuild --native-packager	Collapses two duplicate bundle jobs into the build stage using the supported target.
3	Visual regression moved into Maestro (assertScreenshot)	Deletes a custom pixel-diff script + a whole decoupled job; each widget shard now owns its own visual result.
4	Fast-fail test policy	Kills retry/timeout amplification (a broken shard went from ~25–60 min to ~1–5 min) and adds an up-front smoke check.
5	Caching & snapshots	PAD package, pre-booted AVD snapshot, content-hash dist cache, Gradle/Pods/NDK caches.
6	Reproducibility & robustness	iOS device pin-with-fallback, bounded Android boot poll, pinned actions, concurrency.
7	Observability	Combined artifacts + a $GITHUB_STEP_SUMMARY pass/fail table.
8	Tooling cleanup	Drop the Python toolchain (Node-only scripting); fail loudly instead of silently pinning master.
9	Per-flow failure videos	BrowserStack-style: record every flow, keep only failures. Already paid for itself — the video is what diagnosed the session-cap wedge (§ What this surfaced).

Please also read § What this surfaced and
§ Known limitations before reviewing — they explain why some
runs are still red and what is deliberately left for follow-up.

---

1. Self-contained runtime via Portable App Distribution (PAD)

Before: the test job assembled a runtime by hand — cloning m2ee-tools,
downloading the runtime from a CDN, and using a config file + a health check that
didn't actually confirm the runtime was serving. Fragile and version-drift-prone.

After: the build stage produces a self-contained portable app package
(mxbuild --target=portable-app-package → app.zip, which bundles the Java runtime
and a bin/start). The test job just extracts it, starts it, and polls until it
truly serves HTTP 200 before any test runs.

 .github/scripts/setup-runtime.sh                  | deleted
 .github/scripts/start-runtime-with-verification.sh | deleted
 configs/e2e/m2ee-native.yml                        | deleted
+.github/actions/start-mendix-runtime/action.yaml   | added

New runtime action (the real readiness gate, abridged):

# .github/actions/start-mendix-runtime/action.yaml
unzip -qq "$APP_PACKAGE" -d runtime-app
chmod +x runtime-app/bin/start
(cd runtime-app && nohup ./bin/start etc/Default > "$GITHUB_WORKSPACE/log/runtime.log" 2>&1 &)

deadline=$(( SECONDS + TIMEOUT_SECONDS ))     # default 300s
until curl -fsS -o /dev/null "$RUNTIME_URL"; do
  if [ "$SECONDS" -ge "$deadline" ]; then
    echo "::error::Mendix runtime did not become ready within ${TIMEOUT_SECONDS}s"
    tail -n 100 "$GITHUB_WORKSPACE/log/runtime.log"
    exit 1
  fi
  sleep 3
done

Built once in the project job and shared as the app-package artifact:

+      - name: "Build portable app package (self-contained runtime)"
+        run: |
+          mxbuild --target=portable-app-package \
+            -o Native-Mobile-Resources-main/app.zip \
+            --loose-version-check Native-Mobile-Resources-main/NativeComponentsTestProject.mpr

Validated locally in Docker: produces a ~451 MB self-contained zip, boots and serves
HTTP 200 on :8080 in ~18s with JAVA_HOME + M2EE_ADMIN_PASS + ./bin/start etc/Default.

---

2. Native bundling via mxbuild --native-packager

PAD ships the server runtime, not the RN JS bundle. Previously two near-identical
android-bundle / ios-bundle jobs (plus a create-native-bundle composite) produced
those bundles via an internal-tools hack. This replaces them with the supported
mxbuild --target=deploy --native-packager target, run in the build stage.

 .github/actions/create-native-bundle/action.yml | deleted

+      - name: "Build native bundles (deploy + native-packager)"
+        run: |
+          mxbuild --target=deploy --native-packager \
+            --loose-version-check Native-Mobile-Resources-main/NativeComponentsTestProject.mpr

Ordering gotcha (worth a reviewer's eye): portable-app-package re-prepares and
cleans deployment/, which would wipe freshly-built native bundles. So the
portable app is built first (its app.zip lives outside deployment/), and the
native-packager deploy runs last. Bundle uploads use if-no-files-found: error
so a missing path fails the build loudly instead of silently shipping nothing.

---

3. Visual regression moved into Maestro (assertScreenshot)

Before: flows only did takeScreenshot; a separate compare-screenshots job later
diffed PNGs with a custom pixelmatch script (maestro/helpers/compare_screenshots.js).
A decoupled gate with gh run download || true gaps and three extra npm deps.

After: each flow asserts visually in-line with Maestro's built-in
assertScreenshot (default 95% threshold). The widget's own shard owns its visual
result — no separate job, no extra deps.

 - takeScreenshot:
     path: "maestro/images/actual/${PLATFORM}/animation"
+- assertScreenshot:
+    path: "maestro/images/expected/${PLATFORM}/animation.png"

 maestro/helpers/compare_screenshots.js | deleted
 package.json: -pixelmatch -pngjs -@actions/core   (3 devDeps removed; lockfile refreshed)

This one-line addition is repeated across 52 flows — that's the bulk of the file count.

---

4. Fast-fail test policy (the biggest runtime/stability lever)

The old harness amplified every failure: up to 4 attempts per flow (initial + 3
retries) × a 5-minute driver-startup timeout × 240s × 3 precondition waits, all
under a 60-minute job cap. A broken build could grind for ~25–60 minutes before failing.

Changes:

• MAX_RETRIES 3 → 1, driver-startup timeout 300s → 120s.
• Precondition waits trimmed (240s×3 → 60s×2).
• Job timeout-minutes 60 → 20 (iOS) / 30 (Android) — backstop only.
• New up-front smoke check (Smoke.yaml): launch the app and confirm the Widgets
  menu renders once before running any flow. A fundamentally broken build now fails
  in ~1 min instead of grinding every flow.

-MAX_RETRIES=3
+MAX_RETRIES=1
-MAESTRO_DRIVER_STARTUP_TIMEOUT=300000
+MAESTRO_DRIVER_STARTUP_TIMEOUT=120000

+# Fast-fail smoke check: verify the app launches and the Widgets menu renders ONCE,
+# before running any widget flows. Retries ONCE (resetting the sim/emulator) to absorb
+# the transient Maestro driver-attach flake, then aborts the shard if still broken.
+if ! smoke_check; then
+  exit 1
+fi

Net effect: worst-case broken shard ~25 min → ~1–5 min, and healthy shards finish well
under the cap.

---

5. Caching & snapshots

Several independent caches, each keyed on the inputs that actually invalidate it:

• Portable app package — built once, consumed by every test shard (§1).
• Pre-booted Android AVD snapshot — a dedicated android-avd-cache job is the sole
  writer; test shards restore read-only (no thundering-herd re-saves).
• Content-hash dist cache — on a full build (full_build=true: infra-only
  PRs, JS-action-only changes, or a full dispatch) an exact source-hash hit lets the job
  skip the rebuild entirely. On a partial (widget) PR the job does a fast scoped build
  of just the changed widgets via pnpm --filter instead.
• Build scope excludes e2e/ changes — the test scope and the build scope are now
  separate. A change confined to a widget's e2e/ folder (Maestro flows / screenshots)
  changes only the test, not the built artifact, so it no longer rebuilds the
  widget — the widget is still added to the test matrix and runs against the test
  project's baseline .mpk (exactly like every other not-rebuilt widget in a partial
  run). Concretely, this very PR (mostly e2e/*.yaml + infra) drops from "rebuild all 20
  widgets" to "rebuild 0". A safety-net manifest keeps the resources artifact non-empty
  when a run builds nothing, so the project download never fails.
• Gradle / CocoaPods / NDK caches with restore-keys fallback.

# android-avd-cache job — single writer; shards restore-only against this exact key
key: avd-android-35-x86_64-google_apis-pixel

# resources: rebuild only when widget/JS-action sources or deps actually change
key: resources-dist-v1-${{ hashFiles(
  'packages/pluggableWidgets/*/src/**', 'packages/jsActions/*/src/**',
  'packages/pluggableWidgets/*/package.json', 'packages/jsActions/*/package.json',
  'pnpm-lock.yaml') }}

---

6. Reproducibility & robustness

• iOS device pin-with-fallback: prepare_ios.sh prefers a pinned device
  (PREFERRED_IOS_DEVICE, default iPhone 17 Pro — what macos-26 ships and baselines
  are captured on) for reproducible screenshots; falls back to the newest available
  device with a loud ::warning:: if absent. Avoids both the old hardcoded hard-fail
  and silent device drift.
• Bounded Android boot poll: replaced a blind sleep 30 with a state-based wait
  (adb wait-for-device → sys.boot_completed → package manager ready, 300s cap).
• Pinned test project: mendix/Native-Mobile-Resources is downloaded at a pinned
  commit SHA (NATIVE_MOBILE_RESOURCES_REF, overridable via the test_project_ref
  dispatch input) instead of a moving main, so a build is reproducible.
• Robust PR scoping: the scope job checks out full history and diffs against the
  merge-base with the PR base branch, so every commit in a multi-commit PR is
  considered (it previously diffed only HEAD~1 via an empty github.event.before,
  silently under-scoping). No third-party action added — done in the existing script.
• Pinned third-party actions (by SHA), a single runner image family, and a
  concurrency block to cancel superseded runs.

---

7. Observability

• Combined artifacts: an aggregate-test-results job merges the dozens of per-shard
  outputs into a handful — all-screenshots, all-runtime-logs, all-debug-artifacts
  — so reviewers download once instead of per-shard.
• $GITHUB_STEP_SUMMARY: the same job writes a consolidated per-shard pass/fail
  table (read back via gh api --paginate over the run's jobs) to the run Summary tab.
• Failure artifacts actually upload now: the Archive test results steps were
  missing if: always(), so a failing shard (run script exits non-zero) skipped
  archiving — meaning screenshots/logs were captured only for green shards. Fixed.

---

8. Tooling cleanup — drop Python, fail loudly

• Ported determine-nt-version.py → determine-nt-version.mjs (global fetch + a tiny
  version comparator, no deps, runs on the runner's preinstalled Node). Deleted the
  dead setup-python action and inline pip install. Python is now fully gone.
• The version resolver previously fell back to a silent nt_branch=master (an unpinned
  moving target) on no-match or a transient API blip. It now exits 1 loudly instead.

-          pip install requests packaging
-          python ./.github/scripts/determine-nt-version.py "$VERSION"
+          node ./.github/scripts/determine-nt-version.mjs "$VERSION"

Deliberate scope call: the remaining shell glue (determine-widget-scope.sh,
maestro/helpers/*.sh) is kept as-is. The standardization win was eliminating the
third toolchain (Python), not churning battle-tested bash into JS for its own sake.

---

9. Per-flow failure videos

BrowserStack-style debugging: record a screen video around every flow, keep it
only if the flow fails, delete it on pass. Each kept clip is a short, targeted
recording of exactly the failing interaction — far more useful than a "couldn't find
element" log line. Toggle off with RECORD_VIDEO=false.

This feature already paid for itself. The trial-license session cap in
§ What this surfaced was invisible in the Maestro logs (just
"couldn't find Widgets menu"); the failure video showed the app sitting on the
"maximum number of users" screen, which is what led straight to the runtime-log
confirmation and the fix. It earns its place in the suite.

• iOS: xcrun simctl io booted recordVideo (SIGINT-finalized, no length cap).
• Android: adb shell screenrecord --time-limit 180 (3-min/segment cap). The recorder is
  stopped with adb shell pkill -INT screenrecord (stopping only the local adb client
  leaves the on-device recorder running to its time-limit — which earlier added
  ~2.5 min/flow and blew the shard timeout).

+start_recording "$(basename "${yaml_test_file%.yaml}")"
 if maestro test ... "$yaml_test_file"; then
   echo "✅ Test passed"
+  stop_recording discard          # delete the clip on pass
 else
   echo "❌ Test failed"
+  stop_recording keep             # retain the clip for debugging
 fi

Merged into a single all-failure-videos artifact alongside the others. It's
self-contained (one toggle, no coupling to the rest of the PR) — but on the strength of
the session-cap diagnosis it's a keep, not a cut.

---

What this surfaced (please read)

Tightening retries/timeouts and adding the smoke check did not introduce new
failures — it un-masked an existing one, and the root cause turned out to be the
Mendix trial license, not the widgets.

The runtime runs under a trial license capped at 6 concurrent named users. Every
Maestro flow does launchApp clearState: true, which clears device state but starts a
fresh server-side session that lingers for the default 10-minute SessionTimeout.
A widget's flows therefore accumulate one session each. The residual failures cluster on
exactly the widgets whose flow count pushes a shard past 6 live sessions (smoke launch +
flows + retries):

Shard	Flows (+smoke)	Result
background-image-native	8 (+1)	over cap → wedge
safe-area-view-native	7 (+1)	over cap → wedge
accordion-native	6 (+1)	over cap → wedge
js-actions (mobile-resources 5 + nanoflow 3)	8	over cap → wedge
every other widget	1–5	under cap → green

The runtime log is unambiguous: after 6 sessions it logs Maximum number of sessions exceeded! (You are currently using a trial license), the next launchApp lands on the
user-limit screen, Precondition can't find the "Widgets" menu, and the shard
"wedges" mid-suite. That is the same symptom previously mis-attributed to image-rendering
OOM / per-widget instability — it isn't widget-specific at all.

Why fast-fail exposed it: the old config's 4× retries and long timeouts stretched a
shard's flows past the 10-minute session window, so early sessions expired and freed
license slots before later retries ran — eventual green, just slow and noisy. Fast-fail
concentrates all 6–9 launches inside the 10-minute window, so every session is alive at
once and the cap trips. The speed-up didn't add instability; it removed the accidental
delay that was hiding a session-cap.

Fix: the start-mendix-runtime action now shortens the server session idle timeout
(RUNTIME_PARAMS_SESSIONTIMEOUT=60000) and reaps expired sessions promptly
(RUNTIME_PARAMS_CLUSTERMANAGERACTIONINTERVAL=30000) via the portable app's HOCON env
overrides. At ~30s/flow, each flow's session expires before the next ones pile up, holding
peak concurrency to ~2–3 — comfortably under 6 — without slowing the suite or touching any
test. (Identified from run 28168923730's safe-area-view runtime log; pending a clean
validation run.)

---

Known limitations / deliberately deferred

• Two screenshot baselines are stale. background-image-native baselines could
  not be re-captured on earlier runs because its 8-flow shard tripped the trial-license
  session cap (see § What this surfaced) and wedged mid-suite — the
  capture run never produced clean screenshots. With the session-timeout fix these should
  now be re-capturable; re-baselining is pending a clean validation run.
• No mid-suite fast-abort yet — the smoke check guards shard start only. With the
  session-cap fixed this is far less pressing, but a bail-after-N-consecutive-precondition-
  failures backstop is still worth adding for genuinely wedged apps.
• Android video segments cap at 180s (screenrecord limit) — long flows truncate;
  still useful, but not unbounded like iOS.
• e2e-only changes test against the baseline widget, not a fresh build. Because an
  e2e/-only change no longer rebuilds its widget (see §5), that widget's e2e runs
  against the test project's committed baseline .mpk. This matches how partial builds
  already treat every non-rebuilt widget, and is correct when the repo's widget source
  equals the released baseline; a widget with unreleased source changes on main and a
  test-only PR would be tested against the older released widget. (The same e2e/
  exclusion is not applied to JS-action packages — their js_actions_changed flag
  drives both build and test, and they're only two cheap packages.)
• Open backlog (not in this PR): de-duplicating the
  4 near-identical test jobs into a reusable workflow; per-PR-branch version overrides;
  more aggressive Gradle native-build caching (.cxx/externalNativeBuild).

---

Validation status

• PAD + native bundling: validated locally in Docker and on CI capture runs.
• assertScreenshot, fast-fail, caching, combined artifacts, step summary, Node port:
  exercised on dispatch/PR runs on the fork.
• 52 android / 51 iOS baselines captured on the pinned devices; 2 stale as noted above.