Repository navigation
fix(provider-webdriver): address TestMu AI provider review - #2
Merged
amankansal-lt merged 9 commits intoOct 2, 2026
Merged
Conversation
The registry records a lease, and stamps its expiry, before the lease lifecycle provider allocates the session behind it. Hosted providers can spend 30-80 seconds creating that session, so a lease on the default one-minute inactivity window was expired, or nearly so, by the time its client received it: the next command failed with "Lease is not active", release reported nothing to release, and the paid provider session was left running. The expiry sweep could also reap the lease while the provider was still allocating. Hold a lease work pass for the duration of the provider allocation, the same protection admitted request work gets. The lease cannot expire underneath the allocation, and ending the pass while the requester is still waiting renews the lease for its own TTL from that moment. The response now carries the renewed lease. A requester that hung up still protects nothing, and its allocation is released as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r consumes Lease allocation forwards every provider profile field, but only some routes checked which ones a provider could act on. The connect builders and the BrowserStack/AWS session preparation refused flags they did not own, while the Limrun lease runtime and a remote-config profile skipped that check. A typed client asking Limrun for providerDeviceType 'real' was therefore given a simulator without an error. Each lease provider now declares, for every Cloud provider profile field, whether it consumes or refuses it. The declaration is a total record over CloudProviderProfileFields, so adding a field fails to build until every provider decides. One refusal, derived from that declaration, runs at lease preparation (the WebDriver session manager for BrowserStack, AWS Device Farm and TestMu AI, and Limrun allocation) and at connect, so connect, leases.allocate and remote-config profiles all fail the same way with INVALID_ARGS naming the flag and the provider. This replaces the per-provider TestMu-only, TestMu-unsupported and BrowserStack-only checks, the hand-copied complement of the BrowserStack feature table, and the ./testmu-device-features package subpath. AWS Device Farm now also refuses the hub-only app, OS version, project and build flags it never read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…le file The WebDriver deployment runtime guessed what to upload for a materialized iOS build from file extensions: an extracted .app with a .zip or .ipa archive uploaded that archive. The archive it saw is the outermost one the source arrived as, so a URL .zip wrapping an .ipa uploaded the wrapper zip instead of the .ipa, and a zip holding a .app.tar.gz uploaded a zip with no app bundle at its top. Materialization now records the archive an installable was extracted from directly, and the Apple materializer declares the file a hosted provider uploads: the .ipa itself, or the zip a simulator .app was extracted from. It names nothing when no such file exists. The deployment runtime uploads the declared file and otherwise the installable path, with no inference of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nput handling Several inputs reached a hosted provider in a shape it could not use, or failed with an untyped error: - An empty `lt://` passed as a TestMu AI app reference and reached session creation. A reference is now validated against the app-id grammar, and an invalid one is INVALID_ARGS on connect and at session preparation. - `LT://APP1` or `BS://id` was treated as a file path. App schemes now match case-insensitively and are canonicalized to the spelling the hub accepts. - A directory given as `--provider-app` reached the upload and failed with a raw EISDIR; it is now INVALID_ARGS with a hint. - A missing path given straight to the TestMu AI upload failed with a bare ENOENT; it now gets the same typed refusal as a directory. - Session-detail URLs were built by string concatenation, so a query on TESTMU_API_ENDPOINT or the BrowserStack details endpoint swallowed the route. Routes are now appended to the URL path, keeping the query. - A whitespace-only artifact URL was reported as a ready artifact; it is now treated as absent, and URLs are trimmed. - A 2xx connection-verification answer that is not JSON was reported as a network failure. It is now COMMAND_FAILED with the HTTP status, and each hub supplies its own service-status hint. The connect help and the TestMu AI guide now say what connect checks for `--provider-app` and what is only validated when the session is created. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… real The BrowserStack session-details timeout test threw a prebuilt TimeoutError from the fetch stub, so it passed even with the deadline removed. The fetch now stays pending until the signal the lookup armed aborts, and the test checks that signal is the 15 second deadline. The TestMu AI upload cancellation test aborted before the upload reached fetch, so it never covered a request in flight. It now waits for the stub to start before aborting. The connect suites shared a copied connectWithGeneratedProviderProfile helper; it now lives in the shared test utilities. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… too The daemon hands a run's repeat lease_allocate the lease it already holds, and both the WebDriver runtime and the Limrun runtime returned the live session for a known lease before checking the request's profile fields. A second allocation on the same run that set a refused field, such as providerDeviceType on BrowserStack, therefore succeeded. Both runtimes now refuse before reusing a live session. Because a refusal of that repeat request is a provider allocate failure, the lease handler no longer releases a lease it reused when allocate throws; doing so would end the run's lease and leave the session the first allocation created without an owner. The runtime's profileFields option is now required, and each provider definition passes its own declaration into createRuntime, so a provider that forgets to wire its declaration fails to compile instead of silently refusing nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…connect Connect only checked that an lt:// or bs:// reference had something after the scheme, so `lt://a b` and `lt://a/b` were saved and failed only when the hub created the session. Both hubs' reference grammars now live in the light providers module, and connect, session preparation and the TestMu AI upload reader all validate against them. Connect also verified the raw spelling typed on the command line: the generated profile stored `lt://APP1` for `LT://APP1`, but the flags handed to verification were the raw CLI flags laid over the profile, so the uploaded-app lookup missed and reported a local artifact. The canonical reference now wins in the flags verification reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A materialized build that names no uploadable file falls back to its installable path, which for iOS is the extracted .app directory. The deployment runtime handed that to the hub's uploader, and BrowserStack's read it as a file and failed with a raw EISDIR. The deployment runtime now refuses a directory before calling any hub's uploader, with INVALID_ARGS naming the provider and path and a hint to zip the .app or pass the .ipa, .apk, or .aab. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
install-from-source sends its positional as a URL source. A local path reached the iOS trusted-host check, where `new URL()` threw a TypeError that surfaced as an untyped UNKNOWN "Invalid URL". The CLI now refuses a positional that is not an http(s) URL with INVALID_ARGS and points at `install <app> <path>` for local builds, and the trusted-host check reports an unparsable source as INVALID_ARGS instead of throwing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-ups to the TestMu AI provider from review on callstack#3118, plus a lease timing fix found by running it on TestMu AI production.
fix(daemon): start a lease's TTL when its allocation completes. The lease used to be stamped before the provider allocated, and a hosted allocation takes 30–80 s. So a default 60 s lease was already expired whenleases.allocatereturned, and the next command failed withLease is not active. Allocation now holds a work pass, so the expiry sweeper skips the lease while it is allocating, and the TTL starts once allocation finishes.refactor(provider-webdriver): declare the profile fields each provider consumes.CloudProviderProfileFieldskey, whether they consume or refuse it. The declaration is exhaustive at compile time.leases.allocateand remote-config profiles.rejectTestMuOnlyProviderFlags,rejectUnsupportedTestMuDeviceFeatures,rejectBrowserStackOnlyDeviceFeatures, the hand-copied unsupported-flag table and the./testmu-device-featuressubpath.leases.allocate({ leaseProvider: 'limrun', providerDeviceType: 'real' })is nowINVALID_ARGS.--provider-app,--provider-os-version,--provider-projectand--provider-build. It never read them.fix(provider-webdriver): let the Apple materializer name the uploadable file.MaterializedAppSource.uploadPathreplaces the extension guess, so a zip that wraps an.ipauploads the.ipa.fix(provider-webdriver): tighten app-reference, endpoint and upload input handling.lt://andbs://are validated, matched case-insensitively and canonicalized. An empty id is refused.INVALID_ARGS, and a non-JSON 2xx answer is a typedCOMMAND_FAILED.test(provider-webdriver): deadline and abort tests now act while the request is in flight. The connect test helper is shared between suites.fix(provider-webdriver): refuse profile fields on a repeat allocation too.profileFieldsis required and comes from the provider definition.fix(provider-webdriver): validate and canonicalize app references at connect. Connect verifieslt://APP1even when the user typedLT://APP1.fix(provider-webdriver): refuse a directory before any hosted upload. BrowserStack could otherwise fail with a rawEISDIR.fix(cli): a non-URLinstall-from-sourcesource is a typedINVALID_ARGSthat points toinstall <app> <path>. It used to fail withUNKNOWN "Invalid URL".Validation
test:unit: 12140 passed, 1 skipped;test:integration:provider: 229 passedbuild,format:check,check:quick,check:layering,check:fallow,check:package: pass12bab7abe: real iPhone 16, iOS 18, signed.ipa:connect --provider-device-type real→open→snapshot -i→close→artifacts --jsonopen32 s; 7 artifacts ready417ff5ba7: Android emulator Galaxy S22 Ultra 5G / 14:open, theninstall-from-source <https .apk URL>→open→snapshot -ideployMaterializedApp; uploaded as anlt://app andInstalled: com.wdiodemoappin 62 s; artifacts readyiOS
install-from-sourceonly accepts GitHub Actions and EAS artifact URLs. That is the existing trusted-host rule inprovision-kit. The CLI also takes no local path there, so the iOS materialized-upload path is covered by unit tests only. BrowserStack was not run live, because no account was available.Two allocation concurrency gaps existed before this change and are left for a follow-up:
🤖 Generated with Claude Code