Skip to content

perf(provision-kit): download app sources over Node HTTP instead of Undici - #4958

Open
thymikee wants to merge 6 commits into
0.22from
perf/0.22-node-http-downloader
Open

thymikee wants to merge 6 commits into
0.22from
perf/0.22-node-http-downloader

Conversation

@thymikee

@thymikee thymikee commented Nov 22, 2026 •

Copy link
Copy Markdown
Member

Summary

Use Node HTTP and TLS for app-source downloads in provision-kit. Keep approved-address pinning, URL-host TLS checks, scoped HTTP and HTTPS CONNECT proxies, redirect validation, streaming limits, and cancellation. Limrun’s SDK still bundles Undici on the 0.22 base.

Part of #4674.

Before / after

A reset while streaming previously failed without a network error code. It now returns COMMAND_FAILED with details.networkErrorCode: ECONNRESET. The same field carries system errors while waiting for headers and phase timeouts as connect-timeout, header-timeout, or body-timeout. Caller cancellation has its own outcome. Failed downloads remove the partial file.

Matched pnpm size against dcbf9d8bfa: 206 bytes smaller compressed and 1,931 bytes larger unpacked and installed. The Undici code moves to Limrun’s tunnel chunk. The larger reduction depends on #4846.

Validation

At ba68e9445d, 79 focused tests passed on Node 22.22.2. Packed smoke verified pinned CONNECT, artifact cleanup, and truncated-body errors. pnpm check:affected --run --base origin/0.22 --head HEAD passed all locally runnable gates. GitHub runs the hosted-only checks.


Get evidence
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

View guided diff Turn on auto-fix

@github-actions

github-actions Bot commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.07 MB 6.15 MB +75.3 kB
Package (unpacked) 6.07 MB 6.15 MB +75.3 kB
Package (download) 1.86 MB 1.87 MB +17.8 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 22.9 ms 22.8 ms -0.1 ms
CLI --help 49.4 ms 47.2 ms -2.2 ms

@apex-sentinel-staging

This comment has been minimized.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on ba68e94. The verdict and the full list are in the summary comment.

@apex-sentinel-staging

This comment has been minimized.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on a4a0f5c. The verdict and the full list are in the summary comment.

@thymikee

Copy link
Copy Markdown
Member Author

The PR is ready at a4a0f5c: the code review found nothing that must change. Hosted CI is still running at this commit, so the checks must go green before merge. No conflicts.

Not blocking, and each can be taken or left: (1) the phase-timeout details.networkErrorCode values (connect-timeout, header-timeout, body-timeout) in install-source-download.ts duplicate details.transferPhase and add an error contract to a transport swap that #4674 asks to keep behavior-neutral, so dropping them and the matching docs sentence would keep it neutral; (2) the transport catch at line 131 can be the base rule, rethrow any AppError and wrap the rest, because requestApprovedUrl throws only plain Errors; (3) the CI Size Report at ba68e94 shows no core saving (download +66 B), so the PR body should quote those numbers and the saving can be measured once #4846 takes Limrun out of core; (4) the TransportSubject indirection and the separate proxy-auth lab in the new fixtures could call requestApprovedUrl directly and share the loopback lab; (5) the comment at line 16 about the CLI eager import closure looks stale, since no provision-kit file is in that closure.

The earlier inline thread on the DNS error handling no longer applies at a4a0f5c: approval now runs in its own try and rethrows the refusal unchanged, and a new test covers ENOTFOUND, so it can be resolved: #4958 (comment)

Not verified here: any test run, the packed bundle contents, the Node CONNECT and TLS tunnel behavior, and a real-internet or corporate-proxy run, since the author smoke test is loopback.

@apex-sentinel-staging

This comment has been minimized.

@thymikee

Copy link
Copy Markdown
Member Author

The DNS approval error and the typed cancellation findings are fixed at 22cff6b, but the new transport drops ALPN negotiation, so some HTTPS downloads that work today will fail.

The origin tls.connect options at https://github.com/callstack/agent-device/blob/22cff6b/packages/provision-kit/src/install-source-network-transport.ts#L35 and the HTTPS CONNECT request options near line 147 do not set ALPNProtocols. The base Undici connector offers http/1.1 on both TLS legs. In a loopback comparison of the two transports on Node 26.11.1, the base negotiated http/1.1 and got 200 both directly and through HTTP CONNECT. Head negotiated no protocol and got 421 on both routes against the same origin. An HTTPS proxy that requires ALPN gave 200 on the base and EPROXYTUNNEL on head. So an origin or proxy that needs negotiated HTTP/1.1 now refuses a transfer that used to work. The rule is that every TLS leg the downloader opens must offer http/1.1, and connectThroughProxy plus the origin connection are the only production sites. Please set ALPNProtocols: ['http/1.1'] on both, and assert the negotiated protocol on both legs with the existing TLS loopback fixtures.

The PR text says the Node HTTP downloader preserves the base branch's TLS negotiation behavior for HTTPS origins and HTTPS CONNECT proxies. That does not hold at head, as shown above. The ALPN fix makes the claim true, so no wording change is needed once it lands.

Could install-source-network-transport.ts own network failure classification for the response body? Then install-source-download.ts could keep file and byte-limit errors without the failureSource listeners that infer where an error came from after pipeline teardown starts.

Both earlier inline threads on DNS approval errors (#4958 (comment)) and on typed cancellation (#4958 (comment)) no longer apply. Approval now rethrows the refusal unchanged, and isSignalFailure excludes AppError with a regression test, so both can be resolved.

All 21 checks pass at 22cff6b. Not verified here: the new cancellation test failing without the fix (this rests on reading the old predicate and lookupWithSignal), the size numbers in the PR body, and behavior against a real proxy or the public internet, since the smoke run is loopback only. Once the ALPN fix lands with the loopback assertions, nothing else is needed before merge.

The Node HTTP transport handed `tls.connect` and the HTTPS CONNECT request no
`ALPNProtocols`, while the base Undici connector offered `http/1.1`. An origin or
proxy that requires a negotiated protocol now refuses a transfer that used to
work. Offer `http/1.1` on the origin connection and on the proxy CONNECT request.

The loopback TLS origin and the TLS CONNECT proxy advertise `http/1.1`, so each
handshake reports what the transport offered: the origin echoes the negotiated
protocol, and the proxy records it at `secureConnection`. Dropping either
production option fails that leg's assertion.
@thymikee

Copy link
Copy Markdown
Member Author

ALPN is back on both TLS legs at 39fc971.

Production

Both use one module constant, 'http/1.1', since HTTP/1.1 is the only application protocol this transport speaks.

Assertions on both legs

The two TLS servers in the existing loopback fixtures now advertise http/1.1, so each handshake reports what the transport offered rather than reporting false for every client:

Each assertion fails without its own fix

Measured in a disposable checkout at 22cff6b with pnpm exec vitest run --project unit-core over the two loopback files (20 tests):

Variant Result
Head tests, both options reverted (your finding) 4 failed, 16 passed — alpnHeader: 'false' on the origin leg, proxyAlpnSince [undefined] on the proxy leg
Origin option dropped, proxy option kept 3 failed, 17 passed — origin leg only
Proxy option dropped, origin option kept 2 failed, 18 passed — proxy leg only, undefined vs 'http/1.1'
Both options present 20 passed

So the two legs are detected independently: no single assertion covers both, and no proxy-leg assertion can be satisfied by the origin option.

Checks: pnpm exec vitest run --project unit-core packages/provision-kit 107 passed, plus pnpm lint, pnpm format and pnpm typecheck clean. No changeset.

On the failureSource question: agreed the listener wiring is worth removing, and your comment says nothing else is needed before merge once ALPN lands, so I left it out of this PR. Out of scope here rather than rejected.

@apex-sentinel-staging

This comment has been minimized.

@thymikee

Copy link
Copy Markdown
Member Author

The ALPN problem from the earlier review (#4958 (comment)) is fixed on both TLS legs at 39fc971, with independent loopback assertions. One defect remains: socks5:// proxies no longer work.

An HTTP_PROXY URL with socks5:// passes the unchanged resolveProxyForUrl selector and then hits the new HTTP/HTTPS-only protocol guard at install-source-network-transport.ts#L136. It throws Unsupported install source proxy protocol before it contacts the proxy. Both revisions were run against the same local SOCKS5 proxy and origin, using the Undici 7.30.1 tarball checked against the merge-base lockfile hash. At dcbf9d8 the transport requested a SOCKS tunnel to the approved IP and returned HTTP 200 with the artifact bytes. At 39fc971 it threw, and the proxy saw no tunnel request. The locked ProxyAgent handles both socks5: and socks:. Install-source downloads that relied on a SOCKS proxy now fail even when the origin is reachable and the proxy allows the approved destination. Please keep the SOCKS tunnel path in connectThroughProxy with a maintained SOCKS connector that dials the approved address and observes the caller signal. Keep the Node HTTP and origin TLS paths shared, and add a loopback test that pins the previously successful SOCKS route.

Could the existing loopback fixture own proxy-listener setup, socket tracking and settling for both labs, with the auth suite supplying only its CONNECT handler and TLS handshake observations? startLoopbackLab and startProxyAuthLab both manage these lifecycle steps locally.

The two earlier inline threads on approval handling (#4958 (comment)) and on signal-failure rewriting (#4958 (comment)) are fixed at 39fc971, so they can be resolved.

All 21 checks pass at 39fc971, and there are no conflicts. Not verified here: the author's 4/3/2 revert matrix and any test run. That each ALPN assertion fails without its own option rests on reading the code and on Node's documented behavior. Also not verified here: any run against a real ALPN-enforcing origin or corporate HTTPS proxy, since all evidence is loopback. The SOCKS defect must be fixed before merge. The core saving that #4674 asks for still depends on #4846, as noted in the first round.

…transport

#4674 replaced the Undici-backed install-source transport with a Node HTTP
client that only built HTTP(S) CONNECT tunnels, so a `socks5://` or `socks://`
proxy URL began failing with "Unsupported install source proxy protocol".

Route both SOCKS schemes through the maintained `socks` client and hand it the
approved address as its destination, so the tunnel is built for the address the
resolver already vetted and the URL host is never resolved again. The proxy
credentials come from the proxy URL's userinfo, matching what the previous
`ProxyAgent` accepted. The transport owns the dial to the proxy, so a proxy that
is down fails with the socket's own errno code, and aborting the caller signal
destroys the socket mid-handshake and rejects with the caller's reason.
`socks` is imported on this route only and bundled, so a download that names no
SOCKS proxy never evaluates it.

Add a loopback SOCKS5 lab that speaks RFC 1928 plus the RFC 1929 method and
records the destination each tunnel request names, and cases that pin the
approved-address destination, the artifact bytes, credential forwarding, a
refused tunnel, and an abort mid-handshake. Fold the duplicated origin, socket
tracking and settling of the loopback and proxy-auth labs into a shared base.
@thymikee

Copy link
Copy Markdown
Member Author

SOCKS is restored at 0296f04, and the two labs now share one lifecycle owner.

The SOCKS tunnel route

connectThroughProxy now dispatches by scheme instead of reaching the HTTP/HTTPS-only guard: install-source-network-transport.ts#L111 routes socks: and socks5: (#L105, the same pair the locked ProxyAgent accepts) to connectThroughSocks5Proxy (#L131), and everything else to the unchanged CONNECT body, now connectThroughHttpProxy (#L188). The connector is the maintained socks package (SocksClient.createConnection).

  • Approved address, no re-resolution. The transport dials net.connect to the proxy and hands the connector destination: parseSocksDestination(authority) (#L176), where authority is the same approved host:port the CONNECT path builds. The connector never receives the URL host, so it cannot resolve it. The SOCKS5 host type follows from that address: an IPv4 approved address goes out as 0x01, not 0x03.
  • Credentials. socksProxyCredentials (#L167) passes the proxy URL's percent-decoded userinfo as the RFC 1929 username/password method. SOCKS has no Proxy-Authorization header, so a socks5://user:pass@ URL that previously worked through ProxyAgent authenticates the same way it did before.
  • Signal and timeout. The transport awaits the dial itself (#L143), so a proxy that is down surfaces the socket's own errno (ECONNREFUSED) rather than the connector's text-only wrapper. The abort listener destroys the socket mid-handshake and signal.throwIfAborted() in the catch (#L160) re-raises the caller's reason, so a cancellation stays a cancellation. The connector's internal timer is unref'd, so the caller signal and the existing connect timeout remain the only deadlines.
  • Shared paths. The Node HTTP request, origin TLS, redirect, header and body paths are the same code the HTTP/HTTPS proxy routes use — SOCKS changes only how the tunnel socket is obtained. resolveProxyForUrl, NO_PROXY and the scheme/port scoping rules are untouched.
  • Load cost. socks is imported inside the SOCKS route only, and is bundled in tsdown.config.ts with smart-buffer and ip-address, so a download that names no SOCKS proxy never evaluates it.

Loopback tests — install-source-network-transport-socks.test.ts

The lab's proxies speak RFC 1928 plus the RFC 1929 method over raw sockets (install-source-network-transport-socks.fixtures.ts#L245) and record the destination bytes each tunnel request names, which is the assertion the previous route failed:

  • a socks5 proxy tunnels… and a socks proxy tunnels… (#L65) pin the previously successful route: the proxy is asked for 127.0.0.1:<port>, never approved.invalid:<port>, and the response is HTTP 200 with ARTIFACT_BODY, served with the URL host as Host.
  • credential forwarding (#L86) — the proxy grants the tunnel only after the username/password subnegotiation, and the case asserts the observed method.
  • refused tunnel request (#L106), refused proxy dial (#L126) asserting ECONNREFUSED.
  • abort mid-handshake (#L142) — the proxy takes the greeting and answers nothing; the rejection carries the caller's operator stop, after the proxy is confirmed to have received the greeting.
  • TLS origin through the tunnel (#L159) — child spawned with the fixture CA, asserting the URL host stays the SNI and http/1.1 is negotiated on the origin leg.

All seven SOCKS cases fail without the fix, for the intended reason: with the SOCKS dispatch removed, the two tunnel cases, the credential case and the abort case reject with Unsupported install source proxy protocol, the refused-tunnel case fails the the proxy must have received the SOCKS greeting assertion, the refused-dial case reports that message instead of ECONNREFUSED, and the TLS-through-tunnel case reports two rejections instead of two 200s — matching the proxy-saw-nothing evidence in the review. With ...socksProxyCredentials(proxy) removed alone, only the credential case fails.

Shared fixture lifecycle

startLoopbackLabBase (install-source-network-transport.fixtures.ts#L189) now owns the artifact origin, TLS origin, proxy-listener binding, socket census, per-case markers, slicing, settling and close. startLoopbackLab (#L276), startProxyAuthLab (install-source-network-transport-proxy.fixtures.ts#L100) and startSocksLab (#L80) supply only their proxy listeners and observations; the auth lab keeps just its CONNECT credential check and TLS handshake records, and its local settleSince / liveProxySocketsSince / close duplicates are gone (net −115 lines there). LabMark.proxySockets is one census per listener so labs with different proxy counts share it.

Size

The bundled core grows by one lazily-loaded chunk: dist/src 580 → 581 files, 4,593,501 → 4,663,347 raw (+69,846), 1,594,240 → 1,610,942 gzip (+16,702), against 39fc971 measured with the same pnpm size inputs. The chunk is imported only on the SOCKS route, so startup and plain-HTTP downloads are unchanged.

Verification

pnpm check:affected --run --base origin/0.22 --head HEAD passes on 0296f04 (all runnable checks, exact-commit receipt). Build, lint, format and typecheck pass; the 114 provision-kit tests pass in-process. Not verified here: a real corporate SOCKS or HTTPS proxy, or a non-loopback SOCKS service — every SOCKS assertion above is loopback, as is the existing evidence for the other routes.

The two inline threads on approval handling and signal-failure rewriting can be resolved.

@apex-sentinel-staging

This comment has been minimized.

@thymikee

Copy link
Copy Markdown
Member Author

At 0296f04, the two earlier findings from the first review (39fc971) are fixed, but one new high-priority defect remains on the SOCKS route. All 21 checks pass at 0296f04, and there are no conflicts.

connectThroughSocks5Proxy registers onAbort = () => socket.destroy() and then awaits once(socket, 'connect'). Take a SOCKS proxy whose TCP dial never completes, such as a blackholed host or a dropped SYN. The connect-phase watchdog or an operator cancel aborts the signal. socket.destroy() without an error emits only close, and events.once rejects only on error, so the promise never settles. requestHop in install-source-download.ts has no race against the signal, so nothing else ends the wait. The HTTP proxy path does not hang, because ClientRequest.destroy() emits error. The result is that install-from-source through an unreachable socks5:// or socks:// proxy hangs forever. It never reports connect-timeout, and an operator cancel does not end it, so the daemon command never returns.

The rule is that every await on the SOCKS route must settle when the signal aborts. The smallest change is await once(socket, 'connect', { signal }), or make onAbort call socket.destroy(signalError) so that error fires and the existing catch runs signal.throwIfAborted(). Please add a test with a dial that never connects, for example an unroutable address under a short operatorAbortAfter, and assert that it rejects with the caller error.

The two open inline threads (#4958 (comment) on approval refusals, and #4958 (comment) on signal failures) no longer apply at this head, so they can be resolved.

Not verified here: the PR test suites were not run, and green CI is the only evidence for them. The hang was reproduced with a standalone Node script that has the same once/destroy shape, not by running the PR code. A base comparison against an unreachable SOCKS proxy was not run, so the claim that the old Undici route ended on abort rests on its dispatcher abort wiring. How the 30 s SocksClient handshake timeout interacts with a longer connect phase was not checked. Production size is +1826/-124, above the 700-line threshold, mostly test fixtures and transport, and the measured install-size saving still depends on #4846 removing the remaining Undici users.

Before merge, the SOCKS abort hang must be fixed and covered by the new test.

The SOCKS route awaited `once(socket, 'connect')` without the caller signal, so
a proxy whose TCP dial never completes left the connect-phase wait unsettled on
abort: `socket.destroy()` without an error emits `close`, and `once` rejects only
on `error`. An install-from-source through an unreachable `socks5://` or
`socks://` proxy hung forever, so the connect-phase watchdog and an operator
cancel never ended the command.

Hand the signal to the dial's own `once`, so that await settles with the
caller's reason the way the request and body awaits do, and keep a no-op error
listener for the abandoned socket: `once` drops its own error listener as soon
as the abort wins the race, and a late socket error after that would be
unhandled.

Cover the dial with a case that dials a blackholed address under a short
operator cancel and asserts the caller's error and a destroyed socket. Without
the signal the case reports the harness's `hung` outcome instead of a rejection.
@thymikee

Copy link
Copy Markdown
Member Author

Fixed in 45dc0e1. The dial's own await now carries the caller signal, so every await on the SOCKS route settles on abort.

  • install-source-network-transport.ts:149 — await once(socket, 'connect', { signal }). onAbort still destroys the socket, and the existing catch awaits closeSocket and then signal.throwIfAborted(), so the rejection is the caller's own reason: once rejects with signal.reason, and a signal aborted before the await rejects as AbortError, which throwIfAborted() replaces with the caller's error either way. No change was needed in install-source-download.ts; the transport now settles the way the other two routes do, so requestHop keeps mapping it to the typed cancellation.
  • install-source-network-transport.ts:143 — the dial keeps one no-op error listener. When the abort wins the race, once removes its own error listener, so a late socket error on the abandoned, already-destroyed socket would otherwise be an unhandled error. I could not make a destroyed dial emit one (200-round stress over both a blackholed dial and a refused dial whose connect error was dispatched in the same tick as the abort: zero uncaught), so this is a listener-count guard, not a reproduced crash. requestApprovedUrl already keeps one on its socket for the same reason.

Test: install-source-network-transport-socks.test.ts:144 dials socks5://10.255.255.1:1080, private space no host on the network answers, under operatorAbortAfter(150). It asserts the caller error, that no SOCKS greeting or tunnel request left the process, and that the dial's TCPSocketWrap/ConnectWrap handles settle back to the baseline, i.e. the socket was destroyed. The case is bounded by observeRequest's own case deadline, so it cannot outwait Vitest.

Proof it catches the defect: with await once(socket, 'connect') restored and the test unchanged, the case fails on its first assertion with outcome: 'hung' instead of 'rejection' (reproduced locally, file ran in 3.6 s). With the fix, pnpm exec vitest run --project unit-core packages/provision-kit is 115 passed across 10 files; pnpm lint, pnpm format:check and a forced tsc -b tsconfig.typecheck.json are clean.

The two open inline threads are resolved. One answer to your "not verified" list: the new case does not depend on the 30 s SocksClient handshake timeout — the abort lands in the dial phase, before SocksClient.createConnection is called, so that timer never starts here.

@apex-sentinel-staging

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve · review of 45dc0e1

Since the review of 0296f04: 0 new · 0 still open · 0 no longer raised

No open findings.

Not in the reviewers' prompt (1 file)
  • pnpm-lock.yaml: generated or a lockfile, so its lines were left out

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (e10ad0cc2fb8).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant