fix(server): don't 404 non-websocket upgrade probes like h2c - #410
fix(server): don't 404 non-websocket upgrade probes like h2c#410dennisoelkers wants to merge 4 commits into
Conversation
Node fires the "upgrade" event (not "request") for any request with a Connection: Upgrade header, regardless of the target protocol. OkHttp (used by langchain4j) speculatively sends Upgrade: h2c on plain requests to probe for HTTP/2 cleartext, which made GET /api/tags land in the WebSocket-only upgrade handler and get an unconditional 404. Only treat the request as a WebSocket upgrade when Upgrade: websocket is actually present; otherwise fall through to the normal HTTP pipeline so routes like /api/tags still work.
Node fires the "upgrade" event (not "request") for any request with a Connection: Upgrade header, regardless of the target protocol. OkHttp (used by langchain4j) speculatively sends Upgrade: h2c on plain requests to probe for HTTP/2 cleartext, which made GET /api/tags land in the WebSocket-only upgrade handler and get an unconditional 404. Only treat the request as a WebSocket upgrade when Upgrade: websocket is actually present; otherwise fall through to the normal HTTP pipeline so routes like /api/tags still work.
The earlier h2c-probe fix (rebuilding a ServerResponse via assignSocket)
only worked for bodyless requests like GET /api/tags. Node detaches its
HTTP parser from the socket the moment "upgrade" fires, so any body
bytes land in the `head` buffer instead of `req`'s stream; unshifting
them back onto the socket never reconnects them to `req`, so
readBody(req) resolved empty. A POST like /api/chat then failed JSON
parsing ("Unexpected end of JSON input") instead of surfacing the real
validation error for the request that was actually sent.
Rebuild the request line and headers with Upgrade/Connection dropped,
replay them plus `head` on the socket, and let Node re-parse the
connection from scratch via server.emit("connection", socket). This
reuses Node's own parser for the body (Content-Length or chunked)
instead of hand-rolling body buffering.
…-upgrade-probe # Conflicts: # src/__tests__/ollama.test.ts # src/server.ts
|
@jpr5: Do you think you will have some time too look at this? Fixing this is a prerequisite for us to be able to use it in our project :/ |
commit: |
|
Thanks for this — the diagnosis is right, the write-up is clear, and the raw-socket regression tests are the correct way to exercise a shape I'm holding the merge on one finding. Any h2c-probed request with a body hangs forever on Node 26CI is green because the matrix is Dependency-free reproduction of just the replay technique (no aimock, no vitest — Instrumenting the handler shows exactly where it diverges. The rebuilt headers parse fine on both — same method, same URL, correct No This is what makes it worth blocking on rather than filing: it is strictly worse than the current behaviour on that version. On — so on Node 26 this trades a wrong-but-fast I tried the obvious reformulations and none of them help on 26: single What would unblock itWhichever you prefer:
Option 1 plus a Node 26 matrix entry is the smallest thing that ships the fix you reported without regressing anyone, if you want the narrow path. Full repro script and per-variant results available if useful — say the word and I'll paste it or push it as a test. |
|
Solved — and it turned out not to be fixable by adjusting the replay, so I've opened #418, which carries your four commits with authorship intact and adds the missing piece on top. Credit is yours; closing this in favour of that one. The root cause, for the record: the bytes were never in Node ≥ 26 parses an upgrade request's body onto For anyone who finds this later, the reformulations that looked promising and did nothing: single #418 also adds Node 26 to the unit and pytest CI matrices. It's inside the declared Thanks for the report and the fix — the raw-socket regression tests in particular were the right call, and they're what made the remaining gap easy to characterise. |
…y on every supported Node (#418) Carries **@dennisoelkers'** fix from #410 (their four commits, authorship intact) and adds the piece needed to make it correct on every Node this package supports. Closes #410. ## Their fix Node fires `"upgrade"` rather than `"request"` for **any** `Connection: Upgrade` header, whatever protocol is named. langchain4j's OkHttp client speculatively sends `Upgrade: h2c` on plain requests to probe for HTTP/2 cleartext, so `GET /api/tags` and `POST /api/chat` landed in the WebSocket-only handler and took an unconditional `404`. The fix replays the request with `Upgrade`/`Connection` stripped and lets Node re-parse it as an ordinary request. Diagnosis and approach are both right. I verified the parts that worried me, since this is raw socket work: - real WebSocket upgrades still get `101`, including the mixed-case `Upgrade: WebSocket` form - the replayed request still passes the `AIMOCK_API_KEYS` boundary — an unauthenticated h2c probe gets `401`, not a bypass - keep-alive reuse of the same socket after a replay works: two requests, two `200`s - no listener growth on the server across 25 probes ## What this adds: where the body lives is version-dependent The original replay forwards `head`. That is only where the body is on some versions: - **Node ≤ 24** detaches the parser at `"upgrade"`, so `req` yields nothing and the body lands in `head` — or arrives on the socket afterwards if the client sent it later. - **Node ≥ 26** parses the body onto `req` and leaves `head` **empty**. The bytes are on neither the socket nor `head`. So on Node 26 a `head`-only replay left the re-parsed request waiting on a `Content-Length` worth of bytes that never arrived — no `data`, no `end`, no error. The connection **hung open instead of answering**, which is worse than the `404` it replaced: a `404` fails a test in milliseconds, a hang blocks the caller until its own timeout. `engines` is `>=20.15.0`, so Node 26 is declared supported. Measured on a dependency-free reproduction of just the replay technique: ``` node 20/22/24 GET 200 | POST 200 | POST delayed 200 | POST 200KB 200 node 26 GET 200 | POST (NO RESPONSE) | POST delayed (NO RESPONSE) | POST 200KB (NO RESPONSE) ``` Instrumenting it showed the headers parse fine on 26 — right method, right URL, correct `content-length` — and pinpointed where the bytes went: ``` node 24 cl=7 head=7B socketAfter=0B node 26 cl=7 head=0B socketAfter=0B ← and req yields "{"a":1}" ``` **The fix:** drain `req` and concat it into the replay. The stream has already ended in both cases, so it never waits on the network — on Node ≤ 24 it yields zero bytes and `head` carries the body exactly as before. All four probe shapes now pass on 20, 22, 24 and 26. Worth noting the earlier reformulations that did *not* help, since they look plausible: a single `Buffer.concat([rebuilt, head])` instead of two `unshift`s, `socket.resume()` after the emit, deferring the emit to `process.nextTick`, and bridging through a fresh `Duplex` rather than re-emitting the socket. None move the needle, because the problem was never the socket — the bytes simply weren't in `head`. ## Tests Two added beside the original. The existing test asserts the body was **parsed** — but a validation error only needs enough of the body to see a field is missing, so it would still pass on a **truncated** replay. The new ones assert the body arrived **intact** (matching a fixture on its content, which truncation cannot fake) and that a 200KB body survives, which no single read can carry. Mutation-tested: dropping `parsedBody` from the concat reds 3 of the 95 tests in the file; restored, 95 pass. ## CI matrix Adds Node 26 to the unit and pytest matrices. It sits inside the declared `engines` range and was untested — which is exactly why the body-loss case passed CI on the original PR. ## Verification `typecheck` (all three configs) exit 0 · full suite **183 files / 5694 tests** · eslint and prettier clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Y445N6QQBeAdiLcEvEpqGe
Cuts the accumulated `[Unreleased]` work. **72 non-merge commits since v1.39.0, 42 of them `feat:`/`fix:`** — minor, not patch: two additive features, no breaking changes. **Prepared, not merged.** `publish-release.yml` fires on push-to-main, so merging this publishes to npm, tags, cuts the GitHub Release, dispatches the Docker build, and posts to `#oss-alerts`. Merge when you want it live. ## What ships **Added** — `GET /__aimock/fixtures` fixture-count introspection (#407) · recorded OpenAI/OpenRouter token usage incl. OpenRouter `usage.cost` (#368) · AG-UI subagent lifecycle events + `subagentRunId` attribution (#391) **Fixed** — `X-AIMock-Strict` parsed case-insensitively with whitespace tolerated (#408) · non-websocket upgrade probes no longer 404 and the body survives on every supported Node (#410 → #418) · AG-UI drift collector no longer reports clean for an unreadable failure · AG-UI canonical parser no longer drops a field after a trailing comment · `openrouter` no longer logged as an unknown SSE provider when recording **Changed** — the AG-UI drift CI lane runs every `agui-*.drift.ts` guard rather than one hardcoded file ## Version surfaces — seven Found by grepping the repo for `1.39.0` rather than working from a checklist, because the checklist is what went wrong at v1.34.0 (four of six surfaces shipped stale). **Zero occurrences of the old string remain:** | # | file | | |---|---|---| | 1 | `package.json` | `version` | | 2 | `charts/aimock/Chart.yaml` | `appVersion` | | 3 | `.claude-plugin/plugin.json` | `version` | | 4 | `.claude-plugin/marketplace.json` | the `^` range under `plugins[0].source` | | 5 | `packages/aimock-pytest/.../_version.py` | `AIMOCK_VERSION` — the npm pin the pytest harness downloads by default | | 6 | `docs/index.html` | the version badge | | 7 | `packages/aimock-pytest/README.md` | the documented `--aimock-version` default | Two of those (6, 7) aren't in the release SOP's list but carry the version string and would have shipped stale. `packages/aimock-pytest/pyproject.toml` stays at **0.5.3** — that's the Python package's own version, on its own PyPI cadence. ## Two things I checked rather than assumed **Bumping `_version.py` in the release commit is safe.** `publish-pytest` runs `npm view @copilotkit/aimock@$AIMOCK_VERSION`, which would fail against an unpublished version — but it's `needs: [build, publish]` and gated on `needs.publish.result == 'success'`, so 1.40.0 is on npm by the time it reads the pin. No chicken-and-egg. **`package.json.description` is deliberately not hand-synced.** It diverges from the README subtitle in git, which looks like the drift the SOP warns about. It isn't: `publish-release.yml` rewrites it from the README subtitle *in the runner* before `npm publish`, without committing. Editing it here would fight the workflow. ## README Gains a **Recorded token usage and cost** bullet. The feature list enumerates record/replay capabilities individually (timing-aware replay, multi-turn, ordered blocks), so a user-visible one landing without an entry is a real omission. The new control-API route needs no README line — the README documents no `/__aimock/*` routes at all; that lives in `docs/control-api`. ## Verification `typecheck` (all three configs) exit 0 · `build` exit 0 · release/publish-pin/drift-script workflow tests 44 passing · prettier clean · commit body wrapped ≤100 cols for commitlint. Full-suite note: five tests failed on a first run and all five passed in isolation — `cli.test.ts` SIGTERM, `multimedia.test.ts` transcription frame scheduling, `proxy-buffer-cap.test.ts`, and both `publish-pin-workflow.test.ts` real-pip cases. All wall-clock or network-bound, and none reachable from a version-string change. CI is the arbiter. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Y445N6QQBeAdiLcEvEpqGe
Builds on @dennisoelkers' fix (CopilotKit#410), whose diagnosis and approach are right: Node fires "upgrade" for any `Connection: Upgrade`, so langchain4j's h2c probe took an unconditional 404, and replaying the request with `Upgrade`/`Connection` stripped is the fix. The gap is WHERE THE BODY IS. That differs by Node version, and the original replay forwarded only `head`: - Node <= 24 detaches the parser at "upgrade": `req` yields nothing and the body lands in `head` (or arrives on the socket afterwards if sent later). - Node >= 26 parses the body onto `req` and leaves `head` EMPTY. The bytes are on neither the socket nor `head`. So on Node 26 a `head`-only replay left the re-parsed request waiting on a `Content-Length` worth of bytes that never arrived: no `data`, no `end`, no error — the connection hung open instead of answering. That is worse than the 404 it replaced, because a hang blocks a caller until its own timeout, and `engines` declares `>=20.15.0` so Node 26 is supported. Measured directly on a dependency-free reproduction: node 20/22/24 GET 200 | POST 200 | POST delayed 200 | POST 200KB 200 node 26 GET 200 | POST (NO RESPONSE) x3 Drain `req` and concat it into the replay. It is already ended in both cases, so this never waits on the network: on Node <= 24 it yields zero bytes and `head` carries the body exactly as before. All four probe shapes now pass on 20, 22, 24 and 26. Two tests added beside the original. The existing one asserts the body was PARSED — but a validation error only needs enough of the body to see a field is missing, so it would still pass on a TRUNCATED replay. The new ones assert the body arrived INTACT (matching a fixture on its content, which truncation cannot fake) and that a 200KB body survives, which no single read can carry. Mutation-tested: dropping `parsedBody` from the concat reds 3 of the 95 tests in the file; restored, 95 pass. Also adds Node 26 to the unit and pytest CI matrices. It sits inside the declared `engines` range and was untested, which is exactly why the body-loss case passed CI. typecheck (all three configs) exit 0; full suite 183 files / 5694 tests; eslint and prettier clean.
Cuts the accumulated Unreleased work: 72 non-merge commits since v1.39.0, 42 of them feat/fix. Minor, not patch — two additive features, no breaking changes. ### Added - GET /__aimock/fixtures — read-only fixture-count introspection (CopilotKit#407) - Recorded OpenAI/OpenRouter token usage, including OpenRouter usage.cost (CopilotKit#368) - AG-UI subagent lifecycle events + subagentRunId attribution (CopilotKit#391) ### Changed - The AG-UI drift CI lane runs every agui-*.drift.ts guard, not one file ### Fixed - X-AIMock-Strict is parsed case-insensitively, whitespace tolerated (CopilotKit#408) - Non-websocket upgrade probes no longer 404, and the body survives on every supported Node (CopilotKit#410, CopilotKit#418) - The AG-UI drift collector no longer reports clean for an unreadable failure - The AG-UI drift canonical parser no longer drops a field after a trailing comment - openrouter is no longer logged as an unknown SSE provider when recording Version surfaces bumped — seven, verified by grepping the repo for the old string rather than working from a list (zero occurrences remain): package.json, charts/aimock/Chart.yaml appVersion, .claude-plugin/plugin.json, .claude-plugin/marketplace.json (the `^` range under plugins[0].source), packages/aimock-pytest/src/aimock_pytest/_version.py (the npm pin the pytest harness downloads by default), docs/index.html's version badge, and packages/aimock-pytest/README.md's documented default. packages/aimock-pytest/pyproject.toml stays at 0.5.3 — that is the Python package's own version, released on its own PyPI cadence. README gains a Recorded-token-usage-and-cost bullet: the feature list enumerates record/replay capabilities individually, so a user-visible one landing without an entry there is a real omission. The new control-API route needs no README line — the README documents no /__aimock/* routes at all; it is covered in docs/control-api. package.json's description is deliberately NOT hand-synced: publish-release.yml rewrites it from the README subtitle in the runner before npm publish, without committing, which is why git and the npm page differ by design.
|
Hey @jpr5, thanks for taking care of this, the quick review + fix! Much appreciated! |
Node fires the "upgrade" event (not "request") for any request with a
Connection: Upgradeheader, regardless of the target protocol. langchain4j speculatively sends Upgrade: h2c on plain requests to probe for HTTP/2 cleartext, which madeGET /api/tagsandPOST /api/chat(for ollama) land in the WebSocket-only upgrade handler and get an unconditional 404.Only treat the request as a WebSocket upgrade when
Upgrade: websocketis actually present; otherwise fall through to the normal HTTP pipeline so routes like/api/tagsstill work.