Skip to content

fix(server): don't 404 non-websocket upgrade probes like h2c - #410

Closed
dennisoelkers wants to merge 4 commits into
CopilotKit:mainfrom
dennisoelkers:fix/ollama-tags-h2c-upgrade-probe
Closed

fix(server): don't 404 non-websocket upgrade probes like h2c#410
dennisoelkers wants to merge 4 commits into
CopilotKit:mainfrom
dennisoelkers:fix/ollama-tags-h2c-upgrade-probe

Conversation

@dennisoelkers

@dennisoelkers dennisoelkers commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Node fires the "upgrade" event (not "request") for any request with a Connection: Upgrade header, regardless of the target protocol. langchain4j speculatively sends Upgrade: h2c on plain requests to probe for HTTP/2 cleartext, which made GET /api/tags and POST /api/chat (for ollama) land in the WebSocket-only upgrade handler and get an unconditional 404.

Only treat the request as a WebSocket upgrade when Upgrade: websocket is actually present; otherwise fall through to the normal HTTP pipeline so routes like /api/tags still work.

Node fires the "upgrade" event (not "request") for any request with a
Connection: Upgrade header, regardless of the target protocol. OkHttp
(used by langchain4j) speculatively sends Upgrade: h2c on plain
requests to probe for HTTP/2 cleartext, which made GET /api/tags land
in the WebSocket-only upgrade handler and get an unconditional 404.

Only treat the request as a WebSocket upgrade when Upgrade: websocket
is actually present; otherwise fall through to the normal HTTP
pipeline so routes like /api/tags still work.
Node fires the "upgrade" event (not "request") for any request with a
Connection: Upgrade header, regardless of the target protocol. OkHttp
(used by langchain4j) speculatively sends Upgrade: h2c on plain
requests to probe for HTTP/2 cleartext, which made GET /api/tags land
in the WebSocket-only upgrade handler and get an unconditional 404.

Only treat the request as a WebSocket upgrade when Upgrade: websocket
is actually present; otherwise fall through to the normal HTTP
pipeline so routes like /api/tags still work.
The earlier h2c-probe fix (rebuilding a ServerResponse via assignSocket)
only worked for bodyless requests like GET /api/tags. Node detaches its
HTTP parser from the socket the moment "upgrade" fires, so any body
bytes land in the `head` buffer instead of `req`'s stream; unshifting
them back onto the socket never reconnects them to `req`, so
readBody(req) resolved empty. A POST like /api/chat then failed JSON
parsing ("Unexpected end of JSON input") instead of surfacing the real
validation error for the request that was actually sent.

Rebuild the request line and headers with Upgrade/Connection dropped,
replay them plus `head` on the socket, and let Node re-parse the
connection from scratch via server.emit("connection", socket). This
reuses Node's own parser for the body (Content-Length or chunked)
instead of hand-rolling body buffering.
…-upgrade-probe

# Conflicts:
#	src/__tests__/ollama.test.ts
#	src/server.ts
@dennisoelkers
dennisoelkers marked this pull request as draft September 3, 2026 12:50
@dennisoelkers
dennisoelkers marked this pull request as ready for review September 3, 2026 12:53
@dennisoelkers

Copy link
Copy Markdown
Contributor Author

@jpr5: Do you think you will have some time too look at this? Fixing this is a prerequisite for us to be able to use it in our project :/

@pkg-pr-new

pkg-pr-new Bot commented Sep 8, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@410

commit: c597453

@jpr5

jpr5 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Thanks for this — the diagnosis is right, the write-up is clear, and the raw-socket regression tests are the correct way to exercise a shape http.request() can't produce. GET /api/tags is genuinely fixed, and I verified the guard doesn't break real upgrades: Upgrade: websocket still gets 101, including the mixed-case Upgrade: WebSocket form (your .toLowerCase() handles the RFC's case-insensitive token correctly), keep-alive reuse after a replay works — two requests on one socket, two 200s — and there's no listener growth on the server across 25 probes.

I'm holding the merge on one finding.

Any h2c-probed request with a body hangs forever on Node 26

CI is green because the matrix is [20, 22, 24]. package.json declares engines: {"node": ">=20.15.0"}, which is open-ended, so Node 26 is a version this package claims to support — and it's what a contributor on a current Homebrew/nvm install gets today.

Dependency-free reproduction of just the replay technique (no aimock, no vitest — http.createServer plus your upgrade handler verbatim), run across the matrix and one version past it:

### node:20   GET 200 | POST same-write 200 | POST delayed 200 | POST 200KB 200
### node:22   GET 200 | POST same-write 200 | POST delayed 200 | POST 200KB 200
### node:24   GET 200 | POST same-write 200 | POST delayed 200 | POST 200KB 200
### node:26   GET 200 | POST same-write (NO RESPONSE) | POST delayed (NO RESPONSE) | POST 200KB (NO RESPONSE)

Instrumenting the handler shows exactly where it diverges. The rebuilt headers parse fine on both — same method, same URL, correct content-length — but on Node 26 the unshifted body bytes are never delivered to the re-parsed request:

node v24.20.0
    [handler fired] POST /x content-length=7
    [data] 7B
    [end] total=7B
  POST body same write: HTTP/1.1 200 OK

node v26.8.1
    [handler fired] POST /x content-length=7
  POST body same write: (NO RESPONSE)

No data, no end, no error, no aborted. The request never completes, so nothing ever responds and the connection stays open.

This is what makes it worth blocking on rather than filing: it is strictly worse than the current behaviour on that version. On main today, both requests answer immediately —

main: h2c GET  /api/tags: HTTP/1.1 404 Not Found [immediate]
main: h2c POST /api/chat: HTTP/1.1 404 Not Found [immediate]

— so on Node 26 this trades a wrong-but-fast 404 for an indefinite hang. For a mock server that's the bad direction: a 404 fails a test in milliseconds with a legible symptom, while a hang blocks until the caller's own timeout, which is exactly the situation aimock exists to keep people out of. Your own POST /api/chat test reproduces it — it fails locally at the 15s vitest timeout on Node 26.

I tried the obvious reformulations and none of them help on 26: single Buffer.concat([rebuilt, head]) instead of two unshifts, socket.resume() after the emit, and deferring the emit to process.nextTick. So it isn't unshift ordering — something about how a re-emitted connection's parser sources its body changed, and it needs a different approach rather than a tweak.

What would unblock it

Whichever you prefer:

  1. Scope the replay to bodyless requests. A request with no content-length and no transfer-encoding replays correctly on every version tested, including 26 — that covers GET /api/tags, the case in your report. Anything carrying a body keeps today's behaviour, so no hang is introduced anywhere.
  2. Keep the body path but make failure loud. If the replay can't be made to work on 26, detect it and answer — even a 501/505 beats holding the socket open.
  3. Add the current Node to the CI matrix either way. Node 26 is inside the declared engines range and CI can't see it, which is why this passed. Happy to do that as a separate change if you'd rather not widen this PR.

Option 1 plus a Node 26 matrix entry is the smallest thing that ships the fix you reported without regressing anyone, if you want the narrow path.

Full repro script and per-variant results available if useful — say the word and I'll paste it or push it as a test.

@jpr5

jpr5 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Solved — and it turned out not to be fixable by adjusting the replay, so I've opened #418, which carries your four commits with authorship intact and adds the missing piece on top. Credit is yours; closing this in favour of that one.

The root cause, for the record: the bytes were never in head on Node 26.

node 24   cl=7  head=7B  socketAfter=0B
node 26   cl=7  head=0B  socketAfter=0B

Node ≥ 26 parses an upgrade request's body onto req and leaves head empty — req yields {"a":1} there, while on Node ≤ 24 it yields nothing and head carries the body. So a head-only replay was forwarding an empty body and the re-parsed request sat waiting on a Content-Length that would never be satisfied. Draining req and concatenating it into the replay covers both: the stream has already ended in either case, so it never waits on the network, and on Node ≤ 24 it contributes zero bytes and your original path is unchanged. All four probe shapes now pass on 20, 22, 24 and 26.

For anyone who finds this later, the reformulations that looked promising and did nothing: single Buffer.concat([rebuilt, head]) instead of two unshifts, socket.resume() after the emit, deferring the emit to process.nextTick, and bridging through a fresh Duplex instead of re-emitting the socket. None of them helped, because the problem was never the socket handling — your replay mechanics were correct all along.

#418 also adds Node 26 to the unit and pytest CI matrices. It's inside the declared engines range (>=20.15.0) and wasn't being tested, which is the only reason this got as far as a green CI run.

Thanks for the report and the fix — the raw-socket regression tests in particular were the right call, and they're what made the remaining gap easy to characterise.

@jpr5 jpr5 closed this Sep 9, 2026
jpr5 added a commit that referenced this pull request Sep 9, 2026
…y on every supported Node (#418)

Carries **@dennisoelkers'** fix from #410 (their four commits,
authorship intact) and adds the piece needed to make it correct on every
Node this package supports. Closes #410.

## Their fix

Node fires `"upgrade"` rather than `"request"` for **any** `Connection:
Upgrade` header, whatever protocol is named. langchain4j's OkHttp client
speculatively sends `Upgrade: h2c` on plain requests to probe for HTTP/2
cleartext, so `GET /api/tags` and `POST /api/chat` landed in the
WebSocket-only handler and took an unconditional `404`. The fix replays
the request with `Upgrade`/`Connection` stripped and lets Node re-parse
it as an ordinary request. Diagnosis and approach are both right.

I verified the parts that worried me, since this is raw socket work:

- real WebSocket upgrades still get `101`, including the mixed-case
`Upgrade: WebSocket` form
- the replayed request still passes the `AIMOCK_API_KEYS` boundary — an
unauthenticated h2c probe gets `401`, not a bypass
- keep-alive reuse of the same socket after a replay works: two
requests, two `200`s
- no listener growth on the server across 25 probes

## What this adds: where the body lives is version-dependent

The original replay forwards `head`. That is only where the body is on
some versions:

- **Node ≤ 24** detaches the parser at `"upgrade"`, so `req` yields
nothing and the body lands in `head` — or arrives on the socket
afterwards if the client sent it later.
- **Node ≥ 26** parses the body onto `req` and leaves `head` **empty**.
The bytes are on neither the socket nor `head`.

So on Node 26 a `head`-only replay left the re-parsed request waiting on
a `Content-Length` worth of bytes that never arrived — no `data`, no
`end`, no error. The connection **hung open instead of answering**,
which is worse than the `404` it replaced: a `404` fails a test in
milliseconds, a hang blocks the caller until its own timeout. `engines`
is `>=20.15.0`, so Node 26 is declared supported.

Measured on a dependency-free reproduction of just the replay technique:

```
node 20/22/24   GET 200 | POST 200 | POST delayed 200 | POST 200KB 200
node 26         GET 200 | POST (NO RESPONSE) | POST delayed (NO RESPONSE) | POST 200KB (NO RESPONSE)
```

Instrumenting it showed the headers parse fine on 26 — right method,
right URL, correct `content-length` — and pinpointed where the bytes
went:

```
node 24   cl=7  head=7B  socketAfter=0B
node 26   cl=7  head=0B  socketAfter=0B     ← and req yields "{"a":1}"
```

**The fix:** drain `req` and concat it into the replay. The stream has
already ended in both cases, so it never waits on the network — on Node
≤ 24 it yields zero bytes and `head` carries the body exactly as before.
All four probe shapes now pass on 20, 22, 24 and 26.

Worth noting the earlier reformulations that did *not* help, since they
look plausible: a single `Buffer.concat([rebuilt, head])` instead of two
`unshift`s, `socket.resume()` after the emit, deferring the emit to
`process.nextTick`, and bridging through a fresh `Duplex` rather than
re-emitting the socket. None move the needle, because the problem was
never the socket — the bytes simply weren't in `head`.

## Tests

Two added beside the original. The existing test asserts the body was
**parsed** — but a validation error only needs enough of the body to see
a field is missing, so it would still pass on a **truncated** replay.
The new ones assert the body arrived **intact** (matching a fixture on
its content, which truncation cannot fake) and that a 200KB body
survives, which no single read can carry.

Mutation-tested: dropping `parsedBody` from the concat reds 3 of the 95
tests in the file; restored, 95 pass.

## CI matrix

Adds Node 26 to the unit and pytest matrices. It sits inside the
declared `engines` range and was untested — which is exactly why the
body-loss case passed CI on the original PR.

## Verification

`typecheck` (all three configs) exit 0 · full suite **183 files / 5694
tests** · eslint and prettier clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Y445N6QQBeAdiLcEvEpqGe
@jpr5 jpr5 mentioned this pull request Sep 9, 2026
jpr5 added a commit that referenced this pull request Sep 9, 2026
Cuts the accumulated `[Unreleased]` work. **72 non-merge commits since
v1.39.0, 42 of them `feat:`/`fix:`** — minor, not patch: two additive
features, no breaking changes.

**Prepared, not merged.** `publish-release.yml` fires on push-to-main,
so merging this publishes to npm, tags, cuts the GitHub Release,
dispatches the Docker build, and posts to `#oss-alerts`. Merge when you
want it live.

## What ships

**Added** — `GET /__aimock/fixtures` fixture-count introspection (#407)
· recorded OpenAI/OpenRouter token usage incl. OpenRouter `usage.cost`
(#368) · AG-UI subagent lifecycle events + `subagentRunId` attribution
(#391)

**Fixed** — `X-AIMock-Strict` parsed case-insensitively with whitespace
tolerated (#408) · non-websocket upgrade probes no longer 404 and the
body survives on every supported Node (#410#418) · AG-UI drift
collector no longer reports clean for an unreadable failure · AG-UI
canonical parser no longer drops a field after a trailing comment ·
`openrouter` no longer logged as an unknown SSE provider when recording

**Changed** — the AG-UI drift CI lane runs every `agui-*.drift.ts` guard
rather than one hardcoded file

## Version surfaces — seven

Found by grepping the repo for `1.39.0` rather than working from a
checklist, because the checklist is what went wrong at v1.34.0 (four of
six surfaces shipped stale). **Zero occurrences of the old string
remain:**

| # | file | |
|---|---|---|
| 1 | `package.json` | `version` |
| 2 | `charts/aimock/Chart.yaml` | `appVersion` |
| 3 | `.claude-plugin/plugin.json` | `version` |
| 4 | `.claude-plugin/marketplace.json` | the `^` range under
`plugins[0].source` |
| 5 | `packages/aimock-pytest/.../_version.py` | `AIMOCK_VERSION` — the
npm pin the pytest harness downloads by default |
| 6 | `docs/index.html` | the version badge |
| 7 | `packages/aimock-pytest/README.md` | the documented
`--aimock-version` default |

Two of those (6, 7) aren't in the release SOP's list but carry the
version string and would have shipped stale.

`packages/aimock-pytest/pyproject.toml` stays at **0.5.3** — that's the
Python package's own version, on its own PyPI cadence.

## Two things I checked rather than assumed

**Bumping `_version.py` in the release commit is safe.**
`publish-pytest` runs `npm view @copilotkit/aimock@$AIMOCK_VERSION`,
which would fail against an unpublished version — but it's `needs:
[build, publish]` and gated on `needs.publish.result == 'success'`, so
1.40.0 is on npm by the time it reads the pin. No chicken-and-egg.

**`package.json.description` is deliberately not hand-synced.** It
diverges from the README subtitle in git, which looks like the drift the
SOP warns about. It isn't: `publish-release.yml` rewrites it from the
README subtitle *in the runner* before `npm publish`, without
committing. Editing it here would fight the workflow.

## README

Gains a **Recorded token usage and cost** bullet. The feature list
enumerates record/replay capabilities individually (timing-aware replay,
multi-turn, ordered blocks), so a user-visible one landing without an
entry is a real omission. The new control-API route needs no README line
— the README documents no `/__aimock/*` routes at all; that lives in
`docs/control-api`.

## Verification

`typecheck` (all three configs) exit 0 · `build` exit 0 ·
release/publish-pin/drift-script workflow tests 44 passing · prettier
clean · commit body wrapped ≤100 cols for commitlint.

Full-suite note: five tests failed on a first run and all five passed in
isolation — `cli.test.ts` SIGTERM, `multimedia.test.ts` transcription
frame scheduling, `proxy-buffer-cap.test.ts`, and both
`publish-pin-workflow.test.ts` real-pip cases. All wall-clock or
network-bound, and none reachable from a version-string change. CI is
the arbiter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Y445N6QQBeAdiLcEvEpqGe
pull Bot pushed a commit to TheTechOddBug/aimock that referenced this pull request Sep 9, 2026
Builds on @dennisoelkers' fix (CopilotKit#410), whose diagnosis and approach are right:
Node fires "upgrade" for any `Connection: Upgrade`, so langchain4j's h2c probe
took an unconditional 404, and replaying the request with `Upgrade`/`Connection`
stripped is the fix.

The gap is WHERE THE BODY IS. That differs by Node version, and the original
replay forwarded only `head`:

  - Node <= 24 detaches the parser at "upgrade": `req` yields nothing and the
    body lands in `head` (or arrives on the socket afterwards if sent later).
  - Node >= 26 parses the body onto `req` and leaves `head` EMPTY. The bytes
    are on neither the socket nor `head`.

So on Node 26 a `head`-only replay left the re-parsed request waiting on a
`Content-Length` worth of bytes that never arrived: no `data`, no `end`, no
error — the connection hung open instead of answering. That is worse than the
404 it replaced, because a hang blocks a caller until its own timeout, and
`engines` declares `>=20.15.0` so Node 26 is supported. Measured directly on a
dependency-free reproduction:

    node 20/22/24  GET 200 | POST 200 | POST delayed 200 | POST 200KB 200
    node 26        GET 200 | POST (NO RESPONSE) x3

Drain `req` and concat it into the replay. It is already ended in both cases,
so this never waits on the network: on Node <= 24 it yields zero bytes and
`head` carries the body exactly as before. All four probe shapes now pass on
20, 22, 24 and 26.

Two tests added beside the original. The existing one asserts the body was
PARSED — but a validation error only needs enough of the body to see a field
is missing, so it would still pass on a TRUNCATED replay. The new ones assert
the body arrived INTACT (matching a fixture on its content, which truncation
cannot fake) and that a 200KB body survives, which no single read can carry.

Mutation-tested: dropping `parsedBody` from the concat reds 3 of the 95 tests
in the file; restored, 95 pass.

Also adds Node 26 to the unit and pytest CI matrices. It sits inside the
declared `engines` range and was untested, which is exactly why the body-loss
case passed CI.

typecheck (all three configs) exit 0; full suite 183 files / 5694 tests;
eslint and prettier clean.
pull Bot pushed a commit to TheTechOddBug/aimock that referenced this pull request Sep 9, 2026
Cuts the accumulated Unreleased work: 72 non-merge commits since v1.39.0, 42 of
them feat/fix. Minor, not patch — two additive features, no breaking changes.

### Added

- GET /__aimock/fixtures — read-only fixture-count introspection (CopilotKit#407)
- Recorded OpenAI/OpenRouter token usage, including OpenRouter usage.cost (CopilotKit#368)
- AG-UI subagent lifecycle events + subagentRunId attribution (CopilotKit#391)

### Changed

- The AG-UI drift CI lane runs every agui-*.drift.ts guard, not one file

### Fixed

- X-AIMock-Strict is parsed case-insensitively, whitespace tolerated (CopilotKit#408)
- Non-websocket upgrade probes no longer 404, and the body survives on every
  supported Node (CopilotKit#410, CopilotKit#418)
- The AG-UI drift collector no longer reports clean for an unreadable failure
- The AG-UI drift canonical parser no longer drops a field after a trailing
  comment
- openrouter is no longer logged as an unknown SSE provider when recording

Version surfaces bumped — seven, verified by grepping the repo for the old
string rather than working from a list (zero occurrences remain):
package.json, charts/aimock/Chart.yaml appVersion, .claude-plugin/plugin.json,
.claude-plugin/marketplace.json (the `^` range under plugins[0].source),
packages/aimock-pytest/src/aimock_pytest/_version.py (the npm pin the pytest
harness downloads by default), docs/index.html's version badge, and
packages/aimock-pytest/README.md's documented default.

packages/aimock-pytest/pyproject.toml stays at 0.5.3 — that is the Python
package's own version, released on its own PyPI cadence.

README gains a Recorded-token-usage-and-cost bullet: the feature list
enumerates record/replay capabilities individually, so a user-visible one
landing without an entry there is a real omission. The new control-API route
needs no README line — the README documents no /__aimock/* routes at all; it is
covered in docs/control-api.

package.json's description is deliberately NOT hand-synced: publish-release.yml
rewrites it from the README subtitle in the runner before npm publish, without
committing, which is why git and the npm page differ by design.
@dennisoelkers

Copy link
Copy Markdown
Contributor Author

Hey @jpr5, thanks for taking care of this, the quick review + fix! Much appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants