Skip to content

feat(v2): stream multipart uploads instead of buffering them - #1332

Merged
yagop merged 11 commits into
masterfrom
feat/streaming-uploads
Jul 10, 2026
Merged

feat(v2): stream multipart uploads instead of buffering them#1332
yagop merged 11 commits into
masterfrom
feat/streaming-uploads

Conversation

@yagop

@yagop yagop commented Jul 10, 2026

Copy link
Copy Markdown
Owner
  • npm run check is clean (typecheck src + test + examples, lint:core, check:edge, unit suite: 130 pass on Node; on Bun 129 pass + 1 skip proxied, 130 pass direct)
  • npm run build is clean
  • Generated files regenerated (doc/api.md); generated API sources untouched

Description

Uploads used to hold 2-4x the file size in RAM: fromPath read the whole file, encodeForm copied every part into FormData Blobs, and a ReadableStream InputFile was drained into an in-memory Blob before the send. This PR replaces the FormData encoder with a hand-rolled multipart/form-data body built as a web ReadableStream and passed straight to fetch (duplex: "half"), so file bytes flow from their source without ever being materialized. Measured on a 100MB upload: ~1MB retained vs 200-300MB before.

  • src/core/multipart.ts - the streaming encoder. The body is laid out once as ordered pieces (encoded part headers + each file's Blob/Uint8Array/stream/factory); each send attempt streams the pieces into a fresh ReadableStream. Web-standard APIs only; lint:core + check:edge stay green.
  • Retry semantics are explicit: Blob/Uint8Array uploads re-stream on retry (byte-identical); a bare ReadableStream is one-shot - sent once, a failure surfaces immediately; the new InputFileStreamFactory (() => ReadableStream | Promise<ReadableStream>) opens a fresh stream per attempt and stays retryable.
  • fromPath() returns a disk-backed Blob via fs.openAsBlob (Node >= 19.8, feature-detected; older Nodes fall back to readFile), so disk uploads are flat-memory and retry-safe.
  • Runtimes that cannot stream a request body fall back to sending the same bytes as one buffered, replayable Blob, chosen by a once-per-process probe. Bun is excluded from streaming only when an HTTP(S) proxy is configured (read via Bun.env, which Bun honors automatically): its fetch streams request bodies fine on a direct connection, but a stream body routed through a CONNECT proxy stalls forever (fetch() with a ReadableStream request body stalls when routed through an HTTP(S) CONNECT proxy (direct connections work) oven-sh/bun#33918; Node and curl pass through the same proxy, so no local probe can catch it). Direct-connection Bun streams like Node/Deno; drop the guard when the upstream issue is fixed.
  • meta.contentType is guarded against CR/LF header injection into part headers; part names/filenames use the WHATWG escaping.
  • inputFileToBlob is removed - nothing converts to FormData anymore.
  • Default per-request timeoutMs raised 30s -> 5 min so large uploads are not cut off mid-stream (getUpdates already derives its own long-poll timeout).

Validation

  • Unit suites on Bun and Node, plus the full local gate.
  • Live against api.telegram.org on Node: plain-bytes, fromPath, one-shot stream, stream factory, and nested attach:// media-group uploads (chunked transfer-encoding accepted; confirmed independently with curl).
  • Live against api.telegram.org on Bun via the buffered fallback, plus the scoped e2e upload methods.
  • Synthetic 100MB upload memory measurement before/after (200-300MB retained -> ~1MB), including a full replay/retry re-send.

Supersedes #1331 (same goal, independently implemented): adopts its InputFileStreamFactory and content-type hardening, and additionally removes the FormData double-copy for Blob/Uint8Array uploads and adds the no-stream fallback that keeps proxied Bun working (streamed uploads there stall through CONNECT proxies, which local-HTTP validation does not catch).

🤖 Generated with Claude Code

yagop and others added 3 commits July 10, 2026 11:21
Uploads used to hold 2-4x the file size in RAM: fromPath read the whole
file, encodeForm copied every part into FormData Blobs, and a
ReadableStream InputFile was drained into memory before the send. Replace
the FormData encoder with a hand-rolled multipart/form-data body built as
a web ReadableStream and passed straight to fetch (duplex: "half"), so
file bytes flow from their source without ever being materialized - a
100MB upload now retains ~1MB instead of 200-300MB.

- src/core/multipart.ts: the streaming encoder. The body is laid out once
  as ordered pieces (encoded part headers + each file's Blob/Uint8Array/
  stream); each send attempt streams the pieces into a fresh
  ReadableStream. A once-per-process probe detects runtimes whose fetch
  cannot stream a request body; those get the same bytes buffered into a
  single Blob. Bun is excluded from streaming by name: its fetch accepts
  a stream body but stalls forever against an HTTPS origin (Bun 1.3.x,
  verified live; the same bytes as a plain body succeed), which no local
  probe can catch - drop the guard when Bun's streaming uploads work.
- encodeForm now returns { headers, body: () => ..., replayable }: a
  per-attempt body factory instead of a FormData value.
- Transport builds a fresh body per attempt and sends stream bodies with
  duplex: "half". Retry semantics change deliberately: Blob/Uint8Array
  uploads re-stream on retry (byte-identical), while a caller-provided
  one-shot ReadableStream is sent exactly once and a failure surfaces
  immediately instead of retrying an already-drained stream (previously
  streams were buffered to make them retryable - the worst offender).
- fromPath returns a disk-backed Blob via fs.openAsBlob (feature-
  detected; Node < 19.8 falls back to readFile), so the common upload
  path never loads the file into memory and stays retry-safe.
- Remove inputFileToBlob: its only purpose was feeding FormData.

Verified live against api.telegram.org on Node (plain bytes, fromPath,
one-shot stream, nested attach:// media groups - chunked transfer
accepted) and on Bun via the buffered fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
30s cut off large uploads mid-stream; now that uploads stream end-to-end,
give them room by default. Pass timeoutMs to tighten it for
latency-sensitive bots (getUpdates already derives its own long-poll
timeout).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Graft the two improvements from the PR #1331 review:

- InputFileStreamFactory (() => ReadableStream | Promise<ReadableStream>)
  as a fourth InputFileData variant: a replayable stream source. The
  factory piece is invoked per body build, so each transport attempt
  opens a fresh stream and the upload stays retryable - unlike a bare
  ReadableStream, which remains one-shot. Verified live against
  api.telegram.org.
- safeContentType: meta.contentType is emitted verbatim into a multipart
  part header; refuse CR/LF so a hostile value cannot inject headers or
  forge parts (names/filenames were already escaped).

Also adopt the lazy-encode assertion: building the body must not read
the upload source; bytes flow only when the body is consumed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
yagop and others added 3 commits July 10, 2026 11:25
fromPath streams from disk now (not a full read), and
InputFileStreamFactory is the way to keep retries for arbitrary stream
sources - demo both in examples/08-uploads.ts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a fetch()-based upload (telegram.org/example/video.mp4) as example 2:
the response body streams straight into sendVideo via a stream factory,
so nothing is buffered and a retry re-fetches the source. Also fix the
album step, which uploaded text bytes as a photo - Telegram rejects that
with IMAGE_PROCESS_FAILED; it now streams a fetched PNG instead. Ran
live: all six steps succeed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live 200MB uploads from Deno 2.9 showed +481MB peak RSS: Deno passes the
request-stream probe and streams fine, but its node-compat fs.openAsBlob
reads the entire file into memory at creation (+401MB for a 200MB file),
unlike Node's lazy disk-backed Blob. Wrap the file as an
InputFileStreamFactory instead: each attempt opens a fresh
createReadStream, so memory stays flat on every runtime, retries still
re-read from disk, and the Node >= 19.8 openAsBlob dependency goes away.
stat() up front so a missing path fails at fromPath, not mid-request.

Measured on the same 200MB live upload: Deno +481MB -> +230MB peak RSS
(the remainder is transient chunk churn, comparable to Node against the
same endpoint). Adds the previously missing fromPath unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Bun exclusion in supportsRequestStreams now points at
oven-sh/bun#33918 (fetch with a ReadableStream request body stalls
against HTTPS origins), filed with a dependency-free repro - remove the
guard when it is fixed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@yagop

yagop commented Jul 10, 2026

Copy link
Copy Markdown
Owner Author

The Bun streaming exclusion now has an upstream anchor: filed oven-sh/bun#33918 (oven-sh/bun#33918) with a dependency-free repro - fetch() with a ReadableStream request body stalls forever against HTTPS origins on Bun 1.3.14 (latest stable), while the same bytes as a plain body and the same stream against plain-HTTP origins both succeed; Node 26 passes all four combinations. The supportsRequestStreams guard comment references the issue; when it is fixed upstream, the guard comes off and Bun streams like Node/Deno.

yagop and others added 3 commits July 10, 2026 12:32
Verified outside the sandbox: Bun 1.3.14 streams fetch request bodies
fine on a direct connection - the stall only happens when the request is
routed through an HTTP(S) CONNECT proxy, which Bun picks up from the
proxy env vars automatically (oven-sh/bun#33918, retitled; controls:
Node 26 via NODE_USE_ENV_PROXY and curl chunked both pass through the
same proxy). So exclude Bun from streaming only when a proxy is
configured, read via Bun.env so the core stays free of Node globals:
direct-connection Bun users now get flat-memory streamed uploads;
proxied Bun keeps the correct buffered fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#33918)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bun triage confirmed our report (oven-sh/bun#33918) and root-caused it:
with a ReadableStream body through a CONNECT tunnel, the ProxyHeaders
stage treated the empty stream buffer as already-sent and the chunked
plaintext went to the outer proxy socket instead of the TLS session.
Fixed by oven-sh/bun#32635, first in 1.4.0; verified on canary through
the same sandbox proxy that stalled (all four combos pass).

The guard now buffers only proxied Bun < 1.4.0, so it retires itself as
users upgrade. All four branches unit-verified: proxied 1.3.14
(buffered), direct 1.3.14 (streams), proxied 1.4.0 canary (streams),
Node (streams).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@yagop

yagop commented Jul 10, 2026

Copy link
Copy Markdown
Owner Author

Upstream resolution: Bun triage confirmed oven-sh/bun#33918 and root-caused it to the CONNECT-tunnel stream-body path (fixed by oven-sh/bun#32635, first shipping in Bun 1.4.0; verified here on canary through the same proxy that stalled - all four combinations pass). The guard is now version-gated: only proxied Bun < 1.4.0 takes the buffered fallback, so the workaround retires itself as users upgrade. All four guard branches are unit-verified.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@yagop
yagop merged commit e50831f into master Jul 10, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant