feat(v2): stream multipart uploads instead of buffering them - #1332
Conversation
Uploads used to hold 2-4x the file size in RAM: fromPath read the whole
file, encodeForm copied every part into FormData Blobs, and a
ReadableStream InputFile was drained into memory before the send. Replace
the FormData encoder with a hand-rolled multipart/form-data body built as
a web ReadableStream and passed straight to fetch (duplex: "half"), so
file bytes flow from their source without ever being materialized - a
100MB upload now retains ~1MB instead of 200-300MB.
- src/core/multipart.ts: the streaming encoder. The body is laid out once
as ordered pieces (encoded part headers + each file's Blob/Uint8Array/
stream); each send attempt streams the pieces into a fresh
ReadableStream. A once-per-process probe detects runtimes whose fetch
cannot stream a request body; those get the same bytes buffered into a
single Blob. Bun is excluded from streaming by name: its fetch accepts
a stream body but stalls forever against an HTTPS origin (Bun 1.3.x,
verified live; the same bytes as a plain body succeed), which no local
probe can catch - drop the guard when Bun's streaming uploads work.
- encodeForm now returns { headers, body: () => ..., replayable }: a
per-attempt body factory instead of a FormData value.
- Transport builds a fresh body per attempt and sends stream bodies with
duplex: "half". Retry semantics change deliberately: Blob/Uint8Array
uploads re-stream on retry (byte-identical), while a caller-provided
one-shot ReadableStream is sent exactly once and a failure surfaces
immediately instead of retrying an already-drained stream (previously
streams were buffered to make them retryable - the worst offender).
- fromPath returns a disk-backed Blob via fs.openAsBlob (feature-
detected; Node < 19.8 falls back to readFile), so the common upload
path never loads the file into memory and stays retry-safe.
- Remove inputFileToBlob: its only purpose was feeding FormData.
Verified live against api.telegram.org on Node (plain bytes, fromPath,
one-shot stream, nested attach:// media groups - chunked transfer
accepted) and on Bun via the buffered fallback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
30s cut off large uploads mid-stream; now that uploads stream end-to-end, give them room by default. Pass timeoutMs to tighten it for latency-sensitive bots (getUpdates already derives its own long-poll timeout). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Graft the two improvements from the PR #1331 review: - InputFileStreamFactory (() => ReadableStream | Promise<ReadableStream>) as a fourth InputFileData variant: a replayable stream source. The factory piece is invoked per body build, so each transport attempt opens a fresh stream and the upload stays retryable - unlike a bare ReadableStream, which remains one-shot. Verified live against api.telegram.org. - safeContentType: meta.contentType is emitted verbatim into a multipart part header; refuse CR/LF so a hostile value cannot inject headers or forge parts (names/filenames were already escaped). Also adopt the lazy-encode assertion: building the body must not read the upload source; bytes flow only when the body is consumed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fromPath streams from disk now (not a full read), and InputFileStreamFactory is the way to keep retries for arbitrary stream sources - demo both in examples/08-uploads.ts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a fetch()-based upload (telegram.org/example/video.mp4) as example 2: the response body streams straight into sendVideo via a stream factory, so nothing is buffered and a retry re-fetches the source. Also fix the album step, which uploaded text bytes as a photo - Telegram rejects that with IMAGE_PROCESS_FAILED; it now streams a fetched PNG instead. Ran live: all six steps succeed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live 200MB uploads from Deno 2.9 showed +481MB peak RSS: Deno passes the request-stream probe and streams fine, but its node-compat fs.openAsBlob reads the entire file into memory at creation (+401MB for a 200MB file), unlike Node's lazy disk-backed Blob. Wrap the file as an InputFileStreamFactory instead: each attempt opens a fresh createReadStream, so memory stays flat on every runtime, retries still re-read from disk, and the Node >= 19.8 openAsBlob dependency goes away. stat() up front so a missing path fails at fromPath, not mid-request. Measured on the same 200MB live upload: Deno +481MB -> +230MB peak RSS (the remainder is transient chunk churn, comparable to Node against the same endpoint). Adds the previously missing fromPath unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Bun exclusion in supportsRequestStreams now points at oven-sh/bun#33918 (fetch with a ReadableStream request body stalls against HTTPS origins), filed with a dependency-free repro - remove the guard when it is fixed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The Bun streaming exclusion now has an upstream anchor: filed oven-sh/bun#33918 (oven-sh/bun#33918) with a dependency-free repro - |
Verified outside the sandbox: Bun 1.3.14 streams fetch request bodies fine on a direct connection - the stall only happens when the request is routed through an HTTP(S) CONNECT proxy, which Bun picks up from the proxy env vars automatically (oven-sh/bun#33918, retitled; controls: Node 26 via NODE_USE_ENV_PROXY and curl chunked both pass through the same proxy). So exclude Bun from streaming only when a proxy is configured, read via Bun.env so the core stays free of Node globals: direct-connection Bun users now get flat-memory streamed uploads; proxied Bun keeps the correct buffered fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#33918) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bun triage confirmed our report (oven-sh/bun#33918) and root-caused it: with a ReadableStream body through a CONNECT tunnel, the ProxyHeaders stage treated the empty stream buffer as already-sent and the chunked plaintext went to the outer proxy socket instead of the TLS session. Fixed by oven-sh/bun#32635, first in 1.4.0; verified on canary through the same sandbox proxy that stalled (all four combos pass). The guard now buffers only proxied Bun < 1.4.0, so it retires itself as users upgrade. All four branches unit-verified: proxied 1.3.14 (buffered), direct 1.3.14 (streams), proxied 1.4.0 canary (streams), Node (streams). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Upstream resolution: Bun triage confirmed oven-sh/bun#33918 and root-caused it to the CONNECT-tunnel stream-body path (fixed by oven-sh/bun#32635, first shipping in Bun 1.4.0; verified here on canary through the same proxy that stalled - all four combinations pass). The guard is now version-gated: only proxied Bun < 1.4.0 takes the buffered fallback, so the workaround retires itself as users upgrade. All four guard branches are unit-verified. |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
npm run checkis clean (typecheck src + test + examples,lint:core,check:edge, unit suite: 130 pass on Node; on Bun 129 pass + 1 skip proxied, 130 pass direct)npm run buildis cleandoc/api.md); generated API sources untouchedDescription
Uploads used to hold 2-4x the file size in RAM:
fromPathread the whole file,encodeFormcopied every part intoFormDataBlobs, and aReadableStreamInputFilewas drained into an in-memoryBlobbefore the send. This PR replaces theFormDataencoder with a hand-rolledmultipart/form-databody built as a webReadableStreamand passed straight tofetch(duplex: "half"), so file bytes flow from their source without ever being materialized. Measured on a 100MB upload: ~1MB retained vs 200-300MB before.src/core/multipart.ts- the streaming encoder. The body is laid out once as ordered pieces (encoded part headers + each file'sBlob/Uint8Array/stream/factory); each send attempt streams the pieces into a freshReadableStream. Web-standard APIs only;lint:core+check:edgestay green.Blob/Uint8Arrayuploads re-stream on retry (byte-identical); a bareReadableStreamis one-shot - sent once, a failure surfaces immediately; the newInputFileStreamFactory(() => ReadableStream | Promise<ReadableStream>) opens a fresh stream per attempt and stays retryable.fromPath()returns a disk-backedBlobviafs.openAsBlob(Node >= 19.8, feature-detected; older Nodes fall back toreadFile), so disk uploads are flat-memory and retry-safe.Blob, chosen by a once-per-process probe. Bun is excluded from streaming only when an HTTP(S) proxy is configured (read viaBun.env, which Bun honors automatically): itsfetchstreams request bodies fine on a direct connection, but a stream body routed through a CONNECT proxy stalls forever (fetch() with a ReadableStream request body stalls when routed through an HTTP(S) CONNECT proxy (direct connections work) oven-sh/bun#33918; Node and curl pass through the same proxy, so no local probe can catch it). Direct-connection Bun streams like Node/Deno; drop the guard when the upstream issue is fixed.meta.contentTypeis guarded against CR/LF header injection into part headers; part names/filenames use the WHATWG escaping.inputFileToBlobis removed - nothing converts toFormDataanymore.timeoutMsraised 30s -> 5 min so large uploads are not cut off mid-stream (getUpdatesalready derives its own long-poll timeout).Validation
fromPath, one-shot stream, stream factory, and nestedattach://media-group uploads (chunked transfer-encoding accepted; confirmed independently with curl).Supersedes #1331 (same goal, independently implemented): adopts its
InputFileStreamFactoryand content-type hardening, and additionally removes theFormDatadouble-copy forBlob/Uint8Arrayuploads and adds the no-stream fallback that keeps proxied Bun working (streamed uploads there stall through CONNECT proxies, which local-HTTP validation does not catch).🤖 Generated with Claude Code