feat(upload): Skip decompression - #6358
Draft
jjbayer wants to merge 2 commits into
Draft
Conversation
The `/upload` PATCH endpoint no longer decompresses request bodies. A `zstd` body is forwarded or stored verbatim and the algorithm is recorded on the object via objectstore's `precompressed` API. Any other content encoding is rejected with 415. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![sentry-warden[bot]](https://avatars.githubusercontent.com/in/2755838?s=60&v=4){kind=small}
Comment on lines
+161
to
+164
| /// The compression the stream is already encoded with, if any. | ||
| /// | ||
| /// The stream is uploaded verbatim and the algorithm is recorded on the object. | ||
| pub compression: Option<Compression>, |
There was a problem hiding this comment.
Precompressed uploads bypass decompressed size limits (zip bomb)
Client-controlled Content-Encoding is stored via precompressed metadata while only the compressed wire size is limited, so a small zstd body can expand far past max_upload_size when objectstore or downloads auto-decompress.
Evidence
- The upload PATCH route is merged after
RequestDecompressionLayer, so bodies are no longer decompressed at the edge (endpoints/mod.rsraw_routes). body_compression()accepts clientContent-Encoding(e.g. zstd) and passes it throughupload::Stream.compressioninto thisobjectstore::Stream.compressionfield.attempt_uploadcallsrequest.precompressed(compression)and stores the body verbatim;RequestBodyLimitLayer/BoundedStreamonly cap compressed bytes.- Integration coverage (
test_objectstore_precompressed) asserts downloads auto-decompress the annotated object, so a high-ratio compressed payload expands on read without an ingest-time decompressed size check.
Also found at 1 additional location
relay-server/src/services/upload.rs:14-20
Identified by Warden · security-review · 7J5-ST5
Comment on lines
835
to
+845
| @@ -837,6 +842,7 @@ impl ObjectstoreServiceInner { | |||
| body: TakeOnce::new(stream), | |||
| upload_ref, | |||
| retention, | |||
| compression, | |||
There was a problem hiding this comment.
Precompressed upload path bounds only wire size, not decompressed size
Client Content-Encoding is plumbed into objectstore precompressed() while RequestBodyLimitLayer/BoundedStream only cap compressed bytes, so a small zstd body (e.g. with Upload-Defer-Length) can expand far past max_upload_size on download; enforce a decompressed/logical size cap before accepting precompressed uploads.
Evidence
handle_streamnow forwards attacker-controlledcompressionintoUpload::Stream(this hunk);attempt_uploadcallsrequest.precompressed(compression)and stores the body verbatim.- Upload PATCH is merged outside
RequestDecompressionLayer(endpoints/mod.rsraw_routes), andbody_compression()accepts clientContent-Encoding(e.g. zstd) without inflating or measuring output size. RequestBodyLimitLayer/BoundedStreamonly cap wire bytes (max_upload_size); withUpload-Defer-Lengththe bound is(1, max_upload_size)on compressed bytes only.test_objectstore_precompressedassertssession.get(key)auto-decompresses the annotated object, so a high-ratio payload expands on read with no ingest-time decompressed-size check.
Also found at 1 additional location
relay-server/src/services/objectstore.rs:974-979
Identified by Warden · wrdn-dos-review · XX6-2T3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ref: INGEST-1162