Use HeadObject instead of GetObject for S3 metadata#565
Merged
pjbull merged 2 commits intoJun 10, 2026
Conversation
pjbull
reviewed
Jun 9, 2026
pjbull
left a comment
pjbull
left a comment
Member
There was a problem hiding this comment.
Generally looking good, a few quick comments in there.
Did you test against your s3 compatible server and real s3?
Also, did you make sure linting passes?
| data = self.client.head_object( | ||
| Bucket=cloud_path.bucket, | ||
| Key=cloud_path.key, | ||
| **self.boto3_dl_extra_args, |
Member
There was a problem hiding this comment.
Does head_object respect what we collect in these kwargs?
Contributor
Author
There was a problem hiding this comment.
Yes, checked all 7 (see https://docs.aws.amazon.com/boto3/latest/reference/services/s3/client/head_object.html)
|
|
||
| ## UNRELEASED | ||
|
|
||
| - Changed `S3Client._get_metadata` to read object metadata with `HeadObject` instead of `GetObject`, so `stat`, `etag`, and `size` no longer open the object body. Also fixes a `KeyError` on `ContentLength` against S3-compatible gateways that drop `Content-Length` from `GetObject` responses. (Issue [#564](https://github.com/drivendataorg/cloudpathlib/issues/564), PR [#TODO]) |
Contributor
Author
|
Yes, linting passed and checked on real s3 bucket and s3 compatible server (lakefs). |
Contributor
Author
|
Code quality errors are unrelated to my changes I think. |
Member
|
Looks like linter package version updates. I'll get it fixed on main so you can rebase. |
Member
|
@mattijsdp fixed on the main branch if you want to rebase this. Thanks for bearing with. |
S3Client._get_metadata read object metadata via ObjectSummary.get() (a GetObject), which issues a full object read and leaves an unread response stream instead of returning the connection to the pool. Switch to head_object, which reads only the metadata and keeps Content-Length on S3-compatible gateways that drop it from GetObject responses, while bringing S3 in line with the other backends. Since head_object returns no body, a missing key surfaces as a generic 404 ClientError rather than NoSuchKey, so S3Path.stat now translates a 404 to NoStatError and lets other errors (e.g. 403) propagate. Updates the S3 test mock and adds regression tests. Closes drivendataorg#564 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mattijsdp
force-pushed
the
head-object-for-s3-metadata
branch
from
edce3fa to
8f0a536
Compare
Contributor
Author
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #565 +/- ##
========================================
- Coverage 93.9% 93.2% -0.8%
========================================
Files 28 28
Lines 2222 2226 +4
========================================
- Hits 2088 2075 -13
- Misses 134 151 +17
🚀 New features to boost your workflow:
|
Member
|
live tests expected to fail on fork, moving to repo-local branch to test real backends. |
pjbull
added a commit
that referenced
this pull request
Jun 10, 2026
* Use HeadObject instead of GetObject for S3 metadata S3Client._get_metadata read object metadata via ObjectSummary.get() (a GetObject), which issues a full object read and leaves an unread response stream instead of returning the connection to the pool. Switch to head_object, which reads only the metadata and keeps Content-Length on S3-compatible gateways that drop it from GetObject responses, while bringing S3 in line with the other backends. Since head_object returns no body, a missing key surfaces as a generic 404 ClientError rather than NoSuchKey, so S3Path.stat now translates a 404 to NoStatError and lets other errors (e.g. 403) propagate. Updates the S3 test mock and adds regression tests. Closes #564 * add PR number --------- Co-authored-by: Mattijs De Paepe <33912270+mattijsdp@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #564
S3Client._get_metadata read object metadata via ObjectSummary.get() (a GetObject), which issues a full object read and leaves an unread response stream instead of returning the connection to the pool. Switch to head_object, which reads only the metadata and keeps Content-Length on S3-compatible gateways that drop it from GetObject responses, while bringing S3 in line with the other backends.
Since head_object returns no body, a missing key surfaces as a generic 404 ClientError rather than NoSuchKey, so S3Path.stat now translates a 404 to NoStatError and lets other errors (e.g. 403) propagate. Updates the S3 test mock and adds regression tests.
Contributor checklist:
CONTRIBUTING.mdCloses #issueappears in the PR summary (e.g.,Closes #123).HISTORY.mdwith the issue that is addressed and the PR you are submitting. If the top section is not `## UNRELEASED``, then you need to add a new section to the top of the document for your change.