Skip to content

fix(drivers/s3): support multipart object copy#2826

Merged
PIKACHUIM merged 2 commits into
mainfrom
fix-2358
Jul 23, 2026
Merged

fix(drivers/s3): support multipart object copy#2826
PIKACHUIM merged 2 commits into
mainfrom
fix-2358

Conversation

@jyxjjj

@jyxjjj jyxjjj commented Jul 23, 2026

Copy link
Copy Markdown
Member

Summary / 摘要

  • Keep using CopyObject for objects up to 5 GB.

  • Use CreateMultipartUpload, UploadPartCopy, and CompleteMultipartUpload for larger objects.

  • Dynamically size copy parts while respecting the 10,000-part and 5 GiB-per-part limits.

  • Preserve common HTTP headers and custom object metadata during multipart copies.

  • Abort unfinished multipart uploads when copying or completion fails.

  • Pass object sizes through copy, rename, move, and recursive directory-copy operations.

  • Add tests covering size routing, copy ranges, metadata preservation, completion, part sizing, and failure cleanup.

  • No configuration, public API, or storage-format changes are required.

  • This PR has breaking changes.
    / 此 PR 包含破坏性变更。

  • This PR changes public API, config, storage format, or migration behavior.
    / 此 PR 修改了公开 API、配置、存储格式或迁移行为。

  • This PR requires corresponding changes in related repositories.
    / 此 PR 需要关联仓库同步修改。

Related repository PRs / 关联仓库 PR:

  • OpenList-Frontend: N/A
  • OpenList-Docs: N/A

Related Issues / 关联 Issue

Fixes #2358

Testing / 测试

  • go test ./...
  • Manual test / 手动测试:

Executed checks:

  • go test ./drivers/s3 — passed
  • go vet ./drivers/s3 — passed
  • git diff --check — passed

The complete test suite and manual Backblaze integration testing were not run. Targeted tests use an HTTP test server to exercise the AWS SDK request flow without external storage.

Checklist / 检查清单

  • I have read CONTRIBUTING.
    / 我已阅读 CONTRIBUTING
  • I confirm this contribution follows the repository license, contribution policy, and code of conduct.
    / 我确认此贡献符合仓库许可证、贡献规范和行为准则。
  • I have formatted the changed code with gofmt, go fmt, or prettier where applicable.
    / 我已按适用情况使用 gofmtgo fmtprettier 格式化变更代码。
  • I have requested review from relevant maintainers or code owners where applicable.
    / 我已在适用情况下请求相关维护者或代码所有者审查。

AI Disclosure / AI 使用声明

  • This PR includes AI-assisted content.
    / 此 PR 包含 AI 辅助内容。

Tools used / 使用工具:

  • ChatGPT
  • Codex
  • GitHub Copilot
  • Claude
  • Gemini
  • Other (please specify) / 其他(请注明):

Usage scope / 使用范围:

  • Code generation / 代码生成

  • Refactoring / 重构

  • Documentation / 文档

  • Tests / 测试

  • Translation / 翻译

  • Review assistance / 审查辅助

  • I have reviewed and validated all AI-assisted content included in this PR.
    / 我已审核并验证此 PR 中包含的所有 AI 辅助内容。

  • I have ensured that all AI-assisted commits include Co-Authored-By attribution.
    / 我已确保所有 AI 辅助提交都包含 Co-Authored-By 归属信息。

  • I can reproduce all AI-assisted content included in this PR without any AI tools.
    / 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。

- Route objects larger than 5 GB through multipart copy
- Preserve source metadata and abort incomplete multipart copies
- Cover size routing, byte ranges, metadata, and failure cleanup

Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
Signed-off-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>
@xrgzs xrgzs changed the title fix(s3): support multipart object copy fix(drivers/s3): support multipart object copy Jul 23, 2026
@xrgzs xrgzs added enhancement Module: Driver Driver-Related Issue/PR labels Jul 23, 2026
@PIKACHUIM
PIKACHUIM merged commit fe56ca4 into main Jul 23, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Module: Driver Driver-Related Issue/PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Backblaze S3 对大于 5GB 的文件复制失败,需要支持分段复制。

3 participants