Conversation
GC sent a write-once cohort (up to `gc_bulk_delete_chunk_keys`, 1000) as one `removeManyWriteOnce`, ignoring `objects_chunk_size_to_delete` on S3 and the single-key reality of storages without `DeleteObjects`. Add `IObjectStorage::batchDeleteKeyLimit` and `Backend::bulkDeleteKeyLimit` (forwarded by every decorator). `Gc::bulkDeleteChunkKeys` takes the smaller of the setting and that limit; `removeCohortWriteOnce` cuts the `manifest_deletes` and `cleanupRefObjects` cohorts into requests of that size. The cohort, its budget accounting and the one `authorityHolds` read per cohort are unchanged. The S3 delete paths also pass the store's error name into `S3Exception`: `isRefreshableCredentialError` recognises SDK-unmodelled errors only by `getExceptionName`. Tests: `CASBulkDeleteBackend` limit forwarding and clamping, GC cut for `manifest_deletes` and ref cleanup (no extra `gc/state` reads), and `CASS3BatchDelete` for the S3 limit and error names. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CAS GC sent a write-once delete cohort (up to
cas_gc_bulk_delete_chunk_keys, 1000 keys) as one request, ignoringobjects_chunk_size_to_deleteon S3 and the one-key limit of a storage withoutDeleteObjects. GC now cuts each cohort into requests of the storage's batch-delete limit. The S3 delete paths also keep the store's error name inS3Exception.Changes
IObjectStorage::batchDeleteKeyLimit(default 1).S3ObjectStoragereturnsobjects_chunk_size_to_delete, or 1 onceDeleteObjectsis known unsupported.Cas::Backend::bulkDeleteKeyLimit, forwarded by every decorator;kBulkDeleteMaxKeysfor the in-memory and the emulated backend.Gc::bulkDeleteChunkKeysis the smaller of the setting and that limit. Themanifest_deletesphase andcleanupRefObjectscut a cohort into requests of that size through the existingremoveChunkWriteOnceOrOneByOne. The cohort, its budget accounting and the oneauthorityHoldsread per cohort are unchanged.S3ObjectStorage::removeObjectsIfExistImplanddeleteFileFromS3pass the store's error name intoS3Exception:isRefreshableCredentialErrorrecognises errors the SDK does not model only bygetExceptionName.Risks / notes
IObjectStorage.h, one override and twothrowsites inS3ObjectStorage.cpp, onethrowsite insrc/IO/S3/deleteFileFromS3.cpp. The exception type and message are the same; only the name is added.objects_chunk_size_to_deletebelow 1000 now gets more, smallerDeleteObjectsrequests from GC phases 15 and 17.Testing
CAS*gate, debug and ASan. New:CASBulkDeleteBackendlimit forwarding and clamping (5 tests),CASGCManifestBulkDelete.StorageLimitCutsTheCohortIntoRequestsOfAtMostThatManyKeys,CASRefGc.RefObjectCleanupCutsRequestsToTheStorageLimitWithoutExtraAuthorityReads,CASS3BatchDelete(the S3 limit; error names for a whole-request error and for a per-key error in a 200 reply). WithbulkDeleteChunkKeysignoring the backend limit, both GC-level tests and twoCASBulkDeleteBackendtests fail.CASS3BatchDeleteerror-name test was not run against the oldthrowsites; no integration run on RustFS or S3.Related: #2483
Related: #2468
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
CAS GC batch deletes respect the storage's batch-delete limit (
objects_chunk_size_to_deleteon S3), and S3 delete errors keep the store's error name.Documentation entry for user-facing changes
docs/en/antalya/cas/configuration.md)CI/CD Options
Regression jobs to run: