BACKUP of a table on a CAS disk to an S3(...) destination on the same endpoint fails on AWS S3. The statement throws, and no backup is produced. The table itself is left as it was.
Every MergeTree part hits this. primary.idx, marks (.mrk*), and column files (.bin) are stored as blobs, including when they are tiny. A blob object is a short envelope header plus the file. The backup copies only the file bytes with a ranged UploadPartCopy. AWS accepts a byte range only when the source object is greater than 5 MiB, and answers InvalidRequest ("The specified copy source is not supported as a byte-range copy source") otherwise. primary.idx and the marks of an ordinary part are far below that, so the first such file aborts the BACKUP.
ClickHouse falls back to reading the file and uploading it when multipart copy is disabled, when the store is GCS, or when UploadPartCopy returns AccessDenied. InvalidRequest is not one of those cases, so the refusal fails the statement. A backup that gets past this check is not written with the envelope included; it does not get that far.
Workaround: force the buffer path.
BACKUP TABLE t TO S3('https://s3.amazonaws.com/bucket/path', 'key', 'secret')
SETTINGS s3_allow_multipart_copy = 0allow_s3_native_copy = 0 does the same. Blobs larger than 5 MiB can still use the server-side copy.
Caused by: #2415
Why the fast path rejects a normal part
partFileMustStayBlob in ContentAddressedTransaction.cpp keeps primary.idx, .bin, and .mrk* out of the manifest. BackupWriterS3::tryNativeCopyFromContentAddressedDisk then copies each of those with copyS3File at payload_offset (the envelope length, never zero). CopyFileHelper::performCopy uses UploadPartCopy for every non-zero offset, and performMultipartUploadCopy falls back only on Aws::S3::S3Errors::ACCESS_DENIED.
AWS documents the size rule on UploadPartCopy: a CopySourceRange is valid only if the source object is greater than 5 MiB. https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html
The integration test test_cas_backup_s3_native_copy runs against RustFS with 100000-row parts. That store accepts a ranged copy of a small object, so the AWS refusal is not covered.
Fix direction
Skip the ranged copy when the blob object is not greater than 5 MiB, and copy that file through buffers. Treating this InvalidRequest as the same fallback as AccessDenied covers a store that enforces the rule when the local size check does not.
BACKUPof a table on a CAS disk to anS3(...)destination on the same endpoint fails on AWS S3. The statement throws, and no backup is produced. The table itself is left as it was.Every MergeTree part hits this.
primary.idx, marks (.mrk*), and column files (.bin) are stored as blobs, including when they are tiny. A blob object is a short envelope header plus the file. The backup copies only the file bytes with a rangedUploadPartCopy. AWS accepts a byte range only when the source object is greater than 5 MiB, and answersInvalidRequest("The specified copy source is not supported as a byte-range copy source") otherwise.primary.idxand the marks of an ordinary part are far below that, so the first such file aborts theBACKUP.ClickHouse falls back to reading the file and uploading it when multipart copy is disabled, when the store is GCS, or when
UploadPartCopyreturnsAccessDenied.InvalidRequestis not one of those cases, so the refusal fails the statement. A backup that gets past this check is not written with the envelope included; it does not get that far.Workaround: force the buffer path.
allow_s3_native_copy = 0does the same. Blobs larger than 5 MiB can still use the server-side copy.Caused by: #2415
Why the fast path rejects a normal part
partFileMustStayBlobinContentAddressedTransaction.cppkeepsprimary.idx,.bin, and.mrk*out of the manifest.BackupWriterS3::tryNativeCopyFromContentAddressedDiskthen copies each of those withcopyS3Fileatpayload_offset(the envelope length, never zero).CopyFileHelper::performCopyusesUploadPartCopyfor every non-zero offset, andperformMultipartUploadCopyfalls back only onAws::S3::S3Errors::ACCESS_DENIED.AWS documents the size rule on
UploadPartCopy: aCopySourceRangeis valid only if the source object is greater than 5 MiB. https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.htmlThe integration test
test_cas_backup_s3_native_copyruns against RustFS with 100000-row parts. That store accepts a ranged copy of a small object, so the AWS refusal is not covered.Fix direction
Skip the ranged copy when the blob object is not greater than 5 MiB, and copy that file through buffers. Treating this
InvalidRequestas the same fallback asAccessDeniedcovers a store that enforces the rule when the local size check does not.