Description
When ALTER TABLE ... EXPORT PARTITION ... TO TABLE <s3_dest> targets an S3
destination with invalid credentials, the resulting S3_ERROR
(SignatureDoesNotMatch) is classified as retryable. The export does not
fail fast — it stays PENDING, retries with backoff, and only stops when
export_merge_tree_partition_task_timeout_seconds elapses (default 1 day),
ending as KILLED.
A wrong/expired key is deterministic and permanent: retrying can never succeed,
so burning the whole task timeout before giving up is undesirable.
Introduced by the error-classification changes in PR #1984. The classifier is a
denylist (isNonRetryableExportError); S3_ERROR is not listed, so it falls
through to the retryable bucket — including permanent auth failures.
Version: 26.3.13
Steps to reproduce
CREATE TABLE source (p UInt8, i UInt64)
ENGINE = MergeTree ORDER BY tuple() PARTITION BY p;
INSERT INTO source SELECT 1, rand64() FROM numbers(10);
-- Destination with a WRONG secret key
CREATE TABLE s3_bad_creds (p UInt8, i UInt64)
ENGINE = S3('http://minio:9001/root/data/.../', 'minio_user',
'wrong_secret_key_deadbeef', filename = 's3_bad_creds',
format = 'Parquet', partition_strategy = 'hive');
-- Short timeout so the KILLED transition is observable
ALTER TABLE source EXPORT PARTITION ID '1' TO TABLE s3_bad_creds
SETTINGS export_merge_tree_partition_task_timeout_seconds = 10;
Expected
Fail fast with FAILED and a clear S3 authentication error.
Actual
Each attempt fails with a retryable S3_ERROR, the task stays PENDING, and it
only becomes KILLED once the task timeout elapses:
Code: 499. DB::Exception: ... S3 exception: `SignatureDoesNotMatch`, message:
'The request signature we calculated does not match the signature you provided.
Check your key and signing method.'. (S3_ERROR)
SELECT status FROM system.replicated_partition_exports
WHERE source_table = 'source' AND partition_id = '1';
-- KILLED (after the task timeout)
Impact / concern
- Permanent auth failures (typo'd key, expired/rotated credential, missing
bucket permission) are treated as transient. With the default 1-day timeout the
export appears to "hang" in PENDING before it is KILLED.
KILLED-on-timeout hides the real cause; a fast FAILED with the S3 auth
error would be far more actionable.
Description
When
ALTER TABLE ... EXPORT PARTITION ... TO TABLE <s3_dest>targets an S3destination with invalid credentials, the resulting
S3_ERROR(
SignatureDoesNotMatch) is classified as retryable. The export does notfail fast — it stays
PENDING, retries with backoff, and only stops whenexport_merge_tree_partition_task_timeout_secondselapses (default 1 day),ending as
KILLED.A wrong/expired key is deterministic and permanent: retrying can never succeed,
so burning the whole task timeout before giving up is undesirable.
Introduced by the error-classification changes in PR #1984. The classifier is a
denylist (
isNonRetryableExportError);S3_ERRORis not listed, so it fallsthrough to the retryable bucket — including permanent auth failures.
Version:
26.3.13Steps to reproduce
Expected
Fail fast with
FAILEDand a clear S3 authentication error.Actual
Each attempt fails with a retryable
S3_ERROR, the task staysPENDING, and itonly becomes
KILLEDonce the task timeout elapses:Impact / concern
bucket permission) are treated as transient. With the default 1-day timeout the
export appears to "hang" in
PENDINGbefore it isKILLED.KILLED-on-timeout hides the real cause; a fastFAILEDwith the S3 autherror would be far more actionable.