Skip to content

EXPORT PARTITION retries permanent S3 credential errors until the task timeout #2005

Description

@Selfeer

Description

When ALTER TABLE ... EXPORT PARTITION ... TO TABLE <s3_dest> targets an S3
destination with invalid credentials, the resulting S3_ERROR
(SignatureDoesNotMatch) is classified as retryable. The export does not
fail fast — it stays PENDING, retries with backoff, and only stops when
export_merge_tree_partition_task_timeout_seconds elapses (default 1 day),
ending as KILLED.

A wrong/expired key is deterministic and permanent: retrying can never succeed,
so burning the whole task timeout before giving up is undesirable.

Introduced by the error-classification changes in PR #1984. The classifier is a
denylist (isNonRetryableExportError); S3_ERROR is not listed, so it falls
through to the retryable bucket — including permanent auth failures.

Version: 26.3.13

Steps to reproduce

CREATE TABLE source (p UInt8, i UInt64)
ENGINE = MergeTree ORDER BY tuple() PARTITION BY p;
INSERT INTO source SELECT 1, rand64() FROM numbers(10);

-- Destination with a WRONG secret key
CREATE TABLE s3_bad_creds (p UInt8, i UInt64)
ENGINE = S3('http://minio:9001/root/data/.../', 'minio_user',
            'wrong_secret_key_deadbeef', filename = 's3_bad_creds',
            format = 'Parquet', partition_strategy = 'hive');

-- Short timeout so the KILLED transition is observable
ALTER TABLE source EXPORT PARTITION ID '1' TO TABLE s3_bad_creds
SETTINGS export_merge_tree_partition_task_timeout_seconds = 10;

Expected

Fail fast with FAILED and a clear S3 authentication error.

Actual

Each attempt fails with a retryable S3_ERROR, the task stays PENDING, and it
only becomes KILLED once the task timeout elapses:

Code: 499. DB::Exception: ... S3 exception: `SignatureDoesNotMatch`, message:
'The request signature we calculated does not match the signature you provided.
Check your key and signing method.'. (S3_ERROR)
SELECT status FROM system.replicated_partition_exports
WHERE source_table = 'source' AND partition_id = '1';
-- KILLED (after the task timeout)

Impact / concern

  • Permanent auth failures (typo'd key, expired/rotated credential, missing
    bucket permission) are treated as transient. With the default 1-day timeout the
    export appears to "hang" in PENDING before it is KILLED.
  • KILLED-on-timeout hides the real cause; a fast FAILED with the S3 auth
    error would be far more actionable.

Metadata

Metadata

Assignees

Labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions