Repository navigation
Conversation
Extend the existing S3 FileIO assume-role mechanism (client.role-arn / client.role-session-name) to the boto3 sessions built by GlueCatalog and DynamoDbCatalog. When a role ARN is configured, an STS AssumeRole is performed from the base session and the resulting temporary credentials are used for the Glue/DynamoDB client. Role configuration follows the same precedence already used for the other client.* properties: a service-prefixed override (glue.role-arn / dynamodb.role-arn, glue.role-session-name / dynamodb.role-session-name) falls back to the unified client.role-arn / client.role-session-name. Default behavior (no role ARN configured) is unchanged. Closes apache#2747 Signed-off-by: lemon0333 <147061193+lemon0333@users.noreply.github.com>
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
@lemon0333 I'd like to see this land, so I rebased it on today's main to check where it stands: it applies cleanly and One thing I'd raise before a maintainer review: Do you want to reopen this and carry on? If you don't have time for it anymore, I'm happy to open a new PR from your commit with the refresh added, keeping you as co-author. |
Closes #2747
Rationale for this change
At the moment, assume-role (
client.role-arn/client.role-session-name, via STS) is only honored by the S3 FileIO (pyiceberg/io/pyarrow.py). The AWS catalogs (Glue, DynamoDB) build their ownboto3.Sessionand ignore these properties, so users who authenticate to AWS by assuming a role cannot use it to reach the catalog — even though the otherclient.*properties (region, access key, etc.) are already shared between S3 FileIO and the catalogs.This PR extends the same assume-role behavior to the
GlueCatalogandDynamoDbCatalogboto3 sessions:_get_aws_session_with_assumed_role()(inpyiceberg/catalog/__init__.py) performs the STSAssumeRolefrom the base session and returns a new session using the returned temporary credentials.client.*properties: a service-prefixed override falls back to the unified property:glue.role-arn/dynamodb.role-arn→client.role-arnglue.role-session-name/dynamodb.role-session-name→client.role-session-nameI've kept the new service-prefixed property names consistent with the existing
glue.*/dynamodb.*→client.*convention, but I'm happy to adjust the naming (e.g. unified-only) to match maintainer preference — feedback welcome.Are these changes tested?
Yes. New
moto-based tests (mock STS + Glue + DynamoDB) intests/catalog/test_glue.pyandtests/catalog/test_dynamodb.pycover:client.role-arn/client.role-session-name;glue.role-arn/dynamodb.role-arn) take precedence over the unified ones;Are there any user-facing changes?
Yes — new opt-in configuration properties for assuming an IAM role when connecting to the Glue/DynamoDB catalogs:
client.role-arn/client.role-session-name(now honored by the catalogs too), plus service-specificglue.role-arn/glue.role-session-nameanddynamodb.role-arn/dynamodb.role-session-name. Documented inmkdocs/docs/configuration.md. No change to default behavior.