Skip to content

fix: support env-var auth config for REST catalog (JSON string decode + flat properties)#3423

Open
GayathriSrividya wants to merge 3 commits into
apache:mainfrom
GayathriSrividya:fix/issue-3422-rest-auth-env-var-string-decode
Open

fix: support env-var auth config for REST catalog (JSON string decode + flat properties)#3423
GayathriSrividya wants to merge 3 commits into
apache:mainfrom
GayathriSrividya:fix/issue-3422-rest-auth-env-var-string-decode

Conversation

@GayathriSrividya
Copy link
Copy Markdown

@GayathriSrividya GayathriSrividya commented May 28, 2026

Summary

Fixes #3422.

This PR implements the fix designed by @kevinjqliu in the issue comment.

RestCatalog._create_session() expected auth to be a dict, but values loaded from environment variables always arrive as strings. This caused auth initialization to fail silently or with a confusing AttributeError for all pluggable auth types (basic, oauth2, google, entra, custom) when configured via PYICEBERG_CATALOG__<NAME>__AUTH.

Fixes

Implementation follows the tolerant auth parsing order suggested by @kevinjqliu:

1. JSON string decode (primary fix)

If the auth property value is a str, JSON-parse it before processing.
This unblocks:

export PYICEBERG_CATALOG__REST__AUTH='{"type":"oauth2", "oauth2":{"client_id":"id","client_secret":"secret","token_url":"https://auth.example/token"}}'

2. Flat env-var property support (alternative fix)

Supports the canonical PyIceberg env-var style — no JSON blob required, no quoting/escaping issues:

export PYICEBERG_CATALOG__REST__AUTH__TYPE=oauth2
export PYICEBERG_CATALOG__REST__AUTH__OAUTH2__CLIENT_ID=id
export PYICEBERG_CATALOG__REST__AUTH__OAUTH2__CLIENT_SECRET=secret
export PYICEBERG_CATALOG__REST__AUTH__OAUTH2__TOKEN_URL=https://auth.example/token

The env-var parser maps these to flat properties (auth.type, auth.oauth2.client-id, etc.). _create_session now checks for auth.type when the auth dict is absent, builds the config dict from the flat auth.* properties, and converts kebab-case keys to snake_case to match AuthManager constructor parameters.

Tests

Five new regression tests added to tests/catalog/test_rest.py (covering the test cases specified by @kevinjqliu):

Test What it covers
test_rest_catalog_with_basic_auth_as_json_string JSON string → Basic auth
test_rest_catalog_with_oauth2_auth_as_json_string JSON string → OAuth2 auth
test_rest_catalog_with_invalid_json_auth_string Invalid JSON → descriptive ValueError
test_rest_catalog_with_basic_auth_flat_properties Flat auth.* props → Basic auth
test_rest_catalog_with_oauth2_auth_flat_properties Flat auth.* props → OAuth2 auth

All 13 test_rest_catalog_with_* tests pass; the 4 pre-existing failures (test_token_200, test_config_200, test_auth_header, etc.) are unrelated to this change and also fail on main.

Gayathri Srividya Rajavarapu added 3 commits May 28, 2026 08:49
…auth config

Fixes apache#3422.

RestCatalog._create_session() expected auth to be a dict, but environment
variables always produce string values. This caused auth initialization to
fail for all pluggable auth types (basic, oauth2, google, entra, custom)
when configured via PYICEBERG_CATALOG__<NAME>__AUTH.

Two complementary fixes:

1. JSON string decode: if the 'auth' property is a string, JSON-parse it
   before processing. This supports:
     export PYICEBERG_CATALOG__REST__AUTH='{"type":"oauth2",...}'

2. Flat env-var property support: if 'auth' is absent but 'auth.type' is
   present, build the auth config from flat 'auth.*' properties. This is
   the canonical env-var style that avoids JSON-in-env quoting issues:
     export PYICEBERG_CATALOG__REST__AUTH__TYPE=oauth2
     export PYICEBERG_CATALOG__REST__AUTH__OAUTH2__CLIENT_ID=id
     export PYICEBERG_CATALOG__REST__AUTH__OAUTH2__CLIENT_SECRET=secret
     export PYICEBERG_CATALOG__REST__AUTH__OAUTH2__TOKEN_URL=https://...
   Kebab-case keys (e.g. 'client-id') produced by the env-var parser are
   normalised to snake_case ('client_id') to match AuthManager constructor
   parameters.

Regression tests added for both code paths (basic and oauth2) as well as
invalid JSON detection.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: REST catalog auth cannot be configured via environment variables unless auth JSON strings are decoded

1 participant