Repository navigation
Per-catalog auth state is shared: commit_table mutates the session, and cached S3FileSystem instances share a signer #3970
Description
Two places where authentication state configured for one catalog or table persists onto another.
1. commit_table mutates the shared session's headers
headers = self._session.headers # the live session mapping, not a copy
if table_token := table.config.get(TOKEN):
headers[AUTHORIZATION_HEADER] = f"{BEARER_PREFIX} {table_token}"self._session.headers is the session's own mapping, so assigning into it persists the table-scoped token on the session. Every subsequent request from that RestCatalog carries it, including requests for other tables.
2. fsspec-cached S3FileSystem instances share one signer registration
fs = S3FileSystem(**s3_fs_kwargs)
for event_name, event_function in register_events.items():
fs.s3.meta.events.unregister(event_name, unique_id=1925)
fs.s3.meta.events.register_last(event_name, event_function, unique_id=1925)The signer is registered as a botocore event handler after construction, under a fixed unique_id. It is not part of s3_fs_kwargs, and fsspec caches filesystem instances by constructor arguments (skip_instance_cache is not set). Two catalogs whose anon / client_kwargs / config_kwargs match therefore receive the same S3FileSystem object, and the second unregister + register_last replaces the first catalog's signer with its own.
Note the per-thread lru_cache in get_fs is not involved — _thread_locals is an instance attribute, so that cache is already per-FileIO. The sharing comes from fsspec's instance cache.
Issue investigation generated via claude, reviewed by Sung, Kevin, Fokko.
Activity
ghoshp83 commentedon Sep 18, 2026
I have opened #3999 for the first half of this (the
commit_tablesessionmutation), and while testing it two things came up that I did not fold into that
PR.
The table-scoped token never reaches the wire
RestCatalogsetssession.auth = AuthManagerAdapter(self._auth_manager).requestsappliessession.authinSession.prepare_request, after theper-request
headersare merged, so the auth manager'sAuthorizationreplaceswhatever
commit_tableput there.Against a
requests_mocked catalog withtoken=<catalog token>and a tablewhose config carries
{"token": "<table token>"}, the commit request goes outas:
AuthorizationAuthorizationafter commitBearer <catalog token>Bearer <table token>Bearer <catalog token>So the header assignment currently has no effect on the request, and its only
observable effect is the leak onto the session. Removing the leak does not make
the table token apply.
Making it apply is a design question rather than a fix, which is why I left it
out: it means deciding whether a table-scoped token should override the
configured auth manager, and if so whether that belongs in the auth manager
itself (a per-request override) or in a request-level
auth=argument at thecall site. Happy to implement whichever you prefer, or to leave it if the
intended behaviour is that the auth manager always wins — in which case the
assignment looks like dead code worth deleting.
The
S3FileSystemsigner sharingI have not touched this one. The obvious fix, passing
skip_instance_cache=Truewhen a signer is registered, would give every signed catalog its own filesystem
instance and so give up fsspec's instance caching for those catalogs. That is a
performance and connection-pooling trade-off I did not want to make on your
behalf. If you would rather keep the cache, the alternative is to make the
signer part of the cache key instead — for example by deriving the
unique_idfrom the catalog's identity rather than the fixed
1925, so two catalogs do notoverwrite each other's registration.
Which of those would you like? I am happy to open a PR for either.