[opt](lance) Cache table access resolution in FE

[Gabriel39]

What problem does this PR solve?

Repeated Lance metadata reads call describeTable before opening the Dataset, even when the catalog already shares a Lance metadata Session. This repeats filesystem discovery or REST requests and serializes callers on the namespace lock during query planning.

Cache immutable table URIs and normalized access options for both filesystem and REST catalogs. Cache hits bypass the namespace lock, and concurrent misses for the same table share one load. Dataset opens and snapshot selection still run for every read.

• Add lance.table_access_cache_ttl_seconds (default 60; 0 disables caching), with at most 10,000 entries per catalog client generation. Reads do not extend the TTL.
• Resolve responses containing vended storage options on every read, even when expires_at_millis is present: the BE cannot renew credentials during an arbitrarily long scan. Also bypass caching for credential-bearing or unclassified URIs, including userinfo, query parameters, and fragments. Plain filesystem and REST responses without vended options remain cacheable.
• Explicit table/database refresh, catalog invalidation, and namespace removal retire the access cache, including in-flight loads. Refresh replay invalidates before cache-only object lookup, including when local database/table objects are absent. Routine database-object eviction preserves access entries. Table/database refresh conservatively clears all access entries because Doris refresh names may be mapped names; it does not rotate the native Session.
• Keep index inspection/admission and current index-job locator validation on an uncached path so they verify the current target.

Release note

Reduce repeated Lance query-planning work by caching table access resolution for filesystem and REST catalogs, with credential-safe cache eligibility and explicit refresh invalidation.

Validation

• Added regression tests that first failed on the original implementation: two reads caused two describeTable calls for both filesystem and REST catalogs.
• Added regressions reproduced five failures before the review fixes: credential reuse, signed URIs, routine database-object eviction, and refresh replay with missing table/database objects.
• 66 FE tests passed (0 failures/errors/skips): access-cache expiry, credential handling, concurrent loading/invalidation, catalog lifecycle, property validation, filesystem/REST catalogs, and metadata-cache routing.
• FE Checkstyle passed with zero violations.
• No end-to-end latency benchmark was run.

Check List (For Author)

• Test
  • Unit Test
• Behavior changed:
  • Yes. Table URI/access-option changes can remain cached until TTL expiry or explicit refresh; Dataset versions are not cached here. Set lance.table_access_cache_ttl_seconds=0 to retain per-read resolution.
• Does this need documentation?
  • Yes. English and Chinese documentation: [docs](lance) Document FE table access cache doris-website#4159 (draft pending this implementation). Covers the TTL property, credential exclusions, refresh behavior, and FE/BE cache separation.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Static review result: request changes.

I found four issues:

• cached vended credentials can be handed to an unbounded BE scan with only the fixed reserve remaining;
• signed credential-bearing URIs bypass expiry classification;
• follower REFRESH TABLE replay can miss access-cache invalidation after table-object eviction;
• routine DB-object eviction can repeatedly flush the entire access cache.

Critical checkpoints:

• Correctness/security: blocked by the two credential-lifetime bugs and stale follower replay.
• Lifecycle/concurrency: live/event refresh, generation-swap fencing, catalog ALTER/close/session replacement, and uncached index-sensitive paths otherwise look sound.
• Configuration/compatibility: default/zero/negative TTL behavior and immutable cache key/value handling look consistent.
• Tests: added unit coverage is useful but lacks regressions for the four issues. Per the review contract I did not run builds or tests; the author's reported 60 FE tests and Checkstyle are not independently validated.
• Release note/documentation: the release note is clear. The author marks documentation as required, but no user-facing Lance catalog documentation is in the changed-file set; ensure the property and its refresh/credential semantics are documented before merge.
• User focus: no additional focus was provided.

[github-actions]

[P2] Do not flush access entries for routine DB-object eviction

invalidateDb is also called automatically when the catalog's local database-object cache evicts or expires any entry: its removal listener invokes resetMetaToUninitialized, which reaches this method without checking the removal cause. This new call then rotates the entire 10,000-entry Lance access cache. With the default 1,000 database-object limit, traversing a larger or churn-heavy catalog makes each ordinary eviction discard unrelated hot table-access entries and defeats the namespace-I/O reduction this PR adds. Please separate semantic refresh invalidation from local object-cache cleanup, or target the evicted remote namespace instead of clearing the catalog-wide generation.

[Gabriel39]

Fixed in 46d4b02. Removed access-cache invalidation from generic invalidateDb; explicit database refresh/replay and namespace removal now invalidate through semantic lifecycle paths. The regression warms access, resets an unrelated database object, and verifies that the next read reuses the entry. Explicit refresh and namespace-removal tests verify that those operations still force resolution.

[github-actions]

[P1] Invalidate replayed refreshes without requiring a cached table object

This hook is reached only after replayRefreshTable finds an ExternalTable, but replay deliberately performs a cache-only lookup and returns when that object was evicted. The table-object cache is bounded at 1,000 entries and has no removal listener, while this independent access cache holds 10,000 entries, so a follower can retain the pre-refresh URI/options, skip invalidation, then rebuild the table and hit that stale entry after the primary's REFRESH TABLE. Please retire Lance access by catalog/log identity before the replay early return (catalog-wide clearing is already intentional for name mapping), and add a replay test with the table object absent but access still cached.

[Gabriel39]

Fixed in 46d4b02. Refresh replay now invalidates Lance access by catalog identity before looking up cached database/table objects. Coverage includes absent tables, absent databases, both name-based and legacy ID-based table logs, and database refresh replay with an absent database. Invalidation does not initialize the catalog client.

[github-actions]

[P1] Treat signed table URIs as expiring credentials

This early return assumes that empty storage_options means the access is credential-free, but the namespace URI is retained verbatim and can itself carry presigned/SAS credentials in userinfo or query parameters (the existing LanceIndexDatasetLocator explicitly recognizes those forms). Such a response has no option-map expires_at_millis, so it receives the full configured cache TTL and may be reused after the URI signature expires. Please classify the complete access response: disable caching for credential-bearing URIs unless a trustworthy deadline is available, or bind their TTL to that deadline, and cover the signed-URI/no-options case.

[Gabriel39]

Fixed in 46d4b02. Cache eligibility now checks the URI as well as storage_options. URI userinfo (including registry-based authorities), query parameters, fragments, opaque URIs, and unparsable locators bypass caching. The regression covers signed S3/SAS URIs and URI credentials with no storage_options.

[github-actions]

[P1] Do not reuse credentials that may expire during the scan

The 30-second reserve only guarantees that these cached credentials survive planning and dispatch, but the option map is sent unchanged to the BE and the Lance scanner keeps using that same object-store client for every batch; there is no credential refresh path. A cache hit near this deadline can therefore launch a scan with only about 30 seconds remaining and fail mid-query, even though a fresh describeTable at that time would vend renewed credentials. Because query duration is unbounded, a fixed margin cannot make credential caching safe. Please cache only the stable target while resolving vended credentials per read, or add end-to-end credential renewal/lease semantics.

[Gabriel39]

Fixed in 46d4b02. Removed the fixed 30-second reserve and disabled access-result caching whenever describeTable supplies nonempty storage_options, even with a reported expiry. Each read requests fresh vended access instead of launching a scan with credentials aged by this cache. Plain REST responses remain cacheable. The regression verifies repeated resolution despite a future expiry. Credential renewal during an ongoing scan is outside this change.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Static review result: no new blocking issues on 46d4b022f0e3697c78f58afe148d7358f8021992. Approval opinion: the current head is ready from a static-review perspective.

Critical checkpoints:

• Access-cache correctness and credentials: responses with any vended storage options are resolved for every metadata read, and opaque, signed/query-bearing, fragmented, userinfo-bearing, authority-@, or unparsable locators receive zero cache lifetime. The immutable URI/option snapshot passed to the FE metadata reader is the same snapshot serialized to the BE.
• Refresh and replay lifecycle: explicit table/database refresh clears access by catalog identity; replay does so before cache-only database/table lookups and their missing-object exits. Routine bounded database-object eviction no longer flushes the access cache. Catalog refresh/ALTER/close retire the owning client generation.
• Concurrency and resources: the volatile cache-generation swap prevents in-flight loads from repopulating the generation seen by later reads, while catalog-client leases keep Namespace, Session, and allocator resources alive for admitted operations. Index inspection/admission and current index-job locator validation bypass the query access cache.
• Configuration and compatibility: the new TTL is nonnegative, fixed rather than sliding, bounded to 10,000 entries, and 0 preserves per-read resolution. Full hierarchical table IDs are immutable cache keys.
• Existing inline concerns: the current diff addresses the prior routine-eviction, replay-with-missing-objects, signed-URI, and expiring-vended-credential comments. No substantially similar comment is repeated here.
• Coverage: all nine authoritative changed paths, their relevant FE-to-BE handoff, refresh/replay/ALTER/close call chains, and the changed tests were reviewed. There was no additional user-provided focus point.
• Validation limit: this was static review only; the runner instructions prohibited builds and tests. The reported 66 FE tests and Checkstyle result are author/CI claims, not independent validation here. No end-to-end latency benchmark was reported, and the documentation change remains in the separately linked draft PR.

No inline comments are proposed.

1