[docs](lance) Document FE table access cache - #4159
Merged
Gabriel39 merged 1 commit intoSep 21, 2026
Merged
Conversation
7 tasks
Gabriel39
added a commit
to apache/doris
that referenced
this pull request
Sep 21, 2026
### What problem does this PR solve? Repeated Lance metadata reads call `describeTable` before opening the Dataset, even when the catalog already shares a Lance metadata Session. This repeats filesystem discovery or REST requests and serializes callers on the namespace lock during query planning. Cache immutable table URIs and normalized access options for both filesystem and REST catalogs. Cache hits bypass the namespace lock, and concurrent misses for the same table share one load. Dataset opens and snapshot selection still run for every read. - Add `lance.table_access_cache_ttl_seconds` (default `60`; `0` disables caching), with at most 10,000 entries per catalog client generation. Reads do not extend the TTL. - Resolve responses containing vended storage options on every read, even when `expires_at_millis` is present: the BE cannot renew credentials during an arbitrarily long scan. Also bypass caching for credential-bearing or unclassified URIs, including userinfo, query parameters, and fragments. Plain filesystem and REST responses without vended options remain cacheable. - Explicit table/database refresh, catalog invalidation, and namespace removal retire the access cache, including in-flight loads. Refresh replay invalidates before cache-only object lookup, including when local database/table objects are absent. Routine database-object eviction preserves access entries. Table/database refresh conservatively clears all access entries because Doris refresh names may be mapped names; it does not rotate the native Session. - Keep index inspection/admission and current index-job locator validation on an uncached path so they verify the current target. ### Release note Reduce repeated Lance query-planning work by caching table access resolution for filesystem and REST catalogs, with credential-safe cache eligibility and explicit refresh invalidation. ### Validation - Added regression tests that first failed on the original implementation: two reads caused two `describeTable` calls for both filesystem and REST catalogs. - Added regressions reproduced five failures before the review fixes: credential reuse, signed URIs, routine database-object eviction, and refresh replay with missing table/database objects. - 66 FE tests passed (0 failures/errors/skips): access-cache expiry, credential handling, concurrent loading/invalidation, catalog lifecycle, property validation, filesystem/REST catalogs, and metadata-cache routing. - FE Checkstyle passed with zero violations. - No end-to-end latency benchmark was run. ### Check List (For Author) - Test - [x] Unit Test - Behavior changed: - [x] Yes. Table URI/access-option changes can remain cached until TTL expiry or explicit refresh; Dataset versions are not cached here. Set `lance.table_access_cache_ttl_seconds=0` to retain per-read resolution. - Does this need documentation? - [x] Yes. English and Chinese documentation: apache/doris-website#4159 (draft pending this implementation). Covers the TTL property, credential exclusions, refresh behavior, and FE/BE cache separation. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
Gabriel39
marked this pull request as ready for review
September 21, 2026 08:02
yiguolei
pushed a commit
to apache/doris
that referenced
this pull request
Sep 21, 2026
### What problem does this PR solve? Repeated Lance metadata reads call `describeTable` before opening the Dataset, even when the catalog already shares a Lance metadata Session. This repeats filesystem discovery or REST requests and serializes callers on the namespace lock during query planning. Cache immutable table URIs and normalized access options for both filesystem and REST catalogs. Cache hits bypass the namespace lock, and concurrent misses for the same table share one load. Dataset opens and snapshot selection still run for every read. - Add `lance.table_access_cache_ttl_seconds` (default `60`; `0` disables caching), with at most 10,000 entries per catalog client generation. Reads do not extend the TTL. - Resolve responses containing vended storage options on every read, even when `expires_at_millis` is present: the BE cannot renew credentials during an arbitrarily long scan. Also bypass caching for credential-bearing or unclassified URIs, including userinfo, query parameters, and fragments. Plain filesystem and REST responses without vended options remain cacheable. - Explicit table/database refresh, catalog invalidation, and namespace removal retire the access cache, including in-flight loads. Refresh replay invalidates before cache-only object lookup, including when local database/table objects are absent. Routine database-object eviction preserves access entries. Table/database refresh conservatively clears all access entries because Doris refresh names may be mapped names; it does not rotate the native Session. - Keep index inspection/admission and current index-job locator validation on an uncached path so they verify the current target. ### Release note Reduce repeated Lance query-planning work by caching table access resolution for filesystem and REST catalogs, with credential-safe cache eligibility and explicit refresh invalidation. ### Validation - Added regression tests that first failed on the original implementation: two reads caused two `describeTable` calls for both filesystem and REST catalogs. - Added regressions reproduced five failures before the review fixes: credential reuse, signed URIs, routine database-object eviction, and refresh replay with missing table/database objects. - 66 FE tests passed (0 failures/errors/skips): access-cache expiry, credential handling, concurrent loading/invalidation, catalog lifecycle, property validation, filesystem/REST catalogs, and metadata-cache routing. - FE Checkstyle passed with zero violations. - No end-to-end latency benchmark was run. ### Check List (For Author) - Test - [x] Unit Test - Behavior changed: - [x] Yes. Table URI/access-option changes can remain cached until TTL expiry or explicit refresh; Dataset versions are not cached here. Set `lance.table_access_cache_ttl_seconds=0` to retain per-read resolution. - Does this need documentation? - [x] Yes. English and Chinese documentation: apache/doris-website#4159 (draft pending this implementation). Covers the TTL property, credential exclusions, refresh behavior, and FE/BE cache separation. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repeated Lance query planning can resolve the same table location on every read. Document the FE access cache introduced by apache/doris#68305, including its TTL property, credential exclusions, refresh behavior, and separation from Dataset versions and BE caches.
This draft depends on the product PR. Merge after the implementation is accepted.
Self-review: