Skip to content

Python context layer: pin dictionary prose as retrievable chunks - #284

Open
jat255 wants to merge 1 commit into
jat255/context-layer-retrievalfrom
jat255/context-layer-dictionary-chunks
Open

Python context layer: pin dictionary prose as retrievable chunks#284
jat255 wants to merge 1 commit into
jat255/context-layer-retrievalfrom
jat255/context-layer-dictionary-chunks

Conversation

@jat255

@jat255 jat255 commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Third of four PRs for M4, the Python context layer (kata vp7e). Stacked on #241; #285 folds the chunks into an agent's layer on top.

Both packages already turn a dictionary's prose into retrieval chunks, and nothing held them to the same answer. tests/shared/context_layer.json gains a dictionary_context_chunks section and both suites run it against their own reader: dataset details, then tables in declaration order, then glossary terms, then governed definitions, with a table that has no prose contributing nothing and column-level content staying out.

Python needs no new chunking function. The dictionary already owns it as DataDictionary.context_chunks(), which is where it belongs now that it owns all three of its channels. The milestone plan expected a dictionary_context_chunks() in _context_layer.py; writing one would have been a second implementation of the same thing. What was actually missing is the guard for a source carrying no dictionary at all, which is _dictionary_chunks() and is what the next PR needs.

Governed definitions get no fixture case, deliberately. Both packages render the chunk text the same way and definitions.json pins it, but they compile definitions at different constructors, so they disagree on whether a dictionary not yet attached to a source has any. A case here would pin that disagreement rather than the contract.

R changes

One functional change in pkg-r/R/context-layer.R, found by the new fixture rather than by reading: dictionary_context_chunks() returned a named character vector, and now returns an unnamed one.

The names came from vapply(names(dictionary$tables), ...), which labels each table's chunk with its table name. Because c() keeps names once any element has them, the whole returned vector was named, with "" for the dataset details and the glossary entries. Those names then rode into augment_context_layer() and became names on the layer's docs vector. Nothing read them, so nothing behaved differently, but the R side was carrying metadata the Python side has no way to carry, which made the shared fixture unsatisfiable for both suites at once. unname() on the filtered result is the whole change. Chunk text and chunk order are untouched.

Blast radius is two call sites, and neither reads names. augment_context_layer() concatenates the vector into a layer's documents. The one test caller, test-definitions.R:401, uses grepl() and positional subsetting. A named-vector consumer would have had to index by table name, and no code does.

Why it is safe: unname() cannot change length, order, or element text, so the only observable difference is the absence of names(). I checked both callers by reading them rather than assuming, and ran test-context-layer.R (33 passing), test-data-dictionary.R (68 passing, 3 skipped on CRAN), test-definitions.R (114 passing) and test-citations.R (84 passing), which are the files that touch this function or augment_context_layer(). I did not run the full R suite: it currently hangs for about ten minutes on run_r's uncancelled later callbacks, which #268 fixes.

Worth your scrutiny: the new runner in test-context-layer.R builds each case with new_data_dictionary(case$dictionary), so the fixture drives the real reader and not a shape adapter written for the test. That relies on new_data_dictionary() accepting the same raw parsed shape the Python model validates, which it does because both read the authored data-dict.yaml. If that stops being true the runner is where it will show up.

No hand-written R tests were deleted; the new test is additive. pkg-r/tests/testthat/fixtures/shared/context_layer.json is generated by scripts/sync-shared-fixtures.sh and needs no review.

The dictionary already turns its prose into retrieval chunks in both
packages, but nothing held the two to the same answer. Add a
dictionary_context_chunks section to the shared context_layer fixture and
a runner in each suite, driving each package's own reader from the same
parsed shape.

The fixture exposed one divergence: R named its table chunks after their
tables, because vapply() over names() carries them, and those names rode
into the layer's documents. Chunks are text, so drop the names. Neither
call site read them.

Governed definitions carry no case. The two packages compile them at
different constructors, so they disagree on whether a dictionary that is
not attached to a source has any; definitions.json pins the chunk text.

Python needs no new chunking function: the dictionary owns it as
DataDictionary.context_chunks(). What was missing was the guard for a
source with no dictionary at all, which is what _dictionary_chunks()
adds and what folding source prose into a layer needs next.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant