Skip to content

Make python:3.0 the default Python protocol for lf.query. - #780

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_1001607419
Open

copybara-service[bot] wants to merge 1 commit into
mainfrom
test_1001607419

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Make python:3.0 the default Python protocol for lf.query.

Why

lf.query(prompt, schema, protocol='python') resolved to _LfQueryPythonV2,
which renders:

<preamble, including a worked `Answer(final_answer=2)` example>

REQUEST:
  <request, re-indented by two spaces>

OUTPUT PYTHON TYPE:
  <schema, re-indented by two spaces>

OUTPUT PYTHON OBJECT:

Three problems follow from that layout:

  1. It defeats prefix (KV) cache reuse. The per-request REQUEST block is
    emitted before the static schema, so any change to the request
    invalidates every schema token sitting behind it. Prefix caching only hits
    on a byte-identical prefix, so an agent loop re-encodes its schema on every
    turn even though the schema never changes.
  2. It rewrites the caller's prompt. Request and schema are both piped
    through | indent(2, True), so the model never sees the text the caller
    actually wrote. That matters for whitespace-significant payloads such as
    source code, diffs and tables.
  3. It is longer than it needs to be. That same | indent(2, True) adds
    two characters per line of schema framing, on top of a worked example and
    four section headers on every call.

What

  • Adds _LfQueryPythonV3, registered as python:3.0.
  • Switches _DEFAULT_PROTOCOL_VERSIONS['python'] from '2.0' to '3.0', so
    a bare protocol='python' (the default) now resolves to python:3.0.

python:1.0 and python:2.0 are left exactly as they are and remain
selectable via protocol='python:1.0' / protocol='python:2.0'. Any caller
that has tuned prompts or goldens against the old layout can opt out with a
one-word change. test_explicit_version_survives_default_switch guards exactly
that: it asserts an explicitly versioned request still renders byte-for-byte as
before, and that the default no longer matches it.

How

python:3.0 is a single head-first template, used for every schema:

<static preamble>

OUTPUT_TYPE:
<schema, verbatim>

EXAMPLES:              # emitted only when few-shot examples are supplied
<...>

CONTEXT:               # emitted only when the request carries context
<...>

REQUEST:
<request, verbatim>
  • Head-first. Everything that does not vary with the request -- the
    preamble and OUTPUT_TYPE -- is emitted first, and the caller's text is
    last. For a fixed schema the whole prefix up to and including REQUEST: is
    byte-identical from call to call, so the schema becomes prefix-cacheable
    along with the preamble. Under python:2.0 only the static head was
    cacheable -- its 288-character preamble plus the REQUEST: line, 299
    characters in all -- and the schema, which is unbounded and is the part
    that actually costs tokens, sat behind the caller's text and was re-encoded
    on every call.
  • Verbatim. The | indent(2, True) filters are gone; schema and request
    are rendered exactly as supplied.
  • Constant, pinned overhead. In the common no-examples / no-context case
    the template contributes exactly 439 characters of framing, independent of
    the schema and the request, so query_prompt('what is this?', int) is 455
    characters. Both numbers are asserted, not merely stated:
    test_framing_is_constant_in_schema_size pins the 439 and its invariance
    across int, list[str] and a class schema, and test_query_prompt pins
    the 455 next to the exact rendered string.
  • One rubric instead of a worked example. The preamble states both output
    idioms -- a bare literal for primitive and container types, a
    keyword-argument constructor call for class types -- and illustrates them
    with Point, which no user schema is expected to collide with.

Files

  • core/structured/querying.py -- adds _LfQueryPythonV3 and flips the
    python default from 2.0 to 3.0. Those are the only two hunks;
    _LfQueryPythonV1 and _LfQueryPythonV2 are byte-for-byte untouched.
  • core/structured/querying_test.py -- re-goldens the default-protocol
    expectations and adds LfQueryPythonV3Test (9 tests, one per branch of the
    new template, including the per-example context case) plus regression
    guards for the older protocols.
  • core/structured/parsing_test.py, core/eval/base_test.py -- re-goldened.

Several LMSamplingUsage assertions in parsing_test and under core/eval
are really byte-length snapshots of the rendered prompt (the fake LM reports
len(prompt.text) as the prompt token count), so they move with any change to
the default template.

@copybara-service
copybara-service Bot force-pushed the test_1001607419 branch 6 times, most recently from 55f021f to 7ef3161 Compare October 22, 2026 20:58
## Why

`lf.query(prompt, schema, protocol='python')` resolved to `_LfQueryPythonV2`,
which renders:

```
<preamble, including a worked `Answer(final_answer=2)` example>

REQUEST:
  <request, re-indented by two spaces>

OUTPUT PYTHON TYPE:
  <schema, re-indented by two spaces>

OUTPUT PYTHON OBJECT:
```

Three problems follow from that layout:

1.  **It defeats prefix (KV) cache reuse.** The per-request `REQUEST` block is
    emitted *before* the static schema, so any change to the request
    invalidates every schema token sitting behind it. Prefix caching only hits
    on a byte-identical prefix, so an agent loop re-encodes its schema on every
    turn even though the schema never changes.
2.  **It rewrites the caller's prompt.** Request and schema are both piped
    through `| indent(2, True)`, so the model never sees the text the caller
    actually wrote. That matters for whitespace-significant payloads such as
    source code, diffs and tables.
3.  **It is longer than it needs to be.** That same `| indent(2, True)` adds
    two characters per line of schema framing, on top of a worked example and
    four section headers on every call.

## What

*   Adds `_LfQueryPythonV3`, registered as `python:3.0`.
*   Switches `_DEFAULT_PROTOCOL_VERSIONS['python']` from `'2.0'` to `'3.0'`, so
    a bare `protocol='python'` (the default) now resolves to `python:3.0`.

`python:1.0` and `python:2.0` are left exactly as they are and remain
selectable via `protocol='python:1.0'` / `protocol='python:2.0'`. Any caller
that has tuned prompts or goldens against the old layout can opt out with a
one-word change. `test_explicit_version_survives_default_switch` guards exactly
that: it asserts an explicitly versioned request still renders byte-for-byte as
before, and that the default no longer matches it.

## How

`python:3.0` is a single head-first template, used for every schema:

```
<static preamble>

OUTPUT_TYPE:
<schema, verbatim>

EXAMPLES:              # emitted only when few-shot examples are supplied
<...>

CONTEXT:               # emitted only when the request carries context
<...>

REQUEST:
<request, verbatim>
```

*   **Head-first.** Everything that does not vary with the request -- the
    preamble and `OUTPUT_TYPE` -- is emitted first, and the caller's text is
    last. For a fixed schema the whole prefix up to and including `REQUEST:` is
    byte-identical from call to call, so the schema becomes prefix-cacheable
    along with the preamble. Under `python:2.0` only the static head was
    cacheable -- its 288-character preamble plus the `REQUEST:` line, 299
    characters in all -- and the schema, which is unbounded and is the part
    that actually costs tokens, sat behind the caller's text and was re-encoded
    on every call.
*   **Verbatim.** The `| indent(2, True)` filters are gone; schema and request
    are rendered exactly as supplied.
*   **Constant, pinned overhead.** In the common no-examples / no-context case
    the template contributes exactly 439 characters of framing, independent of
    the schema and the request, so `query_prompt('what is this?', int)` is 455
    characters. Both numbers are asserted, not merely stated:
    `test_framing_is_constant_in_schema_size` pins the 439 and its invariance
    across `int`, `list[str]` and a class schema, and `test_query_prompt` pins
    the 455 next to the exact rendered string.
*   **One rubric instead of a worked example.** The preamble states both output
    idioms -- a bare literal for primitive and container types, a
    keyword-argument constructor call for class types -- and illustrates them
    with `Point`, which no user schema is expected to collide with.

## Files

*   `core/structured/querying.py` -- adds `_LfQueryPythonV3` and flips the
    `python` default from `2.0` to `3.0`. Those are the only two hunks;
    `_LfQueryPythonV1` and `_LfQueryPythonV2` are byte-for-byte untouched.
*   `core/structured/querying_test.py` -- re-goldens the default-protocol
    expectations and adds `LfQueryPythonV3Test` (9 tests, one per branch of the
    new template, including the per-example `context` case) plus regression
    guards for the older protocols.
*   `core/structured/parsing_test.py`, `core/eval/base_test.py` -- re-goldened.

Several `LMSamplingUsage` assertions in `parsing_test` and under `core/eval`
are really byte-length snapshots of the rendered prompt (the fake LM reports
`len(prompt.text)` as the prompt token count), so they move with any change to
the default template.

PiperOrigin-RevId: 1001607419
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant