Skip to content

Commit 7ca32c6

Browse files
timsaucerclaude
andcommitted
docs: unpack the dense passages in the bundle guide
Follow-up to the audience pass, all prose. The paragraph after the two-hook example referred to the hooks as "the first" and "the second", making the reader count back to the code block, and gave its three cases three different shapes. It now names the hooks and groups by where a component goes: codecs and functions in one hook, the planner in the other. "Declare functions rather than registering them yourself inside the hook" named no call, so the practice it warns against was never shown. It now says `register_udf` on the `ctx` you were handed, and contrasts when each is written. Its reason is also corrected: a registration made in the hook was said to be "too early to see the other bundles' codecs", which is true of a table provider and false of a function — the three function getters take no argument at all, so nothing binds them to a codec chain. For functions the reason is the transaction alone, which is what it now says. Table providers are dropped from that paragraph rather than given the codec-visibility caveat they deserve, since the follow-on PR covers them. "Two bundles claiming one name" led with a gerund subject, carried its rationale on a semicolon, and split two qualifications of equal weight across a trailing clause and a paragraph. It now states the rule, shows the error, explains the codec contrast on its own, and lists the two exceptions. One example name runs through all three. The error text quoted in both guides gains the `named` that the message actually contains. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent d47055e commit 7ca32c6

2 files changed

Lines changed: 32 additions & 23 deletions

File tree

docs/source/extension-guide/bundles.md

Lines changed: 31 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -55,22 +55,23 @@ class MyEngineExtension:
5555
return self._make_planner(ctx, fallback=fallback)
5656
```
5757

58-
Implement whichever apply: a codec-only library defines the first, a library
59-
that ships only an optimizing planner defines the second, and a library that
60-
ships only functions defines the first and leaves the codec fields empty. The
61-
caller then writes:
58+
Implement only the hooks you need. Codecs and functions both go in
59+
`__datafusion_session_components__`, with the fields you do not use left empty,
60+
so a codec-only library and a function-only library each define that one alone;
61+
a library shipping nothing but an optimizing planner defines only
62+
`__datafusion_session_planner__`. The caller then writes:
6263

6364
```python
6465
ctx = SessionContext(config).with_extensions(lib_a.Extension(), lib_b.Extension())
6566
ctx.register_table("t", lib_a.TableProvider())
6667
```
6768

68-
Declare functions rather than registering them yourself inside the hook.
69-
Declared components are resolved before anything is written, and they are
70-
registered after every codec is installed; a registration you make during the
71-
hook happens too early to see the other bundles' codecs and is not undone if a
72-
later extension fails. Table providers are still registered by the caller, on
73-
the returned handle — see {ref}`extension_bundles_transaction`.
69+
Return your functions rather than calling `register_udf` on the `ctx` you were
70+
handed. Both put the function on the session, but a registration you make
71+
inside the hook is written the moment it runs — before the other bundles have
72+
been called, and not undone if one of them raises. What you declare is instead
73+
resolved and checked while a failure still costs nothing, then written once
74+
every bundle has succeeded. See {ref}`extension_bundles_transaction`.
7475

7576
`MyPlannerExtension` in [`datafusion-ffi-query-planner-example`] is a complete
7677
Rust implementation of the protocol, including taking the task-context provider
@@ -293,20 +294,28 @@ a capsule.
293294

294295
## Two bundles claiming one name
295296

296-
Within a single call, two extensions declaring a function of the same kind
297-
under the same name is a `ValueError` naming both. Codec ids dispatch on
298-
decode, so a chain can hold many and pick the right one; a function registry
299-
has no such fall-through, and the second registration would silently replace
300-
the first. Names are compared per kind, so a scalar function and an aggregate
301-
may share one.
297+
Two extensions in one call may not declare a function of the same kind under
298+
the same name. Doing so raises:
302299

303-
Shadowing a name the session *already* has is allowed and is not a collision.
304-
The registry holds every DataFusion built-in, and overriding built-ins by name
305-
is a supported thing to do — `enable_spark_functions` is built on it.
300+
```text
301+
ValueError: Two extensions declare a scalar function named 'normalize': ...
302+
```
303+
304+
Codecs get away with sharing a chain because a payload carries the id of the
305+
codec that wrote it, so decode routes to the right one. A function registry has
306+
no such fall-through — one name holds one function — so the second registration
307+
would quietly replace the first. The call refuses instead.
308+
309+
Two cases this does *not* catch:
310+
311+
- **Different kinds never collide.** Names are compared within a kind, so a
312+
scalar function and an aggregate may both be called `normalize`.
313+
- **Shadowing a built-in is allowed.** The registry already holds every
314+
DataFusion function, and replacing one by name is a supported thing to do —
315+
`enable_spark_functions` works that way.
306316

307-
Since the caller cannot repair a collision from their own code, name your
308-
functions so this does not arise: a prefix tying them to your library is the
309-
usual answer.
317+
Your caller cannot rename your function, so stay out of the way: prefix the
318+
names with something tied to your library.
310319

311320
(extension_bundles_transaction)=
312321

docs/source/user-guide/extensions.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@ same kind under the same name, the call raises a `ValueError` naming both,
106106
rather than letting one silently replace the other:
107107

108108
```text
109-
ValueError: Two extensions declare a scalar function 'normalize': ...
109+
ValueError: Two extensions declare a scalar function named 'normalize': ...
110110
```
111111

112112
You cannot rename another library's function from your own code, so the fix is

0 commit comments

Comments
 (0)