Skip to content

Milestone 7: advance the pin to DuckDB v2.0.0 - #6

Merged
kyleconroy merged 34 commits into
mainfrom
pin-v2.0.0
Oct 22, 2026
Merged

kyleconroy merged 34 commits into
mainfrom
pin-v2.0.0

Conversation

@kyleconroy

Copy link
Copy Markdown
Contributor

Moves the pin from the alpha nightly (8cbdaba6ac, v2.0.0-alpha38195) to the v2.0.0 tag (d54cbbe3df). It also rehearses the regeneration workflow (PLAN.md § Milestones, item 7).

Gates (all against the v2.0.0 oracle)

  • TestCorpus: the regenerated corpus has 56,064 cases over 4,925 files (54,106 accept, 1,958 reject; 11 dropped for oracle timeouts). darkwing agrees on every case except post-parse rejects, which are recorded as todo/skip entries the same way as before.
  • TestCorpusErrorMessages: every rejection matches the oracle's first line verbatim.
  • serialize-diff -corpus: 31,515 match, 0 mismatch, 103 skipped. Goldens are regenerated (2,626 statements) and TestSerializeGoldens passes.
  • Dev loop: go build ./... && go vet ./... && gofmt -l . && go test -race ./... is clean. DARKWING_DUCKDB_SRC=… go test ./internal/grammar checks darkwing's assembled grammar against upstream's inlined_grammar.gram.

Oracle

  • Built from source. No v2.0.0 CLI binaries are published yet. I built the CLI from the tag with bundled_extensions.cmake and DUCKDB_VERSION=v2.0.0. It reports v2.0.0 (Cyanoptera) d54cbbe3df, the same commit as the PyPI duckdb==2.0.0 wheel.
  • Library grammar, not the CLI's. The v2.0.0 CLI opens with active_grammar_extensions=['shell'], which re-adds bare expression statements (1 + 1). The library grammar rejects them, and so does the Python library. The oracle and serialize-diff now run SET active_grammar_extensions=[].
  • Agent mode off. The new CLI switches to an agent output format when it detects a coding agent in the environment, so the oracle sets DUCKDB_AGENT_MODE=0.
  • Release URL. duckdbsrc.NightlyURL is replaced by ReleaseURL, and the difftest lane downloads from it. The lane will fail at download until DuckDB publishes the release assets.

Port

  • Grammar: re-vendored verbatim, including the new bare_label_keyword.list and external_resource_source.gram. .gitignore's duckdb pattern was also hiding new files under internal/grammar/duckdb/; it's now anchored to /duckdb.
  • Engine:
    • Tokenizer: backtick identifiers, operator lexing (:, ->, **, //, trimming), stricter digit separators, unterminated block comments, string-literal continuation across lines, raw dollar-quoted text.
    • Grammar loader: ordinary PEG precedence (upstream's new recursive-descent rule parser). The old "/ binds one element" quirk is gone, and CLAUDE.md is updated to say so.
    • Matchers: identifier, number and operator matchers; bare-label and AnyOperatorLiteral overrides; the new packrat list.
  • Statement transformers: the changes to CREATE/ALTER/DROP, DML, COPY and generic options, DESCRIBE/SHOW and the misc statements. Adds the new extension-repository and external-resource statements, with upstream's new Parser Errors copied verbatim.
  • Query/expression transformers: the new expression precedence (IS/NOT/IN, Infix/CustomPrefix operators), literals, nested qualification paths, window and cast errors, FETCH/LIMIT forms, PIVOT, and an expression-depth limit that matches the oracle's 1000 boundary.

Known gaps (not in this PR)

  • Unicode whitespace (predates this PR): darkwing's set differs from upstream's Parser::StripUnicodeSpaces. It rejects U+200B, U+2060 and U+FEFF, and accepts U+0085, U+1680 and U+2028/2029. The corpus doesn't exercise this. A faithful fix rewrites the query before parsing and shifts error offsets, so it needs a decision.
  • Alias-error rendering: "Table reference … cannot have two aliases" shows source text where upstream shows its re-rendered SQL. They differ only for alias: (VALUES …) AS u.

Generated by Claude Code

claude added 30 commits October 21, 2026 19:50
…list

The v2.0.0 grammar adds bare_label_keyword.list (BareLabelKeyword) and a
BareColLabel identifier position whose mask admits only bare-label
keywords. Anchor the .gitignore CLI pattern so it no longer swallows new
files under internal/grammar/duckdb/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
unreserved_keyword.list gained DESTROY, REGISTER, RESOURCES and SECURE,
and the new bare_label_keyword.list classifies 358 of the unreserved and
column-name keywords as usable for a column alias without AS. Pin the
39 that are not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Port of tokenizer/base_tokenizer.cpp and parser_tokenizer.cpp at v2.0.0:

- backtick-quoted identifiers (IsQuotedIdentifierDelimiter), closed and
  escaped by their own delimiter; `` is a zero-length identifier
- ':' is a single-byte operator; IsSpecialOperator is replaced by
  IsCompoundColonToken ("::", ":="), which defers to the operator state
  when an operator character follows. ->, ->>, ** and // are now plain
  operator-state tokens (with true offsets)
- '-' is no longer a single-byte or control-flow character; the operator
  state continues only over the PostgreSQL operator set and ends at a
  comment start (-- or /*)
- PushOperatorToken: the trimming rule also strips a trailing '-', never
  below one character, and applies to an operator at end of input
- digit separators must sit between two digits
- dollar-quoted strings keep their raw text and source span
- an unterminated block comment is a syntax error
- tokens pushed through PushToken record preceded_by_newline and
  preceded_by_block_comment, the basis for string literal continuation

debug-parse -tokens prints the two new flags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Grammar loading (peg_parser.cpp PEGExpressionParser): rules now carry an
expression tree built with textbook PEG precedence (postfix > sequence >
choice). The pre-v2.0.0 matcher factory bound '/' to the immediately
preceding element only and applied '?' '*' '+' to the last list element;
upstream replaced that interpretation, so the factory now maps the tree
one-to-one onto matchers (MatcherFactory::CreateMatcher(expression)).
Negative lookahead is still parsed and ignored. Over the vendored grammar
the only compiled-shape difference is DoubleType <- ('DOUBLE' 'PRECISION'),
whose top-level group is now the rule's own list instead of a nested one.
EndOfInput is no longer added as an empty grammar rule (CreateDefault no
longer does; CheckReference exempts it), so its result is unnamed.

Matchers (matcher/*.hpp, operator_matcher.cpp, compiled_grammar.cpp):
- IdentifierMatcher: backtick-quoted identifiers, unquoted with their own
  delimiter; a token ending in ' (a prefixed literal like x'41') is never
  an identifier
- StringLiteralMatcher: dollar-quoted literals are stripped here; a
  single-quoted literal absorbs plain '...' literals on later lines unless
  a block comment intervenes, its span covering every part
- NumberLiteralMatcher: at most one dot, none in the exponent
- OperatorMatcher: GENERIC_PRECEDENCE now only skips the operators of
  HasSpecialPrecedence (so single-character + - * / etc. match), and the
  new AnyOperatorLiteral override uses ALL_OPERATORS
- the packrat memoized rule list follows upstream's new generated table

The FIRST-set pruning, LiteralChoiceMatcher dispatch and collapsible
rules are optimizations that do not change what matches, the error
position, or (for collapsing) what the transformer computes; they are not
ported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
…mmar

With DARKWING_DUCKDB_SRC pointing at a checkout of the pinned commit, the
test decodes INLINED_PEG_GRAMMAR from inlined_grammar.hpp (what
ParsedGrammar::CreateDefault parses) and compares it line by line with
AssembledText, ignoring blank lines and case. It passes at v2.0.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
TokenizerBehavior::OnLastToken emplaces the last token instead of going
through PushToken, so it never carries preceded_by_newline or
preceded_by_block_comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
v2.0.0's CLI switches to an agent-oriented output format when it detects
a coding agent in the environment, which changes the error lines the
oracle classifies. DUCKDB_AGENT_MODE=0 overrides the detection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
The v2.0.0 CLI opens with active_grammar_extensions=['shell'], which
re-adds bare expression statements that the library grammar rejects.
darkwing models the library parser, so the oracle must too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
…iedName

v2.0.0 wraps the qualified alternatives in QualifiedTableName and
repeats ReservedSchemaQualification (+). Upstream keeps the whole path;
darkwing's three-part names keep the component before the name, which
is what QualifiedName::Schema() (and the serialization) reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Grammar shapes: ColumnDefinition takes a plain Identifier and
NameableColumnConstraint (names parsed, not stored), KeyActions in
either order and optional, GeneratedAlways/GeneratedByDefault,
CreateSecure, CreateIndexStmt without the column list, RelOptionName as
DottedColLabel, WithOptionList on CREATE SCHEMA, ENUM array bounds,
AlterSchemaOptions, DropConstraint, RENAME TO ColIdOrString, ALTER TYPE
... COLLATE, DROP of QualifiedTableFunction / nested function names,
TargetAlias, TruncateOptions, OverridingClause, UpdateSetClauseList.

New transformer Parser Errors (messages verbatim from upstream):
empty table / column list, PERSISTENT outside secrets, temporary
indexes/triggers, ApplyColumnCollation, multiple COLLATE, default
twice, GENERATED BY DEFAULT, compression AUTO, duplicate WITH options,
CREATE RECURSIVE VIEW without columns, non-DEFER_BINDING view options,
CheckIntegerSequenceOption, ADD COLUMN constraint kinds, OVERRIDING with
DEFAULT VALUES, DROP SECRET CASCADE / TEMPORARY+storage. Identifiers in
messages follow v2.0's formatting (Identifier args always quoted,
SQLIdentifier allowing capitals).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
CopyLegacyOption ([USING] DELIMITERS 'x') leads the option list,
IdentifierColId takes any number of suffixes, stdin maps to /dev/stdin,
PARTITION_PATH keeps its single expression (BuildGenericCopyOption),
GenericCopyOptionList rejects duplicates, and duplicate-option messages
quote the option name like upstream's Identifier formatting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
ExtensionRepositoryStatement (CREATE/DROP EXTENSION REPOSITORY), LOAD
... FROM, INSTALL AND LOAD, ATTACH TO / CONNECT TO [NEW TEMPORARY]
EXTERNAL RESOURCE (external_resource_source.gram), the
ExternalResourceKeyword shapes, CollectGenericOptions, COMMENT ON a
string target, USE 'string', ColIdOrString names for PREPARE/EXECUTE/
DEALLOCATE, unrecognized VACUUM options and SET TIME ZONE interval
ranges as Parser Errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
describe.gram splits SHOW (settings-first, ShowType SHOW) from
DESCRIBE/SUMMARIZE; SHOW_UNQUALIFIED became SHOW_SPECIAL; string targets
name the table directly; bare SUMMARIZE, empty-string targets and
catalog-qualified SHOW TABLES FROM are Parser Errors. ShowRef now
serializes its qualified_name path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Every expected message was checked against the pinned v2.0.0 CLI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
…edType

- LIKE: escape on any variation (only ~~/~~* become *_escape), negation
  per ApplyBetweenInLikeOperator, case-insensitive ESCAPE error
- FETCH without a value, FETCH ... OFFSET, ColumnAliases in CTEs
- DESCRIBE/SHOW rewritten for transform_describe.cpp (SHOW type,
  SHOW_SPECIAL, settings-first SHOW name, DESCRIBE 'literal')
- NestedType locations, nested type qualification, TIMESTAMP precision
  errors; ExpressionStatement removed
- serializer: literal, type_expr and qualified_name (v2.0 storage)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
56064 cases over 4925 files (54106 must-accept, 1958 must-reject); 11
statements timed out in the oracle and are dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
# Conflicts:
#	ast/tableref.go
#	internal/serialize/serialize.go
#	parser/transform_select.go
#	parser/transform_tableref.go
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
- transform_depth.go: the generated expression-depth rule table
  (NESTING/CHAIN weights on the transform stack) plus StackCheck for
  NOT/~ runs, matching the oracle's 999/1000 boundary
- EXCLUDE/REPLACE/RENAME conflict messages format identifiers as
  upstream does; dotted EXCLUDE names over 4 parts are rejected
- PIVOT entry check names the pivot column (ON <col> IN (...))
- TestUnconvertibleNumberLiterals: 2.03.0 is a syntax error in v2.0.0
- drop corpus todo entries that now agree with the oracle and a stale
  nested_entries key; add expr_v2_test.go

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
- InsertQueryNode writes its v2.0 qualified_name
- GROUP BY t.* forces aggregates (ExpressionIsEmptyStar)
- ApplyNotKeywords builds the NOT chain in one transform: only the
  outermost NOT carries a location (oracle-verified)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
TransformExplainStatement runs PivotEntryCheck when the SQL option is
set. With bare expression statements gone from the library grammar,
"SELCT 1" now fails at SELCT itself (checked against the v2.0.0 CLI with
its shell extension off and the v2.0.0 Python library).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Bind-time ParserExceptions (reader argument checks, setting values,
make_type, json_deserialize_sql, sampling) are todo; nested query() /
getvariable() parses are skip, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Full sweep: 31515 match, 0 mismatch, 103 skipped; 2626 goldens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
@kyleconroy
kyleconroy merged commit d88ff19 into main Oct 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants