Repository navigation
Milestone 7: advance the pin to DuckDB v2.0.0 - #6
Merged
Merged
Conversation
…list The v2.0.0 grammar adds bare_label_keyword.list (BareLabelKeyword) and a BareColLabel identifier position whose mask admits only bare-label keywords. Anchor the .gitignore CLI pattern so it no longer swallows new files under internal/grammar/duckdb/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
unreserved_keyword.list gained DESTROY, REGISTER, RESOURCES and SECURE, and the new bare_label_keyword.list classifies 358 of the unreserved and column-name keywords as usable for a column alias without AS. Pin the 39 that are not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Port of tokenizer/base_tokenizer.cpp and parser_tokenizer.cpp at v2.0.0:
- backtick-quoted identifiers (IsQuotedIdentifierDelimiter), closed and
escaped by their own delimiter; `` is a zero-length identifier
- ':' is a single-byte operator; IsSpecialOperator is replaced by
IsCompoundColonToken ("::", ":="), which defers to the operator state
when an operator character follows. ->, ->>, ** and // are now plain
operator-state tokens (with true offsets)
- '-' is no longer a single-byte or control-flow character; the operator
state continues only over the PostgreSQL operator set and ends at a
comment start (-- or /*)
- PushOperatorToken: the trimming rule also strips a trailing '-', never
below one character, and applies to an operator at end of input
- digit separators must sit between two digits
- dollar-quoted strings keep their raw text and source span
- an unterminated block comment is a syntax error
- tokens pushed through PushToken record preceded_by_newline and
preceded_by_block_comment, the basis for string literal continuation
debug-parse -tokens prints the two new flags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Grammar loading (peg_parser.cpp PEGExpressionParser): rules now carry an
expression tree built with textbook PEG precedence (postfix > sequence >
choice). The pre-v2.0.0 matcher factory bound '/' to the immediately
preceding element only and applied '?' '*' '+' to the last list element;
upstream replaced that interpretation, so the factory now maps the tree
one-to-one onto matchers (MatcherFactory::CreateMatcher(expression)).
Negative lookahead is still parsed and ignored. Over the vendored grammar
the only compiled-shape difference is DoubleType <- ('DOUBLE' 'PRECISION'),
whose top-level group is now the rule's own list instead of a nested one.
EndOfInput is no longer added as an empty grammar rule (CreateDefault no
longer does; CheckReference exempts it), so its result is unnamed.
Matchers (matcher/*.hpp, operator_matcher.cpp, compiled_grammar.cpp):
- IdentifierMatcher: backtick-quoted identifiers, unquoted with their own
delimiter; a token ending in ' (a prefixed literal like x'41') is never
an identifier
- StringLiteralMatcher: dollar-quoted literals are stripped here; a
single-quoted literal absorbs plain '...' literals on later lines unless
a block comment intervenes, its span covering every part
- NumberLiteralMatcher: at most one dot, none in the exponent
- OperatorMatcher: GENERIC_PRECEDENCE now only skips the operators of
HasSpecialPrecedence (so single-character + - * / etc. match), and the
new AnyOperatorLiteral override uses ALL_OPERATORS
- the packrat memoized rule list follows upstream's new generated table
The FIRST-set pruning, LiteralChoiceMatcher dispatch and collapsible
rules are optimizations that do not change what matches, the error
position, or (for collapsing) what the transformer computes; they are not
ported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
…mmar With DARKWING_DUCKDB_SRC pointing at a checkout of the pinned commit, the test decodes INLINED_PEG_GRAMMAR from inlined_grammar.hpp (what ParsedGrammar::CreateDefault parses) and compares it line by line with AssembledText, ignoring blank lines and case. It passes at v2.0.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
TokenizerBehavior::OnLastToken emplaces the last token instead of going through PushToken, so it never carries preceded_by_newline or preceded_by_block_comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
v2.0.0's CLI switches to an agent-oriented output format when it detects a coding agent in the environment, which changes the error lines the oracle classifies. DUCKDB_AGENT_MODE=0 overrides the detection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
The v2.0.0 CLI opens with active_grammar_extensions=['shell'], which re-adds bare expression statements that the library grammar rejects. darkwing models the library parser, so the oracle must too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
…iedName v2.0.0 wraps the qualified alternatives in QualifiedTableName and repeats ReservedSchemaQualification (+). Upstream keeps the whole path; darkwing's three-part names keep the component before the name, which is what QualifiedName::Schema() (and the serialization) reports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Grammar shapes: ColumnDefinition takes a plain Identifier and NameableColumnConstraint (names parsed, not stored), KeyActions in either order and optional, GeneratedAlways/GeneratedByDefault, CreateSecure, CreateIndexStmt without the column list, RelOptionName as DottedColLabel, WithOptionList on CREATE SCHEMA, ENUM array bounds, AlterSchemaOptions, DropConstraint, RENAME TO ColIdOrString, ALTER TYPE ... COLLATE, DROP of QualifiedTableFunction / nested function names, TargetAlias, TruncateOptions, OverridingClause, UpdateSetClauseList. New transformer Parser Errors (messages verbatim from upstream): empty table / column list, PERSISTENT outside secrets, temporary indexes/triggers, ApplyColumnCollation, multiple COLLATE, default twice, GENERATED BY DEFAULT, compression AUTO, duplicate WITH options, CREATE RECURSIVE VIEW without columns, non-DEFER_BINDING view options, CheckIntegerSequenceOption, ADD COLUMN constraint kinds, OVERRIDING with DEFAULT VALUES, DROP SECRET CASCADE / TEMPORARY+storage. Identifiers in messages follow v2.0's formatting (Identifier args always quoted, SQLIdentifier allowing capitals). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
CopyLegacyOption ([USING] DELIMITERS 'x') leads the option list, IdentifierColId takes any number of suffixes, stdin maps to /dev/stdin, PARTITION_PATH keeps its single expression (BuildGenericCopyOption), GenericCopyOptionList rejects duplicates, and duplicate-option messages quote the option name like upstream's Identifier formatting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
ExtensionRepositoryStatement (CREATE/DROP EXTENSION REPOSITORY), LOAD ... FROM, INSTALL AND LOAD, ATTACH TO / CONNECT TO [NEW TEMPORARY] EXTERNAL RESOURCE (external_resource_source.gram), the ExternalResourceKeyword shapes, CollectGenericOptions, COMMENT ON a string target, USE 'string', ColIdOrString names for PREPARE/EXECUTE/ DEALLOCATE, unrecognized VACUUM options and SET TIME ZONE interval ranges as Parser Errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
describe.gram splits SHOW (settings-first, ShowType SHOW) from DESCRIBE/SUMMARIZE; SHOW_UNQUALIFIED became SHOW_SPECIAL; string targets name the table directly; bare SUMMARIZE, empty-string targets and catalog-qualified SHOW TABLES FROM are Parser Errors. ShowRef now serializes its qualified_name path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Every expected message was checked against the pinned v2.0.0 CLI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
…edType - LIKE: escape on any variation (only ~~/~~* become *_escape), negation per ApplyBetweenInLikeOperator, case-insensitive ESCAPE error - FETCH without a value, FETCH ... OFFSET, ColumnAliases in CTEs - DESCRIBE/SHOW rewritten for transform_describe.cpp (SHOW type, SHOW_SPECIAL, settings-first SHOW name, DESCRIBE 'literal') - NestedType locations, nested type qualification, TIMESTAMP precision errors; ExpressionStatement removed - serializer: literal, type_expr and qualified_name (v2.0 storage) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
56064 cases over 4925 files (54106 must-accept, 1958 must-reject); 11 statements timed out in the oracle and are dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
# Conflicts: # ast/tableref.go # internal/serialize/serialize.go # parser/transform_select.go # parser/transform_tableref.go
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
- transform_depth.go: the generated expression-depth rule table (NESTING/CHAIN weights on the transform stack) plus StackCheck for NOT/~ runs, matching the oracle's 999/1000 boundary - EXCLUDE/REPLACE/RENAME conflict messages format identifiers as upstream does; dotted EXCLUDE names over 4 parts are rejected - PIVOT entry check names the pivot column (ON <col> IN (...)) - TestUnconvertibleNumberLiterals: 2.03.0 is a syntax error in v2.0.0 - drop corpus todo entries that now agree with the oracle and a stale nested_entries key; add expr_v2_test.go Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
- InsertQueryNode writes its v2.0 qualified_name - GROUP BY t.* forces aggregates (ExpressionIsEmptyStar) - ApplyNotKeywords builds the NOT chain in one transform: only the outermost NOT carries a location (oracle-verified) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
TransformExplainStatement runs PivotEntryCheck when the SQL option is set. With bare expression statements gone from the library grammar, "SELCT 1" now fails at SELCT itself (checked against the v2.0.0 CLI with its shell extension off and the v2.0.0 Python library). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Bind-time ParserExceptions (reader argument checks, setting values, make_type, json_deserialize_sql, sampling) are todo; nested query() / getvariable() parses are skip, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Full sweep: 31515 match, 0 mismatch, 103 skipped; 2626 goldens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136J4pq8PUWyZebBdNZX8F5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the pin from the alpha nightly (
8cbdaba6ac,v2.0.0-alpha38195) to thev2.0.0tag (d54cbbe3df). It also rehearses the regeneration workflow (PLAN.md § Milestones, item 7).Gates (all against the v2.0.0 oracle)
TestCorpus: the regenerated corpus has 56,064 cases over 4,925 files (54,106 accept, 1,958 reject; 11 dropped for oracle timeouts). darkwing agrees on every case except post-parse rejects, which are recorded as todo/skip entries the same way as before.TestCorpusErrorMessages: every rejection matches the oracle's first line verbatim.serialize-diff -corpus: 31,515 match, 0 mismatch, 103 skipped. Goldens are regenerated (2,626 statements) andTestSerializeGoldenspasses.go build ./... && go vet ./... && gofmt -l . && go test -race ./...is clean.DARKWING_DUCKDB_SRC=… go test ./internal/grammarchecks darkwing's assembled grammar against upstream'sinlined_grammar.gram.Oracle
bundled_extensions.cmakeandDUCKDB_VERSION=v2.0.0. It reportsv2.0.0 (Cyanoptera) d54cbbe3df, the same commit as the PyPIduckdb==2.0.0wheel.active_grammar_extensions=['shell'], which re-adds bare expression statements (1 + 1). The library grammar rejects them, and so does the Python library. The oracle andserialize-diffnow runSET active_grammar_extensions=[].DUCKDB_AGENT_MODE=0.duckdbsrc.NightlyURLis replaced byReleaseURL, and the difftest lane downloads from it. The lane will fail at download until DuckDB publishes the release assets.Port
bare_label_keyword.listandexternal_resource_source.gram..gitignore'sduckdbpattern was also hiding new files underinternal/grammar/duckdb/; it's now anchored to/duckdb.:,->,**,//, trimming), stricter digit separators, unterminated block comments, string-literal continuation across lines, raw dollar-quoted text./binds one element" quirk is gone, and CLAUDE.md is updated to say so.AnyOperatorLiteraloverrides; the new packrat list.Known gaps (not in this PR)
Parser::StripUnicodeSpaces. It rejects U+200B, U+2060 and U+FEFF, and accepts U+0085, U+1680 and U+2028/2029. The corpus doesn't exercise this. A faithful fix rewrites the query before parsing and shifts error offsets, so it needs a decision.alias: (VALUES …) AS u.Generated by Claude Code