Skip to content

Commit 3ee93ae

Browse files
kyleconroyclaude
kyleconroy
and
claude
committed
Add goldeneye, a module that generates and checks the dialect seeds
Move sqlc-pg-gen and sqlc-duckdb-gen from internal/tools into a nested module, internal/goldeneye, with one package per engine and a check that the committed files under internal/engine/<engine>/dialect match what the database reports byte for byte. `go test ./...` in the module runs the checks, skipping engines whose database is not available, and `go run ./cmd/goldeneye generate [engine]` rewrites the files. ClickHouse gets its own package, built on the binary installer and the `clickhouse local` runner from the ClickHouse testcheck work: types.jsonl is generated from system.data_type_families of the pinned release, carrying the spellings ClickHouse aliases to each type, and is regenerated here. functions.jsonl stays hand-written, since ClickHouse publishes no function signatures. The PostgreSQL generator now leaves out of pg_catalog's list the functions that one of the extensions it describes put there, which adminpack does, so the output no longer depends on whether a previous run had already created the extension; the six adminpack lines leave functions.jsonl and stay in the extension's own directory. It also refuses a server of another major release than the one the dialect is pinned to. CI gets a job that runs the checks against PostgreSQL 16 and the pinned ClickHouse, and the manual generation workflow runs goldeneye instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrrewZQhCf82hp68FWNFLj
1 parent 7bcafb7 commit 3ee93ae

26 files changed

Lines changed: 1469 additions & 311 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,36 @@ jobs:
6666
MYSQL_SERVER_URI: "root:mysecretpassword@tcp(127.0.0.1:3306)/mysql?multiStatements=true&parseTime=true"
6767
CGO_ENABLED: "0"
6868

69+
goldeneye:
70+
name: check dialects
71+
runs-on: ubuntu-24.04
72+
services:
73+
postgres:
74+
image: postgres:16
75+
env:
76+
POSTGRES_USER: postgres
77+
POSTGRES_PASSWORD: postgres
78+
POSTGRES_DB: postgres
79+
ports:
80+
- 5432:5432
81+
# needed because the postgres container does not provide a healthcheck
82+
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
83+
steps:
84+
- uses: actions/checkout@v7
85+
- uses: actions/setup-go@v7
86+
with:
87+
go-version: '1.26.5'
88+
89+
- name: install clickhouse
90+
run: go run ./cmd/goldeneye install clickhouse
91+
working-directory: internal/goldeneye
92+
93+
- name: test internal/goldeneye
94+
run: go test ./...
95+
working-directory: internal/goldeneye
96+
env:
97+
POSTGRESQL_SERVER_URI: postgres://postgres:postgres@localhost:${{ job.services.postgres.ports['5432'] }}/postgres?sslmode=disable
98+
6999
vuln_check:
70100
runs-on: ubuntu-24.04
71101
timeout-minutes: 5

‎.github/workflows/gen.yml‎

Lines changed: 13 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
1-
name: sqlc-pg-gen
1+
name: goldeneye
22
on:
33
workflow_dispatch:
44
jobs:
5-
gen:
6-
name: sqlc-pg-gen
7-
runs-on: ubuntu-22.04
5+
generate:
6+
name: generate dialects
7+
runs-on: ubuntu-24.04
88
services:
99
postgres:
10-
image: postgres:15.0-alpine
10+
image: postgres:16
1111
env:
1212
POSTGRES_USER: postgres
1313
POSTGRES_PASSWORD: postgres
@@ -20,20 +20,16 @@ jobs:
2020
- uses: actions/checkout@v7
2121
- uses: actions/setup-go@v7
2222
with:
23-
go-version-file: go.mod
23+
go-version-file: internal/goldeneye/go.mod
2424
check-latest: true
25-
- run: go build -o sqlc-pg-gen ./internal/tools/sqlc-pg-gen
26-
- run: mkdir -p gen/contrib
27-
- run: ./sqlc-pg-gen gen
25+
- run: go run ./cmd/goldeneye install clickhouse
26+
working-directory: internal/goldeneye
27+
- run: go run ./cmd/goldeneye generate
28+
working-directory: internal/goldeneye
2829
env:
29-
PG_USER: postgres
30-
PG_HOST: localhost
31-
PG_DATABASE: postgres
32-
PG_PASSWORD: postgres
33-
PG_PORT: ${{ job.services.postgres.ports['5432'] }}
30+
POSTGRESQL_SERVER_URI: postgres://postgres:postgres@localhost:${{ job.services.postgres.ports['5432'] }}/postgres?sslmode=disable
3431
- name: Save results
3532
uses: actions/upload-artifact@v7
3633
with:
37-
name: sqlc-pg-gen-results
38-
path: gen
39-
34+
name: dialects
35+
path: internal/engine/*/dialect

‎CLAUDE.md‎

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,20 @@ SQLC_TEST_CORE=1 go test ./internal/endtoend -run 'TestReplay/core'
143143
Go aborts a test binary on panic, so a case that panics the core analyzer ends
144144
the run early. Run a subset to get past one (`-run 'TestReplay/core/^select'`).
145145

146+
### Dialect Checks
147+
148+
The dialect seeds under `/internal/engine/<engine>/dialect/` are generated
149+
from a live database by `/internal/goldeneye`, a nested module, and its tests
150+
verify the committed files against one byte for byte. Engines whose database
151+
is not available skip.
152+
153+
```bash
154+
cd internal/goldeneye
155+
go run ./cmd/goldeneye install clickhouse # download the pinned clickhouse binary once
156+
POSTGRESQL_SERVER_URI="postgres://postgres:postgres@127.0.0.1:5432/postgres?sslmode=disable" go test ./...
157+
go run ./cmd/goldeneye generate postgresql # rewrite the files after a change
158+
```
159+
146160
### Example Tests
147161

148162
- **Location:** `/examples/` directory
@@ -216,10 +230,13 @@ MYSQL_SERVER_URI="root:mysecretpassword@tcp(127.0.0.1:3306)/mysql?multiStatement
216230
- `/dolphin/` - MySQL parser (uses TiDB parser)
217231
- `/sqlite/` - SQLite parser
218232
- `/duckdb/` - DuckDB 2.0 parser (uses darkwing, the pure Go port of
219-
DuckDB's PEG parser); its dialect seeds are generated by
220-
`/internal/tools/sqlc-duckdb-gen` from a live DuckDB CLI
233+
DuckDB's PEG parser)
221234
- `<engine>/dialect/` - The engine's type system and standard library, as
222-
JSONL read by `/internal/core/seed`
235+
JSONL read by `/internal/core/seed`; the generated parts come from
236+
`/internal/goldeneye`
237+
- `/internal/goldeneye/` - Nested module that generates the dialect seeds
238+
under `/internal/engine/<engine>/dialect/` from a live database and checks
239+
the committed ones against it, one package per engine; see its README
223240
- `/internal/core/` - The analysis core: catalog, analyzer and dialect seeds
224241
- `/internal/compiler/` - Query compilation logic
225242
- `/internal/codegen/` - Code generation for different languages

‎Makefile‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.PHONY: build build-endtoend test test-ci test-examples test-endtoend start psql mysqlsh proto
1+
.PHONY: build build-endtoend test test-ci test-examples test-endtoend test-goldeneye start psql mysqlsh proto
22

33
build:
44
go build ./...
@@ -26,8 +26,11 @@ test-ci: test-examples build-endtoend vet
2626
sqlc-dev:
2727
go build -o ~/bin/sqlc-dev ./cmd/sqlc/
2828

29-
sqlc-pg-gen:
30-
go build -o ~/bin/sqlc-pg-gen ./internal/tools/sqlc-pg-gen
29+
goldeneye:
30+
cd ./internal/goldeneye && go build -o ~/bin/goldeneye ./cmd/goldeneye
31+
32+
test-goldeneye:
33+
cd ./internal/goldeneye && go test ./...
3134

3235
sqlc-gen-json:
3336
go build -o ~/bin/sqlc-gen-json ./cmd/sqlc-gen-json

‎internal/engine/clickhouse/dialect/types.jsonl‎

Lines changed: 66 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,66 @@
1-
{"name": "UInt8", "category": "N"}
2-
{"name": "UInt16", "category": "N"}
3-
{"name": "UInt32", "category": "N"}
4-
{"name": "UInt64", "category": "N"}
5-
{"name": "UInt128", "category": "N"}
6-
{"name": "UInt256", "category": "N"}
7-
{"name": "Int8", "category": "N"}
8-
{"name": "Int16", "category": "N"}
9-
{"name": "Int32", "category": "N"}
10-
{"name": "Int64", "category": "N"}
11-
{"name": "Int128", "category": "N"}
12-
{"name": "Int256", "category": "N"}
13-
{"name": "Float32", "category": "N"}
14-
{"name": "Float64", "category": "N"}
15-
{"name": "BFloat16", "category": "N"}
16-
{"name": "Decimal", "category": "N"}
17-
{"name": "Decimal32", "category": "N"}
18-
{"name": "Decimal64", "category": "N"}
19-
{"name": "Decimal128", "category": "N"}
20-
{"name": "Decimal256", "category": "N"}
21-
{"name": "Bool", "category": "B"}
22-
{"name": "String", "category": "S"}
23-
{"name": "FixedString", "category": "S"}
24-
{"name": "UUID", "category": "S"}
25-
{"name": "Date", "category": "D"}
26-
{"name": "Date32", "category": "D"}
27-
{"name": "DateTime", "category": "D"}
28-
{"name": "DateTime64", "category": "D"}
29-
{"name": "IPv4", "category": "S"}
30-
{"name": "IPv6", "category": "S"}
31-
{"name": "JSON", "category": "U"}
32-
{"name": "Enum8", "category": "U"}
33-
{"name": "Enum16", "category": "U"}
34-
{"name": "Nullable", "category": "U"}
35-
{"name": "LowCardinality", "category": "U"}
36-
{"name": "Array", "category": "A"}
37-
{"name": "Map", "category": "U"}
38-
{"name": "Tuple", "category": "U"}
39-
{"name": "Nested", "category": "U"}
40-
{"name": "Nothing", "category": "U"}
1+
{"name":"AggregateFunction","category":"U"}
2+
{"name":"Array","category":"A"}
3+
{"name":"BFloat16","category":"N"}
4+
{"name":"Bool","category":"B","aliases":["boolean"]}
5+
{"name":"Date","category":"D"}
6+
{"name":"Date32","category":"D"}
7+
{"name":"DateTime","category":"D","aliases":["TIMESTAMP"]}
8+
{"name":"DateTime32","category":"D"}
9+
{"name":"DateTime64","category":"D"}
10+
{"name":"Decimal","category":"N","aliases":["DEC","FIXED","NUMERIC"]}
11+
{"name":"Decimal128","category":"N"}
12+
{"name":"Decimal256","category":"N"}
13+
{"name":"Decimal32","category":"N"}
14+
{"name":"Decimal64","category":"N"}
15+
{"name":"Dynamic","category":"U"}
16+
{"name":"Enum","category":"U"}
17+
{"name":"Enum16","category":"U"}
18+
{"name":"Enum8","category":"U"}
19+
{"name":"FixedString","category":"S","aliases":["BINARY"]}
20+
{"name":"Float32","category":"N","aliases":["FLOAT","REAL","SINGLE"]}
21+
{"name":"Float64","category":"N","aliases":["DOUBLE","DOUBLE PRECISION"]}
22+
{"name":"IPv4","category":"S","aliases":["INET4"]}
23+
{"name":"IPv6","category":"S","aliases":["INET6"]}
24+
{"name":"Int128","category":"N"}
25+
{"name":"Int16","category":"N","aliases":["SMALLINT","SMALLINT SIGNED"]}
26+
{"name":"Int256","category":"N"}
27+
{"name":"Int32","category":"N","aliases":["INT","INT SIGNED","INTEGER","INTEGER SIGNED","MEDIUMINT","MEDIUMINT SIGNED"]}
28+
{"name":"Int64","category":"N","aliases":["BIGINT","BIGINT SIGNED","SIGNED"]}
29+
{"name":"Int8","category":"N","aliases":["BYTE","INT1","INT1 SIGNED","TINYINT","TINYINT SIGNED"]}
30+
{"name":"IntervalDay","category":"T"}
31+
{"name":"IntervalHour","category":"T"}
32+
{"name":"IntervalMicrosecond","category":"T"}
33+
{"name":"IntervalMillisecond","category":"T"}
34+
{"name":"IntervalMinute","category":"T"}
35+
{"name":"IntervalMonth","category":"T"}
36+
{"name":"IntervalNanosecond","category":"T"}
37+
{"name":"IntervalQuarter","category":"T"}
38+
{"name":"IntervalSecond","category":"T"}
39+
{"name":"IntervalWeek","category":"T"}
40+
{"name":"IntervalYear","category":"T"}
41+
{"name":"JSON","category":"U"}
42+
{"name":"LineString","category":"U"}
43+
{"name":"LowCardinality","category":"U"}
44+
{"name":"Map","category":"U"}
45+
{"name":"MultiLineString","category":"U"}
46+
{"name":"MultiPolygon","category":"U"}
47+
{"name":"Nested","category":"U"}
48+
{"name":"Nothing","category":"U"}
49+
{"name":"Nullable","category":"U"}
50+
{"name":"Object","category":"U"}
51+
{"name":"Point","category":"U"}
52+
{"name":"Polygon","category":"U"}
53+
{"name":"Ring","category":"U"}
54+
{"name":"SimpleAggregateFunction","category":"U"}
55+
{"name":"String","category":"S","aliases":["BINARY LARGE OBJECT","BINARY VARYING","BLOB","BYTEA","CHAR","CHAR LARGE OBJECT","CHAR VARYING","CHARACTER","CHARACTER LARGE OBJECT","CHARACTER VARYING","CLOB","GEOMETRY","LONGBLOB","LONGTEXT","MEDIUMBLOB","MEDIUMTEXT","NATIONAL CHAR","NATIONAL CHAR VARYING","NATIONAL CHARACTER","NATIONAL CHARACTER LARGE OBJECT","NATIONAL CHARACTER VARYING","NCHAR","NCHAR LARGE OBJECT","NCHAR VARYING","NVARCHAR","TEXT","TINYBLOB","TINYTEXT","VARBINARY","VARCHAR","VARCHAR2"]}
56+
{"name":"Time","category":"D"}
57+
{"name":"Time64","category":"D"}
58+
{"name":"Tuple","category":"U"}
59+
{"name":"UInt128","category":"N"}
60+
{"name":"UInt16","category":"N","aliases":["SMALLINT UNSIGNED","YEAR"]}
61+
{"name":"UInt256","category":"N"}
62+
{"name":"UInt32","category":"N","aliases":["INT UNSIGNED","INTEGER UNSIGNED","MEDIUMINT UNSIGNED"]}
63+
{"name":"UInt64","category":"N","aliases":["BIGINT UNSIGNED","BIT","SET","UNSIGNED"]}
64+
{"name":"UInt8","category":"N","aliases":["INT1 UNSIGNED","TINYINT UNSIGNED"]}
65+
{"name":"UUID","category":"S"}
66+
{"name":"Variant","category":"U"}

‎internal/engine/clickhouse/seed.go‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,15 @@ import (
77
"github.com/sqlc-dev/sqlc/internal/core/seed"
88
)
99

10+
// The dialect directory describes ClickHouse's type system. types.jsonl is
11+
// generated from system.data_type_families of the pinned ClickHouse release
12+
// by goldeneye (internal/goldeneye), which also checks it against one;
13+
// dialect.json and functions.jsonl are authored by hand, since ClickHouse
14+
// publishes no function signatures. Regenerate from internal/goldeneye with:
15+
//
16+
// go run ./cmd/goldeneye install clickhouse
17+
// go run ./cmd/goldeneye generate clickhouse
18+
//
1019
//go:embed dialect
1120
var dialectFS embed.FS
1221

‎internal/engine/duckdb/seed.go‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,10 @@ import (
99

1010
// The dialect directory describes DuckDB's type system. types.jsonl,
1111
// functions.jsonl and operators.jsonl are generated from a live DuckDB 2.0
12-
// CLI by sqlc-duckdb-gen (internal/tools/sqlc-duckdb-gen); dialect.json is
13-
// authored by hand. Regenerate with:
12+
// CLI by goldeneye (internal/goldeneye), which also checks them against one;
13+
// dialect.json is authored by hand. Regenerate from internal/goldeneye with:
1414
//
15-
// DUCKDB=/path/to/duckdb go run ./internal/tools/sqlc-duckdb-gen
15+
// DUCKDB=/path/to/duckdb go run ./cmd/goldeneye generate duckdb
1616
//
1717
//go:embed dialect
1818
var dialectFS embed.FS

‎internal/engine/postgresql/catalog.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ func NewCatalog() *catalog.Catalog {
1616
}
1717

1818
// genPGCatalog and genInformationSchema build the schemas sqlc knows
19-
// PostgreSQL by. Both read the dialect directory, which sqlc-pg-gen writes,
19+
// PostgreSQL by. Both read the dialect directory, which goldeneye writes,
2020
// and which the analysis core seeds its catalog from as well.
2121
func genPGCatalog() *catalog.Schema {
2222
return systemSchema("pg_catalog", pgCatalogFuncs())

‎internal/engine/postgresql/dialect/functions.jsonl‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1695,11 +1695,6 @@
16951695
{"name":"pg_export_snapshot","returns":"text"}
16961696
{"name":"pg_extension_config_dump","args":[{"type":"regclass"},{"type":"text"}],"returns":"void"}
16971697
{"name":"pg_extension_update_paths","args":[{"name":"name","type":"name"}],"returns":"record"}
1698-
{"name":"pg_file_rename","args":[{"type":"text"},{"type":"text"}],"returns":"boolean"}
1699-
{"name":"pg_file_rename","args":[{"type":"text"},{"type":"text"},{"type":"text"}],"returns":"boolean"}
1700-
{"name":"pg_file_sync","args":[{"type":"text"}],"returns":"void"}
1701-
{"name":"pg_file_unlink","args":[{"type":"text"}],"returns":"boolean"}
1702-
{"name":"pg_file_write","args":[{"type":"text"},{"type":"text"},{"type":"boolean"}],"returns":"bigint"}
17031698
{"name":"pg_filenode_relation","args":[{"type":"oid"},{"type":"oid"}],"returns":"regclass"}
17041699
{"name":"pg_function_is_visible","args":[{"type":"oid"}],"returns":"boolean"}
17051700
{"name":"pg_get_backend_memory_contexts","returns":"record"}
@@ -1772,7 +1767,6 @@
17721767
{"name":"pg_lock_status","returns":"record"}
17731768
{"name":"pg_log_backend_memory_contexts","args":[{"type":"integer"}],"returns":"boolean"}
17741769
{"name":"pg_log_standby_snapshot","returns":"pg_lsn"}
1775-
{"name":"pg_logdir_ls","returns":"record"}
17761770
{"name":"pg_logical_emit_message","args":[{"type":"boolean"},{"type":"text"},{"type":"bytea"}],"returns":"pg_lsn"}
17771771
{"name":"pg_logical_emit_message","args":[{"type":"boolean"},{"type":"text"},{"type":"text"}],"returns":"pg_lsn"}
17781772
{"name":"pg_logical_slot_get_binary_changes","args":[{"name":"slot_name","type":"name"},{"name":"upto_lsn","type":"pg_lsn"},{"name":"upto_nchanges","type":"integer"},{"name":"options","type":"text[]","mode":"v","has_default":true}],"returns":"record"}

‎internal/engine/postgresql/extension.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ import (
99
)
1010

1111
// loadExtension reads the named extension's function list from the dialect
12-
// directory, where sqlc-pg-gen writes one directory per extension. An
12+
// directory, where goldeneye writes one directory per extension. An
1313
// extension sqlc does not know is nil, which CREATE EXTENSION treats as
1414
// nothing to add.
1515
func loadExtension(name string) *catalog.Schema {

0 commit comments

Comments
 (0)