Skip to content

feat: support ClickHouse COLUMNS(...) APPLY/EXCEPT/REPLACE transformers (#2631) - #2635

Open
fudianchn wants to merge 3 commits into
JSQLParser:masterfrom
fudianchn:feat/clickhouse-columns-apply-2631
Open

fudianchn wants to merge 3 commits into
JSQLParser:masterfrom
fudianchn:feat/clickhouse-columns-apply-2631

Conversation

@fudianchn

@fudianchn fudianchn commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

AI disclosure: this change was prepared with AI coding agents, reviewed and revised line by line by me.

What

Parse ClickHouse COLUMNS('regexp') select items with ordered, repeatable APPLY(...), EXCEPT (...), and REPLACE (...) transformers, including APPLY(x -> round(x, 2)). See the ClickHouse SELECT documentation.

Why

The query from #2631 fails on the lambda arrow. The simpler COLUMNS('m') APPLY(sum) previously parsed as a function with the alias APPLY(sum), hiding the missing transformer support.

How

  1. ColumnsExpression holds the matcher and its ordered ColumnsTransformer list.
  2. Parse the normal select-item expression first, then recognize transformer suffixes on a direct COLUMNS function. Reuse the existing alias-column grammar at every suffix boundary; no token whitelist or arbitrary scan limit is needed. Keep suffix consumption visible to enclosing syntactic lookaheads and preserve AST parent links and token ranges.
  3. Reuse the existing lambda and select-item expression productions for transformer bodies.
  4. Share child-expression collection across visitors. The adapter uses its existing result-collection hook; the deparser dispatches child visitors instead of bypassing them through toString. The new ExpressionVisitor overload has a default implementation.

Root cause

The grammar lacked column-transformer syntax and fell through to aliases. Self-review of the initial patch also found incomplete child traversal and heuristic alias detection that rejected valid array, IN, and CASE lambda bodies. The revised implementation addresses both parser and consumer behavior.

Testing

  • 63 focused tests, including the issue query, transformer ordering, array/IN/CASE bodies, alias AST compatibility, nested SELECT contexts, custom deparser rewrites, visitor results, EXCEPT traversal, and AST parent/range checks.
  • On the original PR head, 5 consumer tests and 9 syntax cases failed before the fixes. Compatibility probes were also run against the PR base and original PR head; regressions found during refactoring received additional guards.
  • JDK 17: ./gradlew check passed (6,824 tests, zero failures/errors, 25 skipped); mvn clean verify passed (6,806 tests, zero failures/errors, 25 skipped). Grammar ambiguity, formatting, Checkstyle, PMD, SpotBugs, and applicable coverage checks passed.
  • Project JMH parseSQLStatements, unchanged performance.sql, version=latest, JDK 17.0.20 on an i9-13900KS host (32 logical CPUs): 10 forks × 10 one-second measurements, after 3 one-second warmups, per run. Interleaved base/branch ×2: base 21.770±0.139 / 21.923±0.128 versus branch 21.864±0.132 / 21.728±0.102 ms/op (JMH 99.9% CI). The pooled difference is -0.23%, within noise; no measurable regression in this benchmark. Both parsers returned all 54 statements before measurement. The comparison uses PR base 537f3a6d and identical benchmark bytecode, not the subsequently modified master harness.

Behavior notes

  • The intentionally ambiguous APPLY(sum) form becomes a transformer rather than an alias.
  • Bare COLUMNS calls remain Function nodes. Bare APPLY/REPLACE aliases, typed alias columns, REPLACE(a) / REPLACE(a, b) alias lists, and EXCEPT set operations retain their existing interpretations. Explicit AS APPLY(sum) remains an alias.
  • Decorated ordinary functions (attributes, chained parameters, ODBC escaping, KEEP, and outside null handling) do not acquire COLUMNS transformers.
  • This is select-item COLUMNS support, not a ClickHouse semantic validator. Asterisk transformers, STRICT modifiers, and regexp EXCEPT are outside this change.

Verification of the original issue

On PR base 537f3a6d, SELECT COLUMNS('^metric_') APPLY(x -> round(x, 2)) FROM metrics fails to parse. With this change it produces a ColumnsExpression with a LambdaExpression and round-trips. The issue does not specify a JSqlParser version; no claim is made about running that unspecified version or a ClickHouse server.

Fixes #2631

…ansformers

Adds a dedicated select-item branch for a COLUMNS('regexp') matcher followed
by one or more transformers, fixing JSQLParser#2631 (APPLY with a lambda such as
x -> round(x, 2) failed to parse; without a lambda, APPLY(...) was silently
mis-parsed as the select item alias).

- new ColumnsExpression / ColumnsTransformer AST nodes with an ordered
  transformer list, since ClickHouse parses transformers in a loop (they
  may repeat and combine in any order)
- semantic lookahead isColumnsExpressionAhead() keeps a plain
  COLUMNS('regexp') on the regular Function path
- expression visitor, adapter, deparser, validator and TablesNamesFinder
  traverse the new node

Signed-off-by: 付典 <fudianchn@gmail.com>
@manticore-projects

Copy link
Copy Markdown
Contributor

But is it turing complete?

Satisfies the PMD exhaustive-switch rule (Codacy '1 new issue') and fails
loudly if a future ColumnsTransformerType constant is added without
updating appendTo; unreachable for the current APPLY/EXCEPT/REPLACE
constants, so behavior is unchanged. Same pattern as JsonFunction's
Unhandled JsonOnResponseBehavior guard.

Signed-off-by: 付典 <fudianchn@gmail.com>
@fudianchn

Copy link
Copy Markdown
Contributor Author

Hahaha, obviously I haven't made the phone call yet.

Reuse alias grammar at each transformer boundary, preserve JJTree parents and lookahead visibility, and route all child expressions through visitor hooks. Add consumer and grammar regression coverage.

Signed-off-by: 付典 <fudianchn@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] JSQLParser : ClickHouse : COLUMNS APPLY lambda not supported in SELECT

2 participants