fix: give every parsed object a node spanning its source (#2870)

* fix: give arithmetic expressions an AST node

prattArithRest builds operator expressions in plain Java, outside any
production, so JJTree never opened a node for them and `CURRENT_DATE - 7`
or `a || b` had no source range. Each expression the loop builds is now
linked to a node from the token its operand chain starts at to the last
token consumed. The node stays out of the tree, so the operands' nodes
keep their place.

* refactor: let every statement class carry an AST node

The statement classes that implemented Statement alone now extend
ASTNodeAccessImpl, as Select and Import already did, so a parsed
statement can hold the node that records its source range. Nothing
links them yet.

* fix: give every parsed object a node spanning its source

Objects no grammar rule opens a node for, such as a column's table
qualifier, a WHEN clause, nested AND/OR, IS NULL, BETWEEN, parameter
lists, data types, ORDER BY elements and statements, are linked to a
RangeNode: a node outside the parse tree that records the object's first
and last token. getASTRoot() returns the same tree, and no tree node
gains or changes its value.

Ranges that missed text their object prints now cover it: a statement's
WITH, Oracle PRIOR and (+), LIMIT ... BY, sequence options, IF ... ELSE
and merged adjacent literals. A string literal no longer takes the node
of the primary expression around it. Comments get positions, so an
Oracle hint has a range too. getParent() returns null instead of
throwing once it runs out of tree.

ASTNodeCoverageTest parses the test resources' SQL and requires every
ASTNodeAccess object to have a node of at least one character, inside
its holder's range, never partly overlapping another, holding no
parser, and, for expressions and statements, covering text that parses
back to the same object.

* fix: link statements nested in others, and position every comment

Statements that never pass through SingleStatement get their node where
they are built: data-modifying CTEs, statements inside an Oracle block,
CREATE SCHEMA elements, CREATE TABLE ... AS EXECUTE and the GRANT or
REVOKE of ALTER DEFAULT PRIVILEGES. An IF statement's range takes in the
semicolon it prints.

A RangeNode that widens a tree node answers its children, id and dump
too, so a WITH statement's node still shows the tree beneath it.

Comments are positioned in one backwards pass over the recorded match
begins, and comments before the end of input are positioned once
parsing ends, since the end-of-file token gets no token action.

Values in a primary expression, string literals, expression lists and
row patterns get a RangeNode only once something wraps them, so a
production's own tree node is the only node most of them get. On a
20,000-row INSERT ... VALUES this cuts the added allocation from 12.9%
to 3.2%.

The coverage test parses the Oracle tests in the Oracle dialect, checks
that getParent() encloses each range, and runs its checks over the
statements above, which the test corpus does not contain.

* test: split the coverage walk to settle Codacy's PMD findings

`visit` and `checkRange` exceeded PMD's NPath threshold of 200, so the
container walk and the parent check move into methods of their own.
`isHeld` compares with `==` on purpose: a getter that answers a new
object on every call computes its child rather than holding it.

* perf: create the deque of NOT tokens only for a NOT condition

`Condition()` created a deque for the tokens of a leading `NOT` on every
call, though few conditions have one. Each empty `ArrayDeque` allocates
its 16-slot array up front, so parsing `performance.sql` allocated
585 KB more per parse. The deque is now created at the first `NOT`.

* refactor: keep RangeNode internal to the parser package

Only the generated parser creates a `RangeNode`, and callers see it as
the `Node` that `getASTNode()` returns. Keeping the class package-private
leaves its design free to change before a release.

Author: knutwannheden

src/main/java/net/sf/jsqlparser/parser/ASTNodeAccessImpl.java

@@ -71,22 +71,20 @@ public StringBuilder appendTo(StringBuilder builder) {
         return builder;
     }
 
+    /**
+     * The nearest enclosing object in the tree, or null when there is none. A node that carries
+     * only a token range, for an object no grammar rule opens a tree node for, has no parent.
+     */
     public ASTNodeAccess getParent() {
-        Node parent = (Node) node.jjtGetParent();
-        while (parent.jjtGetValue() == null) {
-            parent = (Node) parent.jjtGetParent();
-        }
-
-        return ASTNodeAccess.class.cast(parent.jjtGetValue());
+        return getParent(ASTNodeAccess.class);
     }
 
     public <T extends ASTNodeAccess> T getParent(Class<T> clazz) {
-        Node parent = (Node) node.jjtGetParent();
-        while (parent.jjtGetValue() == null || !clazz.isInstance(parent.jjtGetValue())) {
+        Node parent = node == null ? null : (Node) node.jjtGetParent();
+        while (parent != null && !clazz.isInstance(parent.jjtGetValue())) {
             parent = (Node) parent.jjtGetParent();
         }
-
-        return clazz.cast(parent.jjtGetValue());
+        return parent == null ? null : clazz.cast(parent.jjtGetValue());
     }
 
     @Override

src/main/java/net/sf/jsqlparser/parser/RangeNode.java

@@ -0,0 +1,71 @@
+/*-
+ * #%L
+ * JSQLParser library
+ * %%
+ * Copyright (C) 2004 - 2026 JSQLParser
+ * %%
+ * Dual licensed under GNU LGPL 2.1 or Apache License 2.0
+ * #L%
+ */
+package net.sf.jsqlparser.parser;
+
+/**
+ * A node outside the parse tree that records the tokens an object spans. It serves an object no
+ * grammar rule opens a tree node for, or one whose text runs past its tree node. In that second
+ * case it stands in for the tree node, answering its parent, children and id, with the wider range.
+ */
+final class RangeNode extends Node {
+    private final Node treeNode;
+
+    RangeNode(Token first, Token last, Node treeNode) {
+        super(CCJSqlParserTreeConstants.JJTEXPRESSION);
+        this.treeNode = treeNode;
+        jjtSetFirstToken(first);
+        jjtSetLastToken(last);
+    }
+
+    @Override
+    public Node jjtGetParent() {
+        return treeNode == null ? null : treeNode.jjtGetParent();
+    }
+
+    @Override
+    public Node[] jjtGetChildren() {
+        return treeNode == null ? super.jjtGetChildren() : treeNode.jjtGetChildren();
+    }
+
+    @Override
+    public Node jjtGetChild(int i) {
+        return treeNode == null ? super.jjtGetChild(i) : treeNode.jjtGetChild(i);
+    }
+
+    @Override
+    public int jjtGetNumChildren() {
+        return treeNode == null ? super.jjtGetNumChildren() : treeNode.jjtGetNumChildren();
+    }
+
+    @Override
+    public Object childrenAccept(CCJSqlParserVisitor visitor, Object data) {
+        return treeNode == null ? super.childrenAccept(visitor, data)
+                : treeNode.childrenAccept(visitor, data);
+    }
+
+    @Override
+    public int getId() {
+        return treeNode == null ? super.getId() : treeNode.getId();
+    }
+
+    @Override
+    public String toString() {
+        return treeNode == null ? super.toString() : treeNode.toString();
+    }
+
+    @Override
+    public void dump(String prefix) {
+        if (treeNode == null) {
+            super.dump(prefix);
+        } else {
+            treeNode.dump(prefix);
+        }
+    }
+}

src/main/java/net/sf/jsqlparser/parser/SimpleCharStream.java

@@ -48,6 +48,9 @@ public class SimpleCharStream {
     protected boolean trackLineColumn = true;
     protected int totalCharsRead = 0;
     protected int absoluteTokenBegin = 0;
+    // line, column and absolute position of each match begun since the last positionSpecialTokens()
+    private int[] matchBegins = new int[24];
+    private int matchBeginCount = 0;
     int bufsize;
     int available;
     int tokenBegin;
@@ -107,6 +110,38 @@ public final int getAbsoluteTokenBegin() {
         return absoluteTokenBegin;
     }
 
+    /**
+     * Gives the comments in front of {@code token} the positions where their matches began, from
+     * the begins recorded since the last call, and forgets those. Comments come through no token
+     * action, so this is how they get a position. The comments chain backwards from the token while
+     * the begins are recorded forwards, so one backwards pass pairs them.
+     */
+    public final void positionSpecialTokens(Token token) {
+        int i = matchBeginCount - 3;
+        for (Token comment = token.specialToken; comment != null; comment = comment.specialToken) {
+            while (i >= 0 && (matchBegins[i] != comment.beginLine
+                    || matchBegins[i + 1] != comment.beginColumn)) {
+                i -= 3;
+            }
+            if (i < 0) {
+                break;
+            }
+            comment.absoluteBegin = matchBegins[i + 2];
+            comment.absoluteEnd = comment.absoluteBegin + comment.image.length();
+            i -= 3;
+        }
+        matchBeginCount = 0;
+    }
+
+    private void recordMatchBegin() {
+        if (matchBeginCount + 3 > matchBegins.length) {
+            matchBegins = java.util.Arrays.copyOf(matchBegins, matchBegins.length * 2);
+        }
+        matchBegins[matchBeginCount++] = getBeginLine();
+        matchBegins[matchBeginCount++] = getBeginColumn();
+        matchBegins[matchBeginCount++] = absoluteTokenBegin;
+    }
+
     protected void ExpandBuff(boolean wrapAround) {
         final int newSize = bufsize * 2;
         char[] newbuffer = new char[newSize];
@@ -200,6 +235,7 @@ public final char BeginToken() throws java.io.IOException {
         tokenBegin = bufpos;
 
         absoluteTokenBegin = totalCharsRead;
+        recordMatchBegin();
 
         if (c == '#' && featureConfiguration != null
                 && featureConfiguration.getAsBoolean(Feature.allowHashLineComments)) {
@@ -441,6 +477,7 @@ public final void backup(int amount) {
      * Reinitialise.
      */
     public void ReInit(Provider dstream, int startline, int startcolumn, int buffersize) {
+        matchBeginCount = 0;
         inputStream = dstream;
         line = startline;
         column = startcolumn - 1;

src/main/java/net/sf/jsqlparser/statement/AssertStatement.java

@@ -11,6 +11,7 @@
 
 import net.sf.jsqlparser.expression.Expression;
 import net.sf.jsqlparser.expression.StringValue;
+import net.sf.jsqlparser.parser.ASTNodeAccessImpl;
 
 import java.util.function.Consumer;
 
@@ -22,7 +23,7 @@
  *      "https://cloud.google.com/bigquery/docs/reference/standard-sql/debugging-statements">Debugging
  *      statements</a>
  */
-public class AssertStatement implements Statement {
+public class AssertStatement extends ASTNodeAccessImpl implements Statement {
     private Expression expression;
     private StringValue description;
 

src/main/java/net/sf/jsqlparser/statement/AttachStatement.java

@@ -9,6 +9,7 @@
  */
 package net.sf.jsqlparser.statement;
 
+import net.sf.jsqlparser.parser.ASTNodeAccessImpl;
 import net.sf.jsqlparser.statement.select.PlainSelect;
 
 import java.util.List;
@@ -19,7 +20,7 @@
  *
  * @see <a href="https://duckdb.org/docs/stable/sql/statements/attach">ATTACH</a>
  */
-public class AttachStatement implements Statement {
+public class AttachStatement extends ASTNodeAccessImpl implements Statement {
     private String databasePath;
     private String alias;
     private boolean usingDatabaseKeyword;

src/main/java/net/sf/jsqlparser/statement/Block.java

@@ -10,8 +10,9 @@
 package net.sf.jsqlparser.statement;
 
 import java.util.function.Consumer;
+import net.sf.jsqlparser.parser.ASTNodeAccessImpl;
 
-public class Block implements Statement {
+public class Block extends ASTNodeAccessImpl implements Statement {
     private boolean hasSemicolonAfterEnd = false;
 
     private Statements statements;

src/main/java/net/sf/jsqlparser/statement/Commit.java

@@ -9,7 +9,9 @@
  */
 package net.sf.jsqlparser.statement;
 
-public class Commit implements Statement {
+import net.sf.jsqlparser.parser.ASTNodeAccessImpl;
+
+public class Commit extends ASTNodeAccessImpl implements Statement {
     private TransactionKeyword keyword;
     private TransactionChain chain;
 

src/main/java/net/sf/jsqlparser/statement/ConnectStatement.java

@@ -9,11 +9,13 @@
  */
 package net.sf.jsqlparser.statement;
 
+import net.sf.jsqlparser.parser.ASTNodeAccessImpl;
+
 /**
  * DuckDB's {@code CONNECT target [AS alias]}, which routes the session's queries to a remote
  * database.
  */
-public class ConnectStatement implements Statement {
+public class ConnectStatement extends ASTNodeAccessImpl implements Statement {
     private String target;
     private String alias;
 

src/main/java/net/sf/jsqlparser/statement/CopyStatement.java

@@ -10,6 +10,7 @@
 package net.sf.jsqlparser.statement;
 
 import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
+import net.sf.jsqlparser.parser.ASTNodeAccessImpl;
 import net.sf.jsqlparser.schema.Column;
 import net.sf.jsqlparser.schema.Table;
 import net.sf.jsqlparser.statement.select.PlainSelect;
@@ -25,7 +26,7 @@
  *
  * @see <a href="https://duckdb.org/docs/stable/sql/statements/copy">COPY</a>
  */
-public class CopyStatement implements Statement {
+public class CopyStatement extends ASTNodeAccessImpl implements Statement {
     private Table table;
     private ExpressionList<Column> columns;
     private Select select;

src/main/java/net/sf/jsqlparser/statement/CreateFunctionalStatement.java

@@ -15,13 +15,14 @@
 import java.util.List;
 import java.util.Optional;
 import java.util.function.Consumer;
+import net.sf.jsqlparser.parser.ASTNodeAccessImpl;
 import net.sf.jsqlparser.statement.create.function.FunctionReturnType;
 import net.sf.jsqlparser.statement.create.table.TableElement;
 
 /**
  * A base for the declaration of function like statements
  */
-public abstract class CreateFunctionalStatement implements Statement {
+public abstract class CreateFunctionalStatement extends ASTNodeAccessImpl implements Statement {
 
     private String kind;
     private boolean orReplace = false;