Performance andtlr improvements

[jagalindo]

Summary

Parsing UVL is several times slower than it needs to be because of a single semantic predicate in the NEWLINE lexer rule. This PR removes it from the Python, Java and JavaScript lexers without changing the token stream. Every model lexes to the same tokens, and indented first lines are still rejected.

Problem

All three target lexers define

antlr
NEWLINE: (
{atStartOfInput()}? SPACES
| ( '\r'? '\n' | '\r') SPACES?
) {handleNewline();};

The {atStartOfInput()}? predicate is reachable from the lexer's start state. In that case ANTLR's LexerATNSimulator.matchATN never stores the DFA start state (suppressEdge = s0_closure.hasSemanticContext), so every token recomputes the whole ATN closure instead of reusing the cached DFA.

Measured with the Python target (uvlparser 2.5.0) on a 1.5 MB UVLHub model (BerkeleyDB, 224,698 tokens):

• computeStartState runs 177,973 times, about once per token (it should run once per mode).
• With the start state cached, the lexer misses its DFA cache 146 times in total, and parsing is about 6x faster: 43.0 s down to 7.5 s on that model, and 0.033 s down to 0.005 s on small models. The parse trees are identical.

Java and JavaScript use the same matchATN logic, so they have the same problem.

Change (implemented on a new branch)

• Grammar (uvl/{Python,Java,JavaScript}/UVL*Lexer.g4): NEWLINE loses the predicated alternative and becomes ( '\r'? '\n' | '\r') SPACES?.
• Custom lexers (python/uvl/UVLCustomLexer.py, the Java @members, js/src/UVLJavaScriptCustomLexer.js): a new handleLeadingIndentation() runs once, on the first nextToken() call.
  • It looks at the leading spaces and tabs and the character after them.
  • When the old predicated alternative would have emitted tokens, it queues the same NEWLINE + INDENT pair, with the same offsets, line and column. SKIP_ then consumes the spaces as usual.
  • Each target mirrors its own handleNewline() conditions, which already differ slightly between targets (Java skips only on //, JavaScript does not check /, Python checks / and #), so each target keeps its current behaviour.
• The now-unused atStartOfInput() helpers are removed.

Behaviour

Unchanged, including the rejection of an indented first line:

features
Root

still fails with Line 1:2 - no viable alternative at input ' ', as before.

Tests

• New faulty model test_models/faulty/indented_first_line.uvl. The Python, JavaScript and Java test suites all pick it up and expect a syntax error.
• New Python test test_lexer_caches_its_dfa_start_state: fails if a predicate reachable from the lexer start state is reintroduced.

Verification

• Token streams (Python target): I simulated the regenerated lexer (the generated 2.5.0 lexer with the predicate disabled, plus the new custom lexer) and compared it with the current lexer on the 88 test_models and 13 edge cases:
  • The edge cases covered leading spaces, tabs, mixed indentation, //, # and /* */ comments, blank lines, empty and whitespace-only files, CRLF, and a leading namespace.
  • All 101 inputs gave identical tokens (type, text, line, column, start, stop) and parse results.
• UVLHub corpus: in progress: comparing the stock lexer, the new lexer and the flamapy workaround on all 1,630 UVLHub models (parse results, model fingerprints and operation results). Results will be added here before this PR leaves draft.
• Python test suite: passes, including the new faulty model. The new DFA-caching test only passes on a regenerated parser.
• Not run locally: Java and JavaScript (no JVM or ANTLR tool at hand), and parser regeneration. The "Run Grammar Tests" workflow regenerates all three parsers and runs their suites on this PR.