Skip to content

Measure CometRegex whitelist coverage on representative regex patterns #5812

Description

@sam-1112

What is the problem the feature request solves?

The initial CometRegex whitelist introduced by PR #5415 is deliberately conservative. It rejects commonly used constructs such as ., ^, $, \d, \w, and \s.

We currently do not know what fraction of representative real-world rlike patterns is admitted for automatic native execution. TPC-DS does not contain useful rlike coverage, so it cannot answer this question.

Without an admission-rate measurement, it is difficult to prioritize follow-up compatibility work or estimate how much real-world workload coverage the whitelist provides.

Describe the potential solution

Identify one or more representative regex corpora, such as public SQL workloads, open-source application queries, or safely shareable production-derived patterns.

Run every literal pattern through CometRegex and report:

  • the corpus source and collection method;
  • total, admitted, and rejected pattern counts;
  • the admission percentage;
  • duplicate-handling methodology;
  • rejection counts grouped by unsupported construct or analyzer rule;
  • useful pattern-length and complexity distributions.

Use the results to prioritize follow-up work such as support for leading ^ or safe normalization of selected Java regex constructs.

The whitelist should not be weakened solely to increase the reported admission percentage.

Additional context

Follow-up from #5415 and #5351.

This tracks the representative admission-rate measurement requested during review of #5415: #5415 (review)

The original issue notes that the TPC query corpus does not contain rlike usage: #5351

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions