Skip to content

feat: native randstr implementation compatible with Spark#5035

Open
andygrove wants to merge 2 commits into
apache:mainfrom
andygrove:feat/native-randstr
Open

feat: native randstr implementation compatible with Spark#5035
andygrove wants to merge 2 commits into
apache:mainfrom
andygrove:feat/native-randstr

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

There is no dedicated tracking issue; randstr was previously listed as planned (🔜) in the expression support guide.

Closes #.

Rationale for this change

Spark 4.0+ randstr(length[, seed]) fell back to Spark because Comet had no native implementation. Spark generates a random alphanumeric string per row via ExpressionImplUtils.randStr, seeding an XORShiftRandom with seed + partitionIndex per partition and drawing abs(rng.nextInt() % 62) per character (mapped onto 0-9/a-z/A-Z). Comet already ports that exact XORShiftRandom for native rand/randn, so randstr can run natively and produce results that match Spark bit for bit, keeping the enclosing operator on the native pipeline instead of falling back.

What changes are included in this PR?

This change was scaffolded with the project's implement-comet-expression skill (and audited with audit-comet-expression).

  • Native RandStrExpr (native/spark-expr/src/nondetermenistic_funcs/randstr.rs), modeled on the rand/uuid pattern: it keeps a per-partition XorShiftRandom in a Mutex so state advances across batches, and encodes each length-byte string into a pre-sized StringBuilder. Added XorShiftRandom::next_i32() (Java Random.nextInt()).
  • Proto RandStr { length, seed } message wired through the expression registry with a RandStrBuilder that combines the seed with the partition index (matching rand/randn/shuffle).
  • CometRandStr serde emitting the resolved literal length and seed. Because RandStr is a Spark 4.0+ class, the serde lives under spark-4.x and is registered via Spark4xCometExprShim.sparkVersionSpecificStringExpressions, so Spark 3.4/3.5 (which lack the class) still compile.
  • Support-status doc flipped from planned to supported, and a misc_funcs audit entry recording the cross-version findings.

The audit confirmed ExpressionImplUtils.randStr and the XORShiftRandom seeding are byte-identical across Spark 4.0.1 and 4.1.1, so no intra-4.x shim is needed. length and seed are required-foldable; Comet handles non-negative literal length with a literal seed and otherwise falls back to Spark (which also raises the negative-length error).

How are these changes tested?

  • Rust unit tests in randstr.rs: length and alphanumeric charset, zero length, determinism per (length, seed), and per-batch state continuity.
  • expressions/string/randstr.sql (Spark 4.0+, gated with MinSparkVersion: 4.0): seeded randstr(length, seed) asserts bit-for-bit equality with Spark across single and multiple rows, negative/zero/boundary seeds, lengths 0/1/12/64, and combined with other expressions; a negative length asserts the INVALID_PARAMETER_VALUE.LENGTH fallback; the no-seed randstr(length) form checks deterministic length and alphabet properties. Verified natively on Spark 4.1.2 (default); on Spark 3.5 the build compiles and the gated file is skipped.

…er-row UTF-8 scan

Move the literal-length/literal-seed/non-negative-length restrictions from convert
into getSupportLevel + getUnsupportedReasons so they surface in EXPLAIN and the
compatibility docs, matching the rand/randn serde pattern. Build each string in a
reused String buffer instead of validating a byte buffer per row.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant