Add Rayforce

[alexey-milovidov]

Adds Rayforce — a zero-dependency
columnar analytics and graph engine in pure C (MIT), queried through Rayfall,
its Lisp-like language. queries.sql therefore holds 43 Rayfall expressions,
one per line, the way jd and bqn hold their own dialects.

(Not the same as the closed #940 — that was RayforceDB's internal 10M-row
harness, opened against this repo by mistake. This is a standard-harness entry
on the full dataset.)

Two measured engine properties shaped the entry

Text columns load as SYM (dictionary-encoded), not STR. STR pool
offsets are uint32_t, so a column's pool is capped at 4 GiB, while at 100M
rows URL/Title/Referer/OriginalURL hold 5–11 GB of bytes each. Opening a
splayed table also validates every STR element, and that cost is superlinear:
0.4 s at 2M rows, 199 s at 9.6M rows (99 s for the parted layout). All-SYM
opens in 17 s at 9.6M rows and stores the same subset in 4.4 GB instead of
7.5 GB.

It runs as an IPC server, with BENCH_RESTARTABLE=yes. The table open is
eager (validate every column file, load the dictionary), so a CLI-per-query
would pay it 129 times per run; the server pays it once per start and times
queries with timeit server-side. The restart is not optional either: with the
server left up, drop_caches cannot evict pages a live process still maps, and
a "cold" query measured 0.029 s versus 0.44 s after a real restart — keeping the
process alive would report every run as warm. BENCH_CHECK_TIMEOUT is raised to
1800 s because the readiness probe now waits for that open.

Query notes

HAVING becomes an outer select over the grouped one; LIMIT n OFFSET m is
take: [m n]; DATE_TRUNC('minute', EventTime) is
(xbar EventTime 60000000000); CASE WHEN is the row-wise (if ...). Q29's
REGEXP_REPLACE is rebuilt from str-find/substr/if and reproduces the
regex exactly on this dataset, fallback rows included. Q25/Q27 project
EventTime alongside SearchPhrase because sorting runs after projection, and
Q35 projects its constant group key in an outer select.

Validation

All 43 queries were diffed against clickhouse-local on a 2M-row subset of the
same CSV: 33 match exactly or modulo row order, and the other 10 differ only in
which of several equally-ranked rows a LIMIT over ties returns (the sort-key
multisets are identical). install, load, the stop/drop_caches/start/check
cold cycle, all 43 queries through ./query, 10 concurrent clients, and the
error paths were exercised end to end.

Upstream bugs found on the way

Neither affects this entry, both filed with minimal repros:

• select with take: returns empty strings for pool-backed STR values RayforceDB/rayforce#404 — select with take: returns empty strings for
  pool-backed STR values (silently; (select {from: t take: 1}) is enough).
  A further reason the text columns here are SYM.
• Ungrouped (count (distinct col)) in a select projection returns the row count (STR/SYM) or a type error (I64) RayforceDB/rayforce#405 — ungrouped (count (distinct col)) in a select
  projection returns the row count on STR/SYM and a type error on integers;
  hence Q5/Q6 use the whole-column reducer (count (distinct (at hits 'c))).

🤖 Generated with Claude Code

[github-actions]

Results for rayforce are ready for: c6a.4xlarge.
The result files are committed as 74e151f.

Logs:

• rayforce on c6a.4xlarge: benchmark log, cloud-init log

[github-actions]

The run of rayforce on c6a.xlarge did not produce results.

Logs:

• rayforce on c6a.xlarge: benchmark log, cloud-init log

[github-actions]

The run of rayforce on c6a.2xlarge did not produce results.
The run of rayforce on c6a.metal did not produce results.
The run of rayforce on c7a.metal-48xl did not produce results.
The run of rayforce on c8g.metal-48xl did not produce results.

Logs:

• rayforce on c6a.metal: benchmark log, cloud-init log
• rayforce on c7a.metal-48xl: benchmark log, cloud-init log
• rayforce on c8g.metal-48xl: benchmark log, cloud-init log
• rayforce on c6a.2xlarge: benchmark log, cloud-init log

[github-actions]

The run of rayforce on c6a.large did not produce results.

Logs:

• rayforce on c6a.large: benchmark log, cloud-init log

[github-actions]

The run of rayforce on t3a.small did not produce results.

Logs:

• rayforce on t3a.small: benchmark log, cloud-init log

[github-actions]

Results for rayforce are ready for: c8g.4xlarge.
The result files are committed as 9658bf5.

Logs:

• rayforce on c8g.4xlarge: benchmark log, cloud-init log

[alexey-milovidov]

It tends to randomly degrade up to being totally stuck:

[220.535000,3.190810,1.980040],
bench: ./check did not succeed within 1800s
bench: last ./check stderr was:
    error: io: connection refused: 127.0.0.1:5000

[alexey-milovidov]

But as long as we have two configurations on which the system barely works, it's okay to merge.

[alexey-milovidov]

But no - I don't think that the results are trustworthy. This needs a more detailed investigation.

[github-actions]

Results for rayforce are ready for: c6a.4xlarge.
The result files are committed as 1db5f5e.

Logs:

• rayforce on c6a.4xlarge: benchmark log, cloud-init log