Add DDIR benchmark scorecard - #837
Draft
frankmcsherry wants to merge 3 commits into
Draft
Conversation
The operator scorecard attributed the whole compiled-to-DDIR gap to interpretation, but DDIR renders every program inside an iterative scope and so runs at Product<u64, PointStamp<u64>> where a typed program runs at u64. That is a larger timestamp, a vector-valued lattice, and an enter per input, and it is a cost of the runtime embedding rather than of interpreting anything. Add the compiled-ddir rung to `operators` and `arrange_updates`: the same typed plan, rendered in the iterative scope at DDIR's timestamp. compiled-ddir / compiled is now the embedding tax and ddir-vec / compiled-ddir is interpretation. Validate both compiled rungs rather than assume them. compiled-ddir selects different code paths inside reduce and join, so its agreement is a claim to be tested; render_compiled grows an optional capture that exists only in the untimed validation run. Stop cloning input rows inside the operators ingest region. The rows were already owned and unused afterwards, so the clone charged the DDIR paths a deep copy of every Value that the typed paths, whose rows are Copy, never paid. It showed up as a flat ~2.2ms against ~0.3ms in every case, which is most of the identity row. Require --runs to be a multiple of the implementation count. Rotating the implementation order only removes ordering bias if the rotation completes a whole number of times; five runs over three implementations let one lead twice. Record whether the working tree was dirty. A revision alone does not identify what ran, and both preliminary reports had to admit as much in prose. Replace the constant `correct: true` with `checked_against`, naming the oracle, which differs between scc and the other two. Schema goes to 2. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D4xjfaAy9ksW62KcKLvaUf
Replaces the 2026-08-17 preliminary reports. Those measured a dirty working tree, so no revision identifies them, and their operator ratios attributed DDIR's runtime embedding to interpretation. Every record here carries dirty: false at 1d02e4d. Different machine: an M4 Mac mini rather than the M2 MacBook Air, so absolute times are not comparable across the two reports. The operator scorecard now separates the embedding tax from interpretation. The same typed plan rendered in DDIR's iterative scope costs 1.32x to 2.04x, largest where operator work is smallest, so a substantial share of what the previous report called interpretation was never interpretation. Corgi lands within 1.09x to 1.24x of the embedded compiled plan on the joins and 1.51x on eight chained maps, its best row. SCC does not make the same split: both of its compiled variants already render in nested scopes, at Product<_, usize> rather than at PointStamp, so its Vec / plan column is not comparable to the scorecard's Vec / embedded. The report says so and names the follow-up. Both reports now carry a variation table. Dispersion is under 4% throughout SCC; in the scorecard it is the sub-millisecond rows that move, and those ratios are worth one significant figure. Balancing the rotation did not measurably reduce dispersion here, and the report says that rather than claiming it did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D4xjfaAy9ksW62KcKLvaUf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
V0 of an attempt to maintain more durable benchmarks about the relationship between compiled and interpreted DD.