From 33b4cc52a28c9c2cbcf34bb8a89599a0f661fb65 Mon Sep 17 00:00:00 2001 From: Caetano Sauer <110638587+caetano-sauer_sfemu@users.noreply.github.com> Date: Wed, 12 Aug 2026 17:51:19 +0200 Subject: [PATCH] Support variable query repetition counts For the fastest systems in this benchmark, all queries run in under a second for the hot runs. This leads to a high variability between different executions on the exact same system and hardware. To alleviate this problem, this commit enhances the dashboard and associated metric scores to support an arbitrary number of repetitions instead of the fixed 3. It aligns with the fact that the execution scripts already support this through `BENCH_TRIES` (see `lib/benchmark-common.sh`). Disclaimer: This leads to better results than currently published for our system (Hyper), which we includ in this PR for the `c7a.metal-48xl` machine. We encourage other systems to do the same so we can have more stable results overall. --- CHANGELOG.md | 3 + README.md | 10 ++-- .../results/20260812/c7a.metal-48xl.json | 59 +++++++++++++++++++ .../results/20260812/c7a.metal-48xl.json | 59 +++++++++++++++++++ hyper/results/20260812/c7a.metal-48xl.json | 59 +++++++++++++++++++ index.html | 9 ++- lib/benchmark-common.sh | 4 +- prepare-database.sql | 7 ++- validate-results.py | 18 +++++- 9 files changed, 213 insertions(+), 15 deletions(-) create mode 100644 hyper-parquet-partitioned/results/20260812/c7a.metal-48xl.json create mode 100644 hyper-parquet-single/results/20260812/c7a.metal-48xl.json create mode 100644 hyper/results/20260812/c7a.metal-48xl.json diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a833f985d..9ca8fab458 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ Changes in the benchmark methodology or presentation, as well as major news. +### 2026-08-12 +Allow result rows to contain more than three query repetitions. The first run remains the cold measurement, and the minimum of all subsequent successful runs is used as the hot measurement. Existing three-run results and the default of three runs are unchanged. + ### 2026-07-22 Introducing [ClickBench Playground](https://benchmark.clickhouse.com/playground/), which allows you to run arbitrary SQL queries on 110+ databases using a pre-loaded ClickBench dataset. diff --git a/README.md b/README.md index 70947c07e3..a8c8b5f0a6 100644 --- a/README.md +++ b/README.md @@ -75,12 +75,12 @@ Tl;dr: *All Benchmarks Are ~~Bastards~~ Liars*. To add a new entry, copy-paste one of the existing directories and edit the files accordingly: -- `benchmark.sh`: this is the main script which runs the benchmark on a fresh VM; Ubuntu 24.04 or newer should be used by default. For databases that can be installed locally, the script should be able to run in a fully automated manner so it can be used in the benchmark automation (cloud-init). It should output the results in the following format: - one or more lines `Load time: 1234` with the time in seconds; - a line `Data size: 1234567890` with the data size in bytes; the data size should include indexes and transaction logs if applicable; - 43 consecutive lines in the form of `[1.234, 5.678, 9.012],` for the runtimes of every query; - the output may include other lines with the logs, that are not used for the report. For managed databases, if the setup requires clicking in a UI, write a `README.md` instead. +- `benchmark.sh`: this is the main script which runs the benchmark on a fresh VM; Ubuntu 24.04 or newer should be used by default. For databases that can be installed locally, the script should be able to run in a fully automated manner so it can be used in the benchmark automation (cloud-init). It should output the results in the following format: - one or more lines `Load time: 1234` with the time in seconds; - a line `Data size: 1234567890` with the data size in bytes; the data size should include indexes and transaction logs if applicable; - 43 consecutive JSON-array lines in the form of `[1.234, 5.678, 9.012],` for the runtimes of every query, with at least three values and the same number of values on every line; - the output may include other lines with the logs, that are not used for the report. For managed databases, if the setup requires clicking in a UI, write a `README.md` instead. - `check`, `data-size`, `install`, `load`, `query`, `start`, `stop`: These scripts perform sub-tasks during benchmarking, see `lib/benchmark-common.sh` for an overview. - `README.md`: contains comments and observations if needed. For managed databases, it can describe the setup procedure to be used instead of a shell script. - `create.sql`: a CREATE TABLE statement. If it's a NoSQL system, another file like `wtf.json` can be used instead. - `queries.sql`: contains 43 ClickBench queries to run; -- `run.sh`: a loop that running the queries; every query is run three times, see section "Caching" below for details. +- `run.sh`: a loop that runs the queries; every query is run three times by default, see section "Caching" below for details. - `results/`: put the .json files with the results for every hardware configuration into this directory, under a subdirectory named with the UTC date of the run (`results/YYYYMMDD/.json`). Each new run for an existing machine goes into a new dated subdirectory; older runs are kept for history. The website displays the latest dated copy of each `/` pair. Please double-check that each file is valid JSON (e.g., no comma errors). To introduce a new result for an existing system for a different hardware configuration, add a new file to `results//`. @@ -152,10 +152,10 @@ If a system is of a "multidimensional OLAP" kind, and so is always or implicitly ### Caching -Each of the 43 queries is run three times. +Each of the 43 queries is run three times by default. The shared benchmark driver accepts `BENCH_TRIES` to record more repetitions. We distinguish two cases: -1. Hot runs. This is the second and third run of each query. As the previous first run is supposed to populate all database and operating system caches, the two hot runs are expected to be the fastest runs overall. +1. Hot runs. These are all runs after the first run of each query. As the first run is supposed to populate all database and operating system caches, the hot runs are expected to be the fastest runs overall. 2. Cold runs. This is the first run of each query. There are two sub-cases. @@ -219,7 +219,7 @@ Load time can be zero for stateless query engines like `clickhouse-local` or `Am If you select "Cold Run" or "Hot Run", the aggregation across the queries is performed in the following way: -1. The first run for every query is considered a Cold Run. For the Hot Run, the smaller of the 2nd and 3rd runtime is used if both runs are successful, or null if some were unsuccessful. +1. The first run for every query is considered a Cold Run. For the Hot Run, the smallest of all subsequent runtimes is used if every run is successful, or null if some were unsuccessful. 2. For every query, find a system that demonstrated the best (fastest) query time and use it as a baseline. diff --git a/hyper-parquet-partitioned/results/20260812/c7a.metal-48xl.json b/hyper-parquet-partitioned/results/20260812/c7a.metal-48xl.json new file mode 100644 index 0000000000..cf266b32c5 --- /dev/null +++ b/hyper-parquet-partitioned/results/20260812/c7a.metal-48xl.json @@ -0,0 +1,59 @@ +{ + "system": "Salesforce Hyper (Parquet, partitioned)", + "date": "2026-08-12", + "machine": "c7a.metal-48xl", + "cluster_size": 1, + "proprietary": "yes", + "hardware": "cpu", + "tuned": "no", + "tags": ["C++", "column-oriented", "stateless"], + "load_time": 0.287, + "data_size": 14737666736, + "concurrent_qps": null, + "concurrent_error_ratio": null, + "result": [ + [0.592, 0.009, 0.008, 0.008, 0.01, 0.008, 0.009, 0.008, 0.008, 0.008], + [0.631, 0.014, 0.017, 0.014, 0.014, 0.013, 0.014, 0.015, 0.014, 0.015], + [0.903, 0.032, 0.035, 0.049, 0.031, 0.034, 0.033, 0.031, 0.036, 0.035], + [1.611, 0.043, 0.04, 0.037, 0.035, 0.041, 0.037, 0.038, 0.036, 0.037], + [1.844, 0.151, 0.145, 0.151, 0.137, 0.135, 0.12, 0.133, 0.123, 0.12], + [2.053, 0.146, 0.14, 0.131, 0.137, 0.12, 0.131, 0.138, 0.134, 0.132], + [0.625, 0.027, 0.027, 0.027, 0.026, 0.026, 0.026, 0.025, 0.025, 0.023], + [0.642, 0.015, 0.015, 0.015, 0.014, 0.013, 0.014, 0.018, 0.02, 0.017], + [2.123, 0.175, 0.161, 0.165, 0.16, 0.168, 0.167, 0.155, 0.163, 0.163], + [2.724, 0.248, 0.23, 0.233, 0.217, 0.222, 0.224, 0.218, 0.22, 0.216], + [1.8, 0.059, 0.059, 0.059, 0.058, 0.05, 0.059, 0.059, 0.055, 0.06], + [1.812, 0.062, 0.065, 0.062, 0.054, 0.068, 0.056, 0.056, 0.057, 0.057], + [2.077, 0.172, 0.159, 0.149, 0.148, 0.15, 0.133, 0.144, 0.144, 0.132], + [3.49, 0.297, 0.282, 0.273, 0.266, 0.259, 0.259, 0.265, 0.269, 0.268], + [2.114, 0.178, 0.16, 0.163, 0.151, 0.151, 0.166, 0.16, 0.153, 0.149], + [1.863, 0.145, 0.14, 0.131, 0.125, 0.115, 0.124, 0.122, 0.121, 0.12], + [3.616, 0.324, 0.317, 0.324, 0.312, 0.312, 0.309, 0.311, 0.3, 0.309], + [3.415, 0.231, 0.222, 0.231, 0.229, 0.22, 0.225, 0.227, 0.211, 0.233], + [5.588, 0.586, 0.59, 0.574, 0.584, 0.582, 0.602, 0.583, 0.608, 0.576], + [1.382, 0.032, 0.031, 0.029, 0.028, 0.03, 0.029, 0.029, 0.027, 0.027], + [10.635, 0.214, 0.209, 0.209, 0.219, 0.203, 0.209, 0.198, 0.218, 0.219], + [12.099, 0.222, 0.216, 0.228, 0.229, 0.224, 0.226, 0.223, 0.213, 0.215], + [20.58, 0.319, 0.305, 0.308, 0.3, 0.31, 0.295, 0.296, 0.294, 0.29], + [47.542, 0.865, 0.822, 0.796, 0.803, 0.825, 0.791, 0.783, 0.773, 0.796], + [3.881, 0.104, 0.097, 0.102, 0.094, 0.1, 0.099, 0.098, 0.094, 0.099], + [2.043, 0.078, 0.078, 0.078, 0.076, 0.082, 0.083, 0.078, 0.08, 0.079], + [3.896, 0.099, 0.101, 0.101, 0.098, 0.102, 0.097, 0.099, 0.098, 0.105], + [10.775, 0.229, 0.207, 0.215, 0.211, 0.223, 0.22, 0.216, 0.229, 0.209], + [9.242, 0.955, 1.169, 0.954, 0.978, 0.979, 0.953, 1.002, 0.964, 0.953], + [0.749, 0.033, 0.03, 0.03, 0.029, 0.025, 0.028, 0.028, 0.026, 0.027], + [3.683, 0.185, 0.176, 0.178, 0.169, 0.165, 0.171, 0.168, 0.16, 0.171], + [7.271, 0.232, 0.238, 0.213, 0.238, 0.211, 0.224, 0.227, 0.224, 0.223], + [6.188, 0.787, 0.797, 0.788, 0.808, 0.898, 0.844, 0.814, 0.89, 0.933], + [11.058, 0.658, 0.643, 0.607, 0.609, 0.611, 0.602, 0.625, 0.575, 0.601], + [11.075, 0.659, 0.626, 0.629, 0.632, 0.631, 0.629, 0.603, 0.617, 0.647], + [1.546, 0.137, 0.145, 0.129, 0.121, 0.13, 0.121, 0.131, 0.118, 0.121], + [0.816, 0.097, 0.087, 0.093, 0.076, 0.08, 0.086, 0.086, 0.098, 0.098], + [0.728, 0.04, 0.036, 0.036, 0.04, 0.04, 0.041, 0.043, 0.041, 0.04], + [0.832, 0.028, 0.027, 0.026, 0.028, 0.026, 0.026, 0.027, 0.028, 0.028], + [1.187, 0.149, 0.153, 0.152, 0.138, 0.144, 0.138, 0.152, 0.153, 0.145], + [0.689, 0.021, 0.02, 0.017, 0.021, 0.02, 0.019, 0.02, 0.021, 0.022], + [0.848, 0.019, 0.02, 0.019, 0.016, 0.019, 0.019, 0.02, 0.018, 0.019], + [0.687, 0.034, 0.032, 0.032, 0.032, 0.031, 0.032, 0.033, 0.032, 0.034] + ] +} diff --git a/hyper-parquet-single/results/20260812/c7a.metal-48xl.json b/hyper-parquet-single/results/20260812/c7a.metal-48xl.json new file mode 100644 index 0000000000..aa77d914e2 --- /dev/null +++ b/hyper-parquet-single/results/20260812/c7a.metal-48xl.json @@ -0,0 +1,59 @@ +{ + "system": "Salesforce Hyper (Parquet, single)", + "date": "2026-08-12", + "machine": "c7a.metal-48xl", + "cluster_size": 1, + "proprietary": "yes", + "hardware": "cpu", + "tuned": "no", + "tags": ["C++", "column-oriented", "stateless"], + "load_time": 0.162, + "data_size": 14779976446, + "concurrent_qps": null, + "concurrent_error_ratio": null, + "result": [ + [0.682, 0.008, 0.008, 0.01, 0.007, 0.007, 0.008, 0.007, 0.007, 0.007], + [0.718, 0.012, 0.012, 0.013, 0.013, 0.013, 0.012, 0.012, 0.012, 0.012], + [0.891, 0.025, 0.024, 0.021, 0.021, 0.023, 0.02, 0.021, 0.021, 0.021], + [1.614, 0.03, 0.027, 0.026, 0.027, 0.027, 0.026, 0.028, 0.026, 0.025], + [1.861, 0.112, 0.108, 0.098, 0.093, 0.098, 0.099, 0.093, 0.093, 0.094], + [2.076, 0.121, 0.122, 0.121, 0.116, 0.115, 0.122, 0.113, 0.119, 0.123], + [0.735, 0.019, 0.018, 0.017, 0.017, 0.022, 0.018, 0.016, 0.017, 0.015], + [0.709, 0.012, 0.013, 0.014, 0.013, 0.014, 0.012, 0.013, 0.012, 0.013], + [2.102, 0.133, 0.125, 0.127, 0.126, 0.122, 0.126, 0.122, 0.126, 0.127], + [2.598, 0.186, 0.183, 0.169, 0.164, 0.168, 0.158, 0.167, 0.173, 0.168], + [1.762, 0.043, 0.05, 0.044, 0.042, 0.044, 0.04, 0.04, 0.042, 0.041], + [1.767, 0.047, 0.048, 0.046, 0.049, 0.043, 0.043, 0.042, 0.048, 0.042], + [2.075, 0.138, 0.134, 0.128, 0.131, 0.136, 0.136, 0.13, 0.128, 0.129], + [3.391, 0.252, 0.25, 0.243, 0.229, 0.229, 0.22, 0.224, 0.214, 0.212], + [2.144, 0.144, 0.142, 0.144, 0.143, 0.142, 0.136, 0.137, 0.136, 0.136], + [1.882, 0.13, 0.135, 0.119, 0.12, 0.117, 0.128, 0.126, 0.129, 0.121], + [3.537, 0.29, 0.286, 0.279, 0.265, 0.272, 0.261, 0.256, 0.264, 0.261], + [3.345, 0.21, 0.214, 0.204, 0.215, 0.221, 0.198, 0.201, 0.199, 0.19], + [5.399, 0.592, 0.579, 0.558, 0.547, 0.545, 0.539, 0.552, 0.533, 0.54], + [1.456, 0.027, 0.027, 0.025, 0.025, 0.025, 0.023, 0.023, 0.024, 0.03], + [10.637, 0.209, 0.204, 0.205, 0.21, 0.21, 0.214, 0.213, 0.215, 0.211], + [12.07, 0.226, 0.219, 0.218, 0.215, 0.215, 0.217, 0.215, 0.217, 0.215], + [20.558, 0.307, 0.304, 0.294, 0.294, 0.299, 0.296, 0.292, 0.296, 0.303], + [47.465, 0.876, 0.866, 0.833, 0.829, 0.826, 0.852, 0.836, 0.822, 0.815], + [3.777, 0.095, 0.09, 0.088, 0.085, 0.087, 0.087, 0.088, 0.089, 0.088], + [2.036, 0.075, 0.078, 0.073, 0.073, 0.069, 0.076, 0.075, 0.076, 0.072], + [3.773, 0.095, 0.093, 0.093, 0.091, 0.088, 0.091, 0.092, 0.089, 0.09], + [10.77, 0.203, 0.205, 0.209, 0.202, 0.201, 0.201, 0.204, 0.204, 0.209], + [9.407, 1.118, 1.052, 1.051, 1.263, 1.042, 1.102, 1.047, 1.025, 1.078], + [0.85, 0.021, 0.02, 0.02, 0.021, 0.019, 0.019, 0.021, 0.02, 0.019], + [3.542, 0.158, 0.159, 0.158, 0.159, 0.154, 0.15, 0.152, 0.144, 0.138], + [6.981, 0.188, 0.189, 0.193, 0.185, 0.193, 0.177, 0.182, 0.179, 0.178], + [6.006, 0.824, 0.793, 0.771, 0.75, 0.806, 0.764, 0.834, 0.773, 0.819], + [11.102, 0.616, 0.585, 0.574, 0.558, 0.536, 0.548, 0.532, 0.536, 0.547], + [11.126, 0.628, 0.615, 0.576, 0.576, 0.578, 0.544, 0.544, 0.547, 0.529], + [1.563, 0.118, 0.109, 0.106, 0.107, 0.105, 0.102, 0.107, 0.101, 0.104], + [0.83, 0.09, 0.077, 0.086, 0.09, 0.076, 0.077, 0.09, 0.074, 0.085], + [0.757, 0.043, 0.04, 0.04, 0.039, 0.042, 0.041, 0.038, 0.044, 0.039], + [0.804, 0.026, 0.026, 0.028, 0.026, 0.026, 0.028, 0.026, 0.026, 0.027], + [0.958, 0.141, 0.15, 0.143, 0.143, 0.149, 0.142, 0.142, 0.143, 0.151], + [0.743, 0.02, 0.021, 0.015, 0.02, 0.015, 0.02, 0.02, 0.016, 0.015], + [0.902, 0.018, 0.014, 0.018, 0.016, 0.018, 0.017, 0.014, 0.014, 0.018], + [0.789, 0.032, 0.032, 0.032, 0.034, 0.034, 0.034, 0.035, 0.031, 0.03] + ] +} diff --git a/hyper/results/20260812/c7a.metal-48xl.json b/hyper/results/20260812/c7a.metal-48xl.json new file mode 100644 index 0000000000..b079e9ef25 --- /dev/null +++ b/hyper/results/20260812/c7a.metal-48xl.json @@ -0,0 +1,59 @@ +{ + "system": "Salesforce Hyper", + "date": "2026-08-12", + "machine": "c7a.metal-48xl", + "cluster_size": 1, + "proprietary": "yes", + "hardware": "cpu", + "tuned": "no", + "tags": ["C++", "column-oriented"], + "load_time": 334.355, + "data_size": 18959040512, + "concurrent_qps": null, + "concurrent_error_ratio": null, + "result": [ + [0.025, 0.008, 0.008, 0.008, 0.008, 0.007, 0.008, 0.008, 0.007, 0.008], + [0.113, 0.007, 0.007, 0.007, 0.007, 0.008, 0.008, 0.007, 0.008, 0.007], + [0.48, 0.009, 0.009, 0.008, 0.009, 0.009, 0.009, 0.008, 0.008, 0.009], + [1.103, 0.008, 0.009, 0.009, 0.008, 0.008, 0.008, 0.008, 0.009, 0.008], + [0.656, 0.082, 0.076, 0.076, 0.074, 0.075, 0.07, 0.069, 0.067, 0.067], + [2.019, 0.032, 0.031, 0.03, 0.031, 0.031, 0.046, 0.031, 0.032, 0.044], + [0.091, 0.003, 0.003, 0.002, 0.002, 0.003, 0.003, 0.003, 0.003, 0.002], + [0.081, 0.011, 0.011, 0.011, 0.011, 0.011, 0.012, 0.011, 0.011, 0.012], + [1.841, 0.12, 0.122, 0.118, 0.114, 0.109, 0.114, 0.111, 0.107, 0.107], + [3.296, 0.136, 0.123, 0.122, 0.122, 0.119, 0.111, 0.115, 0.112, 0.117], + [1.18, 0.015, 0.015, 0.015, 0.016, 0.015, 0.016, 0.014, 0.014, 0.013], + [1.188, 0.015, 0.015, 0.014, 0.015, 0.014, 0.014, 0.015, 0.015, 0.014], + [2.035, 0.057, 0.053, 0.051, 0.048, 0.046, 0.046, 0.046, 0.048, 0.046], + [4.157, 0.121, 0.109, 0.109, 0.113, 0.096, 0.097, 0.098, 0.098, 0.099], + [2.046, 0.052, 0.059, 0.06, 0.047, 0.046, 0.047, 0.047, 0.058, 0.046], + [1.186, 0.107, 0.104, 0.109, 0.111, 0.112, 0.104, 0.106, 0.107, 0.108], + [4.195, 0.172, 0.141, 0.137, 0.134, 0.135, 0.152, 0.139, 0.138, 0.158], + [4.097, 0.073, 0.067, 0.062, 0.06, 0.06, 0.076, 0.063, 0.059, 0.06], + [8.201, 0.491, 0.526, 0.548, 0.523, 0.536, 0.537, 0.542, 0.531, 0.539], + [0.141, 0.003, 0.003, 0.003, 0.002, 0.003, 0.003, 0.003, 0.002, 0.002], + [14.8, 0.05, 0.045, 0.043, 0.038, 0.037, 0.036, 0.047, 0.032, 0.029], + [16.25, 0.046, 0.042, 0.04, 0.039, 0.036, 0.034, 0.047, 0.048, 0.033], + [17.179, 0.032, 0.03, 0.03, 0.043, 0.028, 0.044, 0.026, 0.027, 0.042], + [10.703, 0.117, 0.044, 0.267, 0.055, 0.07, 0.057, 0.063, 0.011, 0.01], + [1.155, 0.064, 0.031, 0.01, 0.006, 0.005, 0.008, 0.004, 0.005, 0.004], + [1.998, 0.009, 0.009, 0.009, 0.008, 0.009, 0.009, 0.008, 0.008, 0.008], + [1.246, 0.024, 0.013, 0.023, 0.004, 0.005, 0.01, 0.004, 0.004, 0.004], + [15.211, 0.053, 0.047, 0.046, 0.044, 0.049, 0.048, 0.041, 0.041, 0.039], + [12.79, 0.599, 0.526, 0.515, 0.512, 0.517, 0.509, 0.503, 0.51, 0.533], + [0.188, 0.011, 0.011, 0.011, 0.01, 0.011, 0.011, 0.01, 0.011, 0.01], + [4.86, 0.052, 0.048, 0.046, 0.047, 0.047, 0.045, 0.044, 0.044, 0.044], + [8.406, 0.064, 0.065, 0.072, 0.056, 0.059, 0.057, 0.073, 0.057, 0.056], + [6.371, 0.682, 0.67, 0.661, 0.647, 0.661, 0.653, 0.643, 0.656, 0.654], + [14.945, 0.184, 0.178, 0.171, 0.167, 0.17, 0.174, 0.184, 0.166, 0.173], + [14.937, 0.199, 0.184, 0.179, 0.175, 0.172, 0.174, 0.177, 0.175, 0.181], + [0.719, 0.084, 0.097, 0.082, 0.077, 0.074, 0.075, 0.085, 0.078, 0.074], + [0.138, 0.01, 0.01, 0.01, 0.009, 0.009, 0.009, 0.008, 0.01, 0.01], + [0.115, 0.006, 0.006, 0.005, 0.006, 0.006, 0.006, 0.006, 0.006, 0.006], + [0.105, 0.006, 0.005, 0.005, 0.005, 0.005, 0.004, 0.005, 0.005, 0.005], + [0.155, 0.014, 0.016, 0.014, 0.016, 0.014, 0.015, 0.015, 0.016, 0.015], + [0.167, 0.003, 0.003, 0.003, 0.002, 0.003, 0.003, 0.003, 0.003, 0.002], + [0.177, 0.006, 0.005, 0.005, 0.005, 0.005, 0.006, 0.005, 0.006, 0.005], + [0.102, 0.005, 0.006, 0.005, 0.005, 0.005, 0.006, 0.005, 0.005, 0.005] + ] +} diff --git a/index.html b/index.html index 3f64e1bde9..e0a79aa774 100644 --- a/index.html +++ b/index.html @@ -872,7 +872,8 @@

Detailed Comparison

function selectRun(timings, metric) { if (timings == null) return null; const cold_timing = timings[0]; - const hot_timing = (timings[1] !== null && timings[2] !== null ? Math.min(timings[1], timings[2]) : null); + const hot_timings = timings.slice(1); + const hot_timing = (hot_timings.length >= 2 && hot_timings.every(timing => timing != null) ? Math.min(...hot_timings) : null); return metric == 'cold' ? cold_timing : metric == 'hot' ? hot_timing : (hot_timing !== null && cold_timing !== null ? (hot_timing * combined_hot_share + cold_timing * combined_cold_share) / (combined_hot_share + combined_cold_share) : null); @@ -939,9 +940,11 @@

Detailed Comparison

const num_queries = filtered_data[0].result.length; + const real_data = filtered_data.filter(elem => !elem.fake); + const num_runs = Math.max(...real_data.flatMap(elem => elem.result.map(timings => timings?.length ?? 0))); const baseline_data = [...filtered_data[0].result.keys()].map(query_num => - [...Array(3).keys()].map(run_num => - Math.min(...filtered_data.filter(elem => !elem.fake).map(elem => elem.result[query_num]?.[run_num]).filter(x => x != null)))); + [...Array(num_runs).keys()].map(run_num => + Math.min(...real_data.map(elem => elem.result[query_num]?.[run_num]).filter(x => x != null)))); const min_load_time = Math.min(...filtered_data.map(elem => elem.load_time).filter(x => x && x > 5)); const min_data_size = Math.min(...filtered_data.map(elem => elem.data_size).filter(x => x && x > 1e9)); diff --git a/lib/benchmark-common.sh b/lib/benchmark-common.sh index b7d8f18e66..aa3ca31d91 100755 --- a/lib/benchmark-common.sh +++ b/lib/benchmark-common.sh @@ -216,7 +216,7 @@ bench_load() { fi } -# Run a single query script and emit a single JSON-array `[t1,t2,t3],` line. +# Run a single query script and emit one JSON array containing BENCH_TRIES timings. # Per-try timing is also appended to result.csv as `,,`. bench_run_query() { local query="$1" @@ -299,7 +299,7 @@ bench_run_query() { echo "${query_num},${i},${timing}" >> result.csv done - # Emit "[t1,t2,t3]," for compatibility with the existing log format. + # Emit one JSON-array line for compatibility with the existing log format. local out="[" local j for j in "${!results[@]}"; do diff --git a/prepare-database.sql b/prepare-database.sql index 4b2fe6ffe8..ff02edbe70 100644 --- a/prepare-database.sql +++ b/prepare-database.sql @@ -53,7 +53,8 @@ WITH match(content, 'Load time:\s*(?:COPY \d+\n)?(\d+)') ? arraySum(x -> toFloat64(x), extractAll(content, 'Load time:\s*(?:COPY \d+\n)?(\d+)')) : NULL AS load_time, match(content, 'Data size: *(\d+)') ? arraySum(x -> toUInt64(x), extractAll(content, 'Data size: *(\d+)')) : NULL AS data_size, - extractAllGroups(content, '\n *\[([\d\.]+|null),\s*([\d\.]+|null),\s*([\d\.]+|null)\]') AS runtimes, + extractAll(content, '\n *(\[(?:[\d\.]+|null)(?:,\s*(?:[\d\.]+|null)){2,}\])') AS runtime_rows, + arrayMap(x -> extractAll(x, '[\d\.]+|null'), runtime_rows) AS runtimes, '[\n' || arrayStringConcat(arrayMap(x -> ' [' || arrayStringConcat(arrayMap(v -> v == 'null' ? v : round(v::Float64, 3)::String, x), ', ') || ']', runtimes), ',\n') || '\n]' AS runtimes_formatted, -- Concurrent QPS test (lib/benchmark-common.sh bench_concurrent_qps). @@ -70,7 +71,9 @@ WITH -- 'ClickBench PR: ' that may occur later in untrusted bench output. toUInt32OrZero(extract(content, 'ClickBench PR: (\d*)')) AS clickbench_pr, - load_time IS NOT NULL AND length(runtimes) = 43 AND data_size >= 5000000000 + load_time IS NOT NULL AND length(runtimes) = 43 + AND arrayAll(x -> length(x) = length(runtimes[1]), runtimes) + AND data_size >= 5000000000 AND arrayExists(x -> arrayExists(y -> toFloat64OrZero(y) > 0.1, x), runtimes) AS good SELECT time, system, machine, system_name, proprietary, tuned, tags, total_time, disk_space_diff, load_time, data_size, length(runtimes) AS num_results, runtimes, runtimes_formatted, diff --git a/validate-results.py b/validate-results.py index 9fa8c8527f..d1345d94fc 100644 --- a/validate-results.py +++ b/validate-results.py @@ -9,7 +9,7 @@ EXPECTED_QUERIES = 43 -EXPECTED_RUNS = 3 +MINIMUM_RUNS = 3 OUTLIER_SECONDS = 24 * 60 * 60 DATE_DIR_RE = re.compile(r"^\d{8}$") ISO_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$") @@ -89,13 +89,25 @@ def validate_result_matrix(path, data, active, problems): if len(result) != EXPECTED_QUERIES: add(problems, severity, path, f"result must contain {EXPECTED_QUERIES} query rows, got {len(result)}") + expected_runs = None for query_index, row in enumerate(result, 1): if not isinstance(row, list): add(problems, severity, path, f"result row {query_index} must be an array") continue - if len(row) != EXPECTED_RUNS: - add(problems, severity, path, f"result row {query_index} must contain {EXPECTED_RUNS} timings, got {len(row)}") + if expected_runs is None: + expected_runs = len(row) + elif len(row) != expected_runs: + add( + problems, + severity, + path, + f"result rows must contain the same number of timings; " + f"row {query_index} has {len(row)}, expected {expected_runs}", + ) + + if len(row) < MINIMUM_RUNS: + add(problems, severity, path, f"result row {query_index} must contain at least {MINIMUM_RUNS} timings, got {len(row)}") for run_index, value in enumerate(row, 1): if value is None: