Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,8 @@ jobs:
- uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4
- name: just test
run: just test
- name: Compile opt-in read benchmarks
run: just readbench-check

race-shards:
name: Race shards
Expand Down
670 changes: 670 additions & 0 deletions docs/proposals/0006-read-performance.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/proposals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,6 @@ Earlier proposals predate the template and keep their original structure.
| 0003 | [Milestone 5: correct metrics and the system page](0003-agent-dashboards-m5.md) | Implemented |
| 0004 | [Full UI without an LLM key, and PitotQL](0004-pitotql.md) | Accepted |
| 0005 | [Dedicated attribute columns and batch format 5](0005-dedicated-attribute-columns.md) | Accepted |
| 0006 | [Read performance under sustained ingest](0006-read-performance.md) | Draft |

Settled decisions that a proposal leads to are recorded in [../adr](../adr/README.md).
52 changes: 49 additions & 3 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,16 +167,62 @@ used by the dashboard. Retain failures and timeout counts alongside latency; do
make a slow or failed run disappear by retrying until it passes. These targets
are acceptance criteria, not guarantees for arbitrary telemetry volumes.

The executor compiles ordinary structured aggregates against raw
telemetry. Its disposable read-cache version 4 stores completed-batch trace
candidates and parts, not general minute aggregates. Service maps combine
The executor uses analytical minute summaries for supported span/log aggregates,
filters and dimensions; other shapes read raw telemetry. The disposable trace
read cache stores completed-batch candidates and parts. Service maps combine
analytical rollups with a raw client-span query for external peers. That query
materializes the filtered client projection once for reuse by peer and edge
aggregates; its child-span and observed-service checks also read spans. Health
panels use analytical rollups. Rollup data has separate watermark lag and approximate latency
semantics. Do not assume those rollups answer arbitrary raw counts,
rates or quantiles exactly, or report every dashboard query as cached.

`just readbench-check` compiles the opt-in native read harness in CI without
running its large fixtures. Run the local executor microbenchmark explicitly:

```sh
rtk bash scripts/with-duckdb.sh go test ./internal/query -tags=readbench \
-run '^$' -bench '^BenchmarkPanelQueries24Hours$' -benchtime=100x -count=1 -v
```

Without the `readbench` build tag this selection runs no benchmarks. The default
generated fixture is regression data, not capacity evidence. Representative
replay uses `PITOT_PANEL_BENCH_DATA_DIR` and must pass format-5 verification.
Executor measurements exclude browser paint and sustained external ingest;
record result-cache policy and analytical warmup with their results.

An opt-in mixed-load probe exercises the Executor while ingestion, analytical
rollups, completed-batch caching, compaction and maintenance run concurrently:

```sh
rtk env PITOT_MIXED_READ=1 PITOT_MIXED_RATE=40 PITOT_MIXED_SCALE=4 \
PITOT_MIXED_SEED_STATE=settled \
bash scripts/with-duckdb.sh go test ./internal/query -tags=readbench \
-run '^TestMixedReadLoad$' -count=1 -timeout=3m -v
```

Use `PITOT_MIXED_SEED_STATE=backlog` to include initial compaction catch-up.
Freeze `PITOT_MIXED_END` to the same in-retention UTC RFC3339 timestamp for paired
runs. The default probe lasts 15 seconds; `PITOT_MIXED_DURATION` accepts up to
five minutes. A 90-second run at 40 requests/sec offers 1,200 requests per kind,
allowing p99 reporting when at least 1,000 complete successfully. These remain
local synthetic measurements, not a sustained production capacity qualification.
Its rate counts user requests, including multi-panel requests, whereas ingest
rates count telemetry records. Report missed arrivals and backend refusals
alongside latency; a completed probe does not by itself qualify RSS, debt or
freshness. See [the read-performance investigation](proposals/0006-read-performance.md).

The separate `filebench` experiment compares the complete bound trace-ranking
query across candidate expressions and materialization policies:

```sh
rtk bash scripts/with-duckdb.sh go test ./internal/engine -tags=filebench \
-run '^TestTraceCandidateWindowPlanComparisonBenchmark$' -count=1 -v
```

It checks rankings against raw spans before collecting interleaved warm samples
on one pinned file snapshot. Its timings exclude binding and user-interface work.

The native tests exercise whole-window totals, bounded newest buckets
and width-aware 24-hour bucket selection:

Expand Down
2 changes: 1 addition & 1 deletion internal/engine/batch_cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import (
// anomaly_log is not one of them: it holds detector output that nothing can
// rebuild, so the drop keeps it. Its table is created once; any change to its
// schema must be a separate, explicit migration step.
const readCacheVersion = 18
const readCacheVersion = 19

const (
readCacheBatchLimit = 64
Expand Down
287 changes: 287 additions & 0 deletions internal/engine/checkpoint_retry_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,287 @@
package engine

import (
"context"
"database/sql"
"database/sql/driver"
"errors"
"fmt"
"runtime"
"strings"
"testing"
"time"

"github.com/DATA-DOG/go-sqlmock"
duckdb "github.com/duckdb/duckdb-go/v2"
"github.com/labstack/pitot/internal/engine/writegate"
"github.com/labstack/pitot/internal/metrics"
"github.com/labstack/pitot/internal/queryrows"
"github.com/labstack/pitot/internal/telemetry"
telemetrystore "github.com/labstack/pitot/internal/telemetry/store"
"github.com/prometheus/client_golang/prometheus/testutil"
)

func checkpointOldReader(t *testing.T, d *Duck, repo *telemetrystore.Repository) queryrows.Rows {
t.Helper()
at := time.Date(2026, 11, 1, 12, 0, 0, 0, time.UTC)
if err := repo.Commit(t.Context(), telemetrystore.Batch{ID: "checkpoint-reader", Logs: []telemetry.Log{{TimeUnixNanos: at.UnixNano(), EventUnixNanos: at.UnixNano(), IngestedAt: at.UnixNano(), Body: "checkpoint reader"}}}); err != nil {
t.Fatal(err)
}
if _, err := d.writer().ExecContext(t.Context(), "CREATE TABLE checkpoint_progress(value INTEGER); INSERT INTO checkpoint_progress VALUES (0)"); err != nil {
t.Fatal(err)
}
w := queryrows.Window{Start: at, End: at.Add(time.Minute)}
rows, err := d.QueryContext(queryrows.WithWindow(t.Context(), w), "SELECT count(*) FROM logs WHERE time>=?::TIMESTAMP_NS::TIMESTAMPTZ_NS AND time<?::TIMESTAMP_NS::TIMESTAMPTZ_NS", w.Start, w.End)
if err != nil {
t.Fatal(err)
}
// New committed WAL work after the reader's snapshot makes the native
// checkpoint refuse while that reader remains open. An idle WAL would let
// CHECKPOINT succeed without doing work, hiding the mixed-load failure.
if _, err := d.writer().ExecContext(t.Context(), "UPDATE checkpoint_progress SET value=value+1"); err != nil {
rows.Close()
t.Fatal(err)
}
return rows
}

// Wait for the real maintenance goroutine to park inside checkpoint's retry,
// not a guessed delay before releasing the old reader. The old implementation
// returns the native refusal first, so this test is red on the observed bug.
func awaitCheckpointRetry(t *testing.T, ctx context.Context, result <-chan error) {
t.Helper()
ticker := time.NewTicker(time.Millisecond)
defer ticker.Stop()
buffer := make([]byte, 1<<20)
for {
for stack := range strings.SplitSeq(string(buffer[:runtime.Stack(buffer, true)]), "\n\n") {
if strings.Contains(stack, "[select]") && strings.Contains(stack, "engine.(*Duck).checkpoint") && strings.Contains(stack, "engine.(*Duck).runRepositoryMaintenance") {
return
}
}
select {
case err := <-result:
t.Fatalf("maintenance returned before old reader drained instead of retrying: %v", err)
case <-ctx.Done():
t.Fatalf("maintenance never reached checkpoint retry: %v", ctx.Err())
case <-ticker.C:
}
}
}

func TestMaintenanceCheckpointWaitsForOldReaderToDrain(t *testing.T) {
d, repo := newBatchCacheTest(t)
rows := checkpointOldReader(t, d, repo)
ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second)
defer cancel()
result, finished := make(chan error, 1), make(chan struct{})
go func() { defer close(finished); result <- d.runRepositoryMaintenance(ctx) }()
defer func() { rows.Close(); cancel(); <-finished }()
awaitCheckpointRetry(t, ctx, result)
if release := d.writeGate.TryLock(writegate.RollupService); release != nil {
release()
t.Fatal("checkpoint retry released the analytical write gate")
}
if release := d.cacheGate.TryLock(writegate.ReadCache); release != nil {
release()
t.Fatal("checkpoint retry released the completed-batch write gate")
}
if err := rows.Close(); err != nil {
t.Fatal(err)
}
if err := <-result; err != nil {
t.Fatalf("checkpoint failed after the old reader drained: %v", err)
}
lastSuccess, _, failures, err := d.MaintenanceHealth()
if lastSuccess.IsZero() || failures != 0 || err != nil {
t.Fatalf("successful retry reported maintenance failure: success=%s failures=%d err=%v", lastSuccess, failures, err)
}
}

func TestMaintenanceCheckpointRetryHonorsCallerCancellation(t *testing.T) {
d, repo := newBatchCacheTest(t)
rows := checkpointOldReader(t, d, repo)
ctx, cancel := context.WithCancel(t.Context())
result, finished := make(chan error, 1), make(chan struct{})
go func() { defer close(finished); result <- d.runRepositoryMaintenance(ctx) }()
defer func() { rows.Close(); cancel(); <-finished }()
watch, stopWatch := context.WithTimeout(t.Context(), 2*time.Second)
defer stopWatch()
awaitCheckpointRetry(t, watch, result)
cancel()
select {
case err := <-result:
if !errors.Is(err, context.Canceled) {
t.Fatalf("caller cancellation was hidden: %v", err)
}
case <-watch.Done():
t.Fatal("checkpoint retry ignored caller cancellation")
}
}

func TestMaintenanceCheckpointRetryExhaustionIsExplicitAndBounded(t *testing.T) {
d, repo := newBatchCacheTest(t)
rows := checkpointOldReader(t, d, repo)
defer rows.Close()
ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second)
defer cancel()
before := testutil.ToFloat64(metrics.DuckDBCheckpointFailures)
started := time.Now()
err := d.runRepositoryMaintenance(ctx)
if !errors.Is(err, context.DeadlineExceeded) || !strings.Contains(err.Error(), "checkpoint retry budget exhausted") || !strings.Contains(err.Error(), "other write transactions active") {
t.Fatalf("exhausted checkpoint was hidden or mislabeled: %v", err)
}
if elapsed := time.Since(started); elapsed > time.Second {
t.Fatalf("250ms retry budget was not bounded: %s", elapsed)
}
if after := testutil.ToFloat64(metrics.DuckDBCheckpointFailures); after != before+1 {
t.Fatalf("attempts counted as logical checkpoint failures: before=%v after=%v", before, after)
}
_, _, failures, healthErr := d.MaintenanceHealth()
if failures != 1 || !errors.Is(healthErr, context.DeadlineExceeded) {
t.Fatalf("checkpoint exhaustion not reflected in maintenance health: failures=%d error=%v", failures, healthErr)
}
}

func TestCheckpointRetryMatchesOnlyPinnedNativeDiagnostic(t *testing.T) {
busy := &duckdb.Error{Type: duckdb.ErrorTypeTransaction, Msg: checkpointBusyMessage}
for _, check := range []struct {
name string
err error
want bool
}{
{"native", busy, true},
{"wrapped native", fmt.Errorf("wrapped: %w", busy), true},
{"untyped same text", errors.New(checkpointBusyMessage), false},
{"different transaction failure", &duckdb.Error{Type: duckdb.ErrorTypeTransaction, Msg: "TransactionContext Error: unrelated failure"}, false},
{"different native type", &duckdb.Error{Type: duckdb.ErrorTypeOutOfMemory, Msg: checkpointBusyMessage}, false},
{"nil", nil, false},
} {
t.Run(check.name, func(t *testing.T) {
if got := checkpointRetryable(check.err); got != check.want {
t.Fatalf("retryable=%t want=%t for %v", got, check.want, check.err)
}
})
}
}

func TestCheckpointRetriesConflictWithoutCountingTransientFailure(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatal(err)
}
defer db.Close()
d := &Duck{DB: db, writeDB: db}
for range 2 {
mock.ExpectExec("^CHECKPOINT$").WillReturnError(&duckdb.Error{Type: duckdb.ErrorTypeTransaction, Msg: checkpointBusyMessage})
}
mock.ExpectExec("^CHECKPOINT$").WillReturnResult(sqlmock.NewResult(0, 0))
before := testutil.ToFloat64(metrics.DuckDBCheckpointFailures)
if err := d.checkpoint(t.Context()); err != nil {
t.Fatal(err)
}
if after := testutil.ToFloat64(metrics.DuckDBCheckpointFailures); after != before {
t.Fatalf("successful logical checkpoint counted retry attempts as errors: before=%v after=%v", before, after)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatal(err)
}
}

func TestCheckpointDoesNotRetryUnrelatedError(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatal(err)
}
defer db.Close()
d := &Duck{DB: db, writeDB: db}
injected := errors.New("injected checkpoint I/O failure")
mock.ExpectExec("^CHECKPOINT$").WillReturnError(injected)
if err := d.checkpoint(t.Context()); !errors.Is(err, injected) {
t.Fatalf("unrelated failure was retried or hidden: %v", err)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatal(err)
}
}

func TestCheckpointRetainsCallerExecutionBudget(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second)
defer cancel()
wantDeadline, _ := ctx.Deadline()
db := sql.OpenDB(checkpointExecutionConnector{execute: func(actual context.Context) (driver.Result, error) {
deadline, ok := actual.Deadline()
if !ok || deadline != wantDeadline {
return nil, fmt.Errorf("native checkpoint execution deadline=%s; want unchanged caller deadline=%s", deadline, wantDeadline)
}
return driver.RowsAffected(0), nil
}})
defer db.Close()
d := &Duck{DB: db, writeDB: db}
// Inspect the context at the driver seam, without sleeping for a slow I/O
// operation: the 250ms retry budget must never cap native execution.
if err := d.checkpoint(ctx); err != nil {
t.Fatal(err)
}
}

func TestCheckpointNativeExecutionHonorsCallerCancellation(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
entered := make(chan struct{})
db := sql.OpenDB(checkpointExecutionConnector{execute: func(actual context.Context) (driver.Result, error) {
close(entered)
<-actual.Done()
return nil, actual.Err()
}})
defer func() { cancel(); db.Close() }()
d := &Duck{DB: db, writeDB: db}
result := make(chan error, 1)
go func() { result <- d.checkpoint(ctx) }()
select {
case <-entered:
cancel()
case <-time.After(time.Second):
t.Fatal("native execution was not reached")
}
select {
case err := <-result:
if !errors.Is(err, context.Canceled) {
t.Fatalf("native caller cancellation hidden: %v", err)
}
case <-time.After(time.Second):
t.Fatal("native execution ignored caller cancellation")
}
}

type checkpointExecutionConnector struct {
execute func(context.Context) (driver.Result, error)
}

func (c checkpointExecutionConnector) Connect(context.Context) (driver.Conn, error) {
return checkpointExecutionConn(c), nil
}
func (checkpointExecutionConnector) Driver() driver.Driver { return checkpointExecutionDriver{} }

type checkpointExecutionDriver struct{}

func (checkpointExecutionDriver) Open(string) (driver.Conn, error) {
return nil, errors.New("checkpoint test requires its connector")
}

type checkpointExecutionConn struct {
execute func(context.Context) (driver.Result, error)
}

func (checkpointExecutionConn) Close() error { return nil }
func (checkpointExecutionConn) Prepare(string) (driver.Stmt, error) {
return nil, errors.New("checkpoint test does not prepare statements")
}
func (checkpointExecutionConn) Begin() (driver.Tx, error) {
return nil, errors.New("checkpoint test does not begin transactions")
}
func (c checkpointExecutionConn) ExecContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Result, error) {
if query != "CHECKPOINT" || len(args) != 0 {
return nil, fmt.Errorf("unexpected checkpoint execution: %s", query)
}
return c.execute(ctx)
}
Loading
Loading