Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,16 @@ releases, check out the harness revision recorded in that report. New reports us
report formats are not accepted; use the harness revision recorded in the
archived evidence to read them. No published measurement is rewritten.

New read reports add scheduled-arrival accounting and the proposed
`interactive-http-draft/v1` qualification goals. They retain historical HTTP
service-time fields while reporting scheduler queue delay, arrival-to-completion
p95/p99, intended/performed operations, and during-load versus drain completions
separately. Each performed kind needs 1,000 successful samples for p99
qualification; short sweeps therefore use the existing best-completion topology
fallback. See [read measurement and goals](docs/metrics.md#proposed-http-completion-profile).
Historical reports without this metadata are not qualified against the new
goals. HTTP completion is separate from browser last-panel paint.

Pitot Bench drives the official OpenTelemetry `telemetrygen` client, measures what Pitot actually accepts, samples the server throughout the run, and emits publication-ready JSON and Markdown. It is deliberately harder to fool than a one-line requests-per-second script.

## What the headline means
Expand Down
29 changes: 27 additions & 2 deletions docs/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,34 @@ A report passes its evidence-quality gates when:
3. the Pitot process start time does not change;
4. Pitot's accepted-row counter advances in every phase.

When read load is enabled, the sustained mixed phase must also complete at least 95% of the requested operation rate with zero operation failures and zero scheduler shedding. The report records operations canceled exactly at the phase boundary separately; they are not hidden as successful work.
When read load is enabled, the sustained mixed phase must also complete at least 95% of the requested operation rate **during the offered-load interval**, with zero operation failures, cancellations, scheduler-missed arrivals and scheduler shedding. It must meet the per-kind HTTP completion profile below. The scheduler stops when generators finish, before the final metrics scrape. Already dispatched work drains normally; successes completed during drain are reported separately and do not increase during-load goodput.

The read scheduler is open-loop and bounded. `query_attempts` counts scheduled operations, `query_latency_ms.count` counts valid completed operations, `query_shed` counts ticks rejected because the concurrency bound was full, and `query_canceled_at_phase_end` counts in-flight operations canceled when load stopped.
The read scheduler is open-loop and bounded. `query_attempts` retains its delivered scheduling-attempt count; `query_latency_ms` retains successful HTTP service time from worker dequeue through the operation's last response. Both exclude arrivals missed by the scheduler; the latency also excludes its queue. `query_shed` counts delivered attempts rejected because the concurrency bound was full. `query_canceled_at_phase_end` retains its historical name and counts canceled dispatched work; ordinary phase completion drains rather than cancels it.

New `read_evidence` accounts for offered arrival slots from monotonic elapsed time and requested rate, independently of ticker deliveries. On a delayed wake, skipped slots become `scheduler_missed`; they are never replayed as an unbounded catch-up burst. At shutdown, every remaining slot due through the captured cutoff is likewise accounted for. Per-kind intended counts include scheduler misses and sheds. Target rotation advances with offered slots, so overloaded or fallback work cannot silently change the reported intended mix. Performed-kind counts retain actual fallback attribution.

After drain, the accounting identities are:

```text
scheduled = query_attempts + scheduler_missed
scheduled = scheduler_missed + query_shed + dispatched
dispatched = successful + failures + canceled
successful = completed_during_load + completed_during_drain
```

`successful_arrival_to_completion_ms` measures scheduled arrival through final successful HTTP response, including scheduler queue delay. `dispatched_queue_ms` measures queue delay for every dispatched operation, including failures and cancellations. Success percentiles never include shed or failed work; outcome counts remain adjacent and use all offered arrivals as their denominator. `intended_operations` records requested kind, missed/shed/dispatched counts, outcomes and fallbacks; `performed_operations` records actual kind, outcomes and latency. These typed fields supplement the existing methodology tables.

### Proposed HTTP completion profile

`settings.read_profile` declares `interactive-http-draft/v1`, a versioned engineering goal rather than a measured product guarantee. It participates in the workload hash. New qualification requires matching profile metadata and arrival accounting; historical reports retain their original fields and recorded gates, and are labeled as legacy evidence without qualification against this profile.

| Performed HTTP user operation | p95 goal | p99 goal |
|---|---:|---:|
| Complete dashboard batch and annotations | 1,500 ms | 3,000 ms |
| Trace drill-down | 750 ms | 1,500 ms |
| Bounded trace detail | 300 ms | 750 ms |

Each kind needs at least 1,000 successful samples before its p99 can qualify. Fewer samples are **insufficient evidence**, not a pass. The same gate controls topology eligibility and sustained qualification. Consequently a standard 30-second sweep at 20 operations/s cannot supply enough samples per kind: topology selection uses the documented best-completion fallback, and only a sufficiently sampled sustained phase can qualify. These are HTTP operation completion goals; browser trigger-to-last-panel-paint measurements remain separate. The 500 ms individual-panel goal is not applied to an entire dashboard batch.

### Dashboard panel evidence

Expand Down
4 changes: 4 additions & 0 deletions docs/publishing.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ A performance number is useful only when another engineer can explain and reprod
- [ ] signal-specific peak results and a sustained mixed result;
- [ ] a separate read-under-write result when claiming interactive performance;
- [ ] requested, completed, failed, and shed operation rates beside read latency;
- [ ] scheduled arrivals, scheduler-missed slots, queue delay and successful arrival-to-completion p95/p99 by intended/performed operation kind; report during-load completions separately from drain;
- [ ] the declared HTTP completion profile and at least 1,000 successful samples per performed kind before p99 qualification; insufficient samples never count as a pass;
- [ ] all evidence-quality gates passing;
- [ ] limitations stated next to the headline, not hidden in a footnote.

Expand All @@ -29,6 +31,8 @@ Then publish the separate span, log-record, and metric-point sweeps. They explai

Use median values across repetitions for the headline and retain every raw repetition. Report variability; do not select only the fastest run.

The sweep `--repeat` count does not repeat the final sustained phase. Use independent invocations when repeating sustained qualification. Current short sweeps are under-sampled for the HTTP p99 profile and therefore use the documented topology fallback; publish qualification only from a sustained phase passing the declared sample, goodput, outcome and latency gates. HTTP operation completion does not measure browser paint time. Older reports without arrival accounting/profile metadata remain historical evidence and cannot be relabeled as passing the current goals.

## Language

Prefer:
Expand Down
98 changes: 47 additions & 51 deletions internal/bench/query.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ import (
"sort"
"strings"
"sync"
"sync/atomic"
"time"
)

Expand All @@ -25,32 +24,39 @@ const (
)

type queryRun struct {
targets []queryTarget
traceMu sync.Mutex
traces map[string]traceReference
selections map[string]drillSelection
kindsMu sync.Mutex
kinds map[string]*operationStats
panelsMu sync.Mutex
panels map[string]*panelStats
wg sync.WaitGroup
hist *histogram
failures int64
attempts int64
shed int64
canceled int64
stop chan struct{}
stopOnce sync.Once
cancel context.CancelFunc
scheduleMu sync.Mutex
startedAt, cutoff time.Time
rate float64
expected, missed, completedDuringLoad, completedDuringDrain int64
intended map[string]ReadIntended
endToEnd, queue *histogram
targets []queryTarget
traceMu sync.Mutex
traces map[string]traceReference
selections map[string]drillSelection
kindsMu sync.Mutex
kinds map[string]*operationStats
panelsMu sync.Mutex
panels map[string]*panelStats
wg sync.WaitGroup
hist *histogram
failures int64
attempts int64
shed int64
canceled int64
stop chan struct{}
stopOnce sync.Once
cancel context.CancelFunc
}

type operationStats struct {
dispatched, failures, canceled int64
hist *histogram
endToEnd, queue *histogram
}

func newQueryRun() *queryRun {
return &queryRun{targets: []queryTarget{{name: "dashboard-refresh", body: []byte(`{}`)}}, traces: map[string]traceReference{}, selections: map[string]drillSelection{}, kinds: map[string]*operationStats{}, panels: map[string]*panelStats{}, hist: newHistogram(), stop: make(chan struct{}), cancel: func() {}}
return &queryRun{targets: []queryTarget{{name: "dashboard-refresh", body: []byte(`{}`)}}, traces: map[string]traceReference{}, selections: map[string]drillSelection{}, kinds: map[string]*operationStats{}, panels: map[string]*panelStats{}, hist: newHistogram(), endToEnd: newHistogram(), queue: newHistogram(), intended: map[string]ReadIntended{}, stop: make(chan struct{}), cancel: func() {}}
}

func (q *queryRun) start(ctx context.Context, client *http.Client, base, cookie string, rate float64) {
Expand All @@ -69,15 +75,18 @@ func (q *queryRun) start(ctx context.Context, client *http.Client, base, cookie
// busy and the queue is full. An unbuffered channel would also shed
// whenever no worker happened to be parked in its receive at that instant,
// which reports backpressure the server never caused.
jobs := make(chan int, workers)
jobs := make(chan readJob, workers)
runCtx, cancel := context.WithCancel(ctx)
q.cancel = cancel
q.scheduleMu.Lock()
q.startedAt, q.rate = time.Now(), rate
q.scheduleMu.Unlock()
q.wg.Add(workers + 1)
for worker := 0; worker < workers; worker++ {
go func() {
defer q.wg.Done()
for targetIndex := range jobs {
q.request(runCtx, client, base, cookie, targetIndex)
for job := range jobs {
q.requestJob(runCtx, client, base, cookie, job)
}
}()
}
Expand All @@ -86,21 +95,22 @@ func (q *queryRun) start(ctx context.Context, client *http.Client, base, cookie
defer close(jobs)
ticker := time.NewTicker(interval)
defer ticker.Stop()
targetIndex := 0
for {
select {
case <-runCtx.Done():
q.stopScheduling()
return
case <-q.stop:
return
case <-ticker.C:
atomic.AddInt64(&q.attempts, 1)
select {
case jobs <- targetIndex:
targetIndex++
default:
atomic.AddInt64(&q.shed, 1)
}
q.advanceSchedule(time.Now(), func(job readJob) bool {
select {
case jobs <- job:
return true
default:
return false
}
})
}
}
}()
Expand All @@ -111,36 +121,22 @@ func (q *queryRun) start(ctx context.Context, client *http.Client, base, cookie
// already being read, and those abort as read errors rather than transport
// errors, which the trial would then count as server failures.
func (q *queryRun) finish() {
q.stopOnce.Do(func() { close(q.stop) })
q.stopScheduling()
q.wg.Wait()
q.cancel()
}

func (q *queryRun) request(ctx context.Context, client *http.Client, base, cookie string, targetIndex int) {
query := q.targets[targetIndex%len(q.targets)]
q.requestJob(ctx, client, base, cookie, readJob{targetIndex: targetIndex, scheduledAt: time.Now(), intended: q.targets[targetIndex%len(q.targets)].name})
}

func (q *queryRun) requestJob(ctx context.Context, client *http.Client, base, cookie string, job readJob) {
query := q.targets[job.targetIndex%len(q.targets)]
kind := query.name
started := time.Now()
success := false
defer func() {
elapsed := time.Since(started)
q.kindsMu.Lock()
defer q.kindsMu.Unlock()
stats := q.kinds[kind]
if stats == nil {
stats = &operationStats{hist: newHistogram()}
q.kinds[kind] = stats
}
stats.dispatched++
if success {
q.hist.record(elapsed)
stats.hist.record(elapsed)
} else if ctx.Err() != nil {
atomic.AddInt64(&q.canceled, 1)
stats.canceled++
} else {
atomic.AddInt64(&q.failures, 1)
stats.failures++
}
q.recordOperation(job, kind, started, time.Now(), success, ctx.Err() != nil)
}()
refresh := func() bool {
body, ok, absent := queryPanelHTTP(ctx, client, base, cookie, queryRunPath, query.body)
Expand Down
Loading
Loading