feat(app): show live ClickHouse query progress during a search - #3103
feat(app): show live ClickHouse query progress during a search#3103wrn14897 wants to merge 2 commits into
Conversation
The results table footer and the histogram's "Scanned Rows | Elapsed Time"
strip now report rows read and percent complete while a search runs, instead
of an unqualified spinner.
Progress is read from ClickHouse's JSONEachRowWithProgress events, streamed
in the response body. Progress headers are not an option in the browser:
ClickHouse stops emitting X-ClickHouse-Progress once the body starts, and
fetch() cannot surface headers incrementally anyway. The format needs >= 25.1
for its meta/exception events, so the format is chosen per connection and
older servers silently keep the previous non-streaming path.
The percentage is time coverage, not rows. The only row denominator available
is an EXPLAIN estimate that ignores skip indexes and early LIMIT termination,
and is wrong outright when a chart is rewritten onto a materialized view, so a
row-based bar would routinely stall short of the end. Coverage also matches
what the surrounding UI already says ("Searched ... across about 1 month").
Search results paginate one time window at a time, each its own fetch, so
isFetching dips between windows. Progress deliberately survives those dips —
clearing on that edge made the bar vanish and the elapsed timer restart at
every window boundary — and idle gaps are excluded from elapsed. A fresh run
resets it from the queryFn, which is keyed to an actual new query rather than
racing a timeout. Windows also register themselves as in-flight at 0% before
reading, so the placeholder page for a just-opened window is not credited its
whole range.
Hidden during live tail, where the query refetches every few seconds and a bar
would only flicker.
🦋 Changeset detectedLatest commit: 09280ea The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryAdds live ClickHouse query progress to search results and histogram status displays while preserving compatibility with older ClickHouse servers.
Confidence Score: 4/5The functional progress fixes appear sound, but the explicit semantic-color requirement must be satisfied before merging; live-tail progress collection should also be disabled to avoid unused recurring work. The previous offset-pagination issue is fixed by withholding full-window completion until an empty page, the shared histogram observers now receive matching execution options, and the prohibited test Files Needing Attention: packages/app/src/DBSearchPage.tsx, packages/app/src/components/QueryProgressIndicator.tsx
|
| Filename | Overview |
|---|---|
| packages/app/src/hooks/useOffsetPaginatedQuery.tsx | Streams supported queries and now preserves counters without marking offset-paginated windows complete prematurely. |
| packages/app/src/hooks/useQueryProgress.ts | Aggregates active and settled progress across requests sharing a time-window chunk. |
| packages/app/src/DBSearchPage.tsx | Enables histogram progress throughout search views, but leaves collection enabled during live tail when its display is suppressed. |
| packages/app/src/components/QueryProgressIndicator.tsx | Adds inline and block progress displays, with raw palette colors that should use semantic tokens. |
| packages/api/src/routers/api/clickhouseProxy.ts | Flushes each compressed upstream chunk and destroys started responses safely on upstream failure. |
| packages/app/src/components/SearchTotalCountChart.tsx | Aligns progress and parallel-query options across observers sharing the histogram query key. |
Sequence Diagram
sequenceDiagram
participant UI as Search UI
participant Query as Query hooks
participant Proxy as ClickHouse proxy
participant CH as ClickHouse
UI->>Query: Start search with progress enabled
Query->>Proxy: JSONEachRowWithProgress query
Proxy->>CH: Forward query
loop While query runs
CH-->>Proxy: Progress or row chunk
Proxy-->>Query: Flush chunk immediately
Query-->>Query: Aggregate window/page progress
Query-->>UI: Update rows, elapsed time, and coverage
end
CH-->>Proxy: Metadata and completion
Proxy-->>Query: Final chunks
Query-->>UI: Render completed results
Reviews (2): Last reviewed commit: "fix(app): correct progress accounting fo..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
E2E Test Results✅ All tests passed • 357 passed • 1 skipped • 1402s
Tests ran across 4 shards in parallel. |
…ervers Three issues from review. A window serves its results over several offset pages, each its own ClickHouse query. Marking the window complete after every page credited its whole range while pages were still running, and let each page's counters replace the previous page's instead of adding to them, so the reported row count could drop. A window is now credited its full range only once a page comes back empty, which is the same condition getNextPageParam uses to move on, and settled counters are banked on the progress entry so several requests can share one chunk. coverageFromPages applies the same rule to pages restored from cache. reportProgress is not part of the query key, so whichever observer wins the deduplicated fetch decides whether progress is emitted at all. The count text shares the histogram key and renders before the strip, so it was usually the one to own the request — with progress reporting off. It now takes and forwards the flag like the other two observers. Also types a test's promise resolver instead of using any.
| } | ||
| histogramTimeChartConfig={histogramTimeChartConfig} | ||
| enableParallelQueries | ||
| reportProgress |
There was a problem hiding this comment.
During live tail, the visible progress indicator is suppressed, but the mounted histogram observers still receive reportProgress. Every poll therefore continues collecting and storing progress that cannot be displayed, adding avoidable streaming and client-state overhead. Disable progress reporting when live tail is active instead of only hiding the indicator. The same issue occurs for the time-chart observer at line 2624.
| animated={isIndeterminate} | ||
| size="xs" | ||
| w={72} | ||
| color="gray.6" |
There was a problem hiding this comment.
This progress bar uses the raw Mantine palette value color="gray.6", and the block variant repeats it at line 71. The repository UI directive requires semantic color tokens for application content when one exists so components remain consistent across themes and palette changes. This repository requirement must be satisfied before merging.
Context Used: CLAUDE.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
1 participant
Why
A search over a long range shows an unqualified spinner for its whole duration.
The user cannot tell whether ClickHouse is 5% or 95% of the way through, or
whether it is doing anything at all. This surfaces ClickHouse's own progress
counters in two places: the results-table footer and the histogram's
"Scanned Rows | Elapsed Time" strip.
How
Transport. Progress comes from
JSONEachRowWithProgress, which interleaves{"progress":...}lines with rows in the response body. Headers were a deadend: ClickHouse stops emitting
X-ClickHouse-Progressonce the body starts(
WriteBufferFromHTTPServerResponse::onProgresschecksheaders_finished_sending),and
fetch()cannot surface headers incrementally anyway. The format needsClickHouse >= 25.1 (PR #74181 added the
meta/exceptionevents); it ischosen per connection via
metadata.getServerVersion, and older or unknownservers silently keep the existing non-streaming path.
Proxy. The API's
clickhouse-proxynow flushes each upstream chunk throughcompression()instead of letting zlib buffer the tiny progress lines untilenough bytes accumulate. It also no longer tries to write a 500 after the body
has started — that threw
ERR_HTTP_HEADERS_SENTfrom inside the errorlistener, an uncaught exception that takes down the process in dev.
Percent is time coverage, not rows. The only row denominator available is
an EXPLAIN estimate that ignores skip indexes and early LIMIT termination, and
is wrong outright when a chart is rewritten onto a materialized view — a
row-based bar would routinely stall short of the end. Instead each time window
contributes its share of the total range, weighted by how far ClickHouse has
scanned it. This also matches what the surrounding UI already says
("Searched ... across about 1 month").
Surviving window boundaries. Search pagination fetches one time window at a
time (
15m, 6h, 6h, 12h, 24h, ...), soisFetchingdips between them.Progress deliberately survives those dips — clearing on that edge made the bar
vanish and the elapsed timer restart at every window, dozens of times over a
month-long range — and idle gaps are excluded from elapsed. A fresh run resets
it from the
queryFn, which is keyed to an actual new query rather than racinga timeout. Windows also register themselves as in-flight at 0% before reading,
so a just-opened window's placeholder page is not credited its whole range.
Hidden during live tail, where the query refetches every few seconds and a bar
would only flicker.
Testing
make ci-lint,make ci-unit(7,855 tests),make dev-int FILE=clickhouseProxy— all greenproxy
headersSentguard, progress surviving window gaps): each test failswithout its fix
yarn devrestart.Marked draft for that reason.
Known caveats
shows ~0% for a moment before the bar disappears. Accurate, but may look odd.
windows (
DBRowTable.tsx:1255). Pre-existing; not addressed here.