Skip to content

feat(http): migrate ClickHouse sink to Hyper 1 - #26157

Merged
pront merged 10 commits into
masterfrom
pront-http-v1-honeycomb
Sep 2, 2026
Merged

feat(http): migrate ClickHouse sink to Hyper 1#26157
pront merged 10 commits into
masterfrom
pront-http-v1-honeycomb

Conversation

@pront

@pront pront commented Aug 19, 2026

Copy link
Copy Markdown
Member

Summary

Motivation

The HTTP v1 migration needs a real production sink caller so the native client can be validated beyond isolated transport tests. This PR migrates the ClickHouse sink, which has Docker-backed integration coverage, while keeping the existing request, retry, delivery, and internal telemetry contracts.

The earlier Honeycomb migration is reverted in this PR's history; Honeycomb remains on the existing client. No configuration surface is added, and the migration is limited to ClickHouse so behavior can be compared against the released binary before more sinks move.

Changes

  • Migrate ClickHouse healthchecks, Arrow schema discovery, and event batches to the private native Hyper 1 client.
  • Add a shared native Hyper 1 HTTP sink service for buffered request bodies.
  • Apply authentication directly to native HTTP 1 request headers; no V0-to-V1 request bridge remains on the migrated ClickHouse path.
  • Use version-neutral names at the migrated sink boundary; ClickHouse now constructs one native client and shares it across schema discovery, healthchecks, and event delivery.
  • Adapt Hyper 1 request/response metadata to the existing HTTP internal telemetry events.
  • Generalize HTTP status retry logic so the native client preserves retriable transport-error behavior.

References

Vector configuration

No production configuration changes. The exact YAML used for the local A/B comparison was:

api:
  enabled: true
  address: 127.0.0.1:8686

sources:
  input:
    type: stdin
    framing:
      method: newline_delimited
    decoding:
      codec: json
  internal:
    type: internal_metrics
    scrape_interval_secs: 0.5

sinks:
  clickhouse:
    type: clickhouse
    inputs:
      - input
    endpoint: http://127.0.0.1:18123
    database: default
    table: events
    compression: none
    batch:
      max_events: 1
      timeout_secs: 1
  telemetry:
    type: console
    inputs:
      - internal
    target: stdout
    encoding:
      codec: json

The same configuration was run against a deterministic local mock ClickHouse endpoint with the custom branch binary and the released Vector 0.57.0 binary.

How did you test this PR?

  • cargo test -p vector sinks::clickhouse --lib -- --test-threads=1 (40 passed)
  • cargo test -p vector http::transport_tests --lib -- --test-threads=1 (26 passed)
  • FEATURES='default,rdkafka/dynamic-linking' make check-clippy
  • make check-events (0 errors)
  • Deterministic custom-vs-release A/B comparison:
    • both binaries sent one SELECT 1 healthcheck and one ClickHouse JSONEachRow event batch;
    • both delivered one event, with receivedEventsTotal=1, sentEventsTotal=1, and sentBytesTotal=119 for the ClickHouse sink;
    • the request method, query parameters, content type, and event body matched;
    • component telemetry was queried through the local gRPC API and the ClickHouse delivery counters matched.

The A/B comparison exposed one expected wire-level difference for follow-up: the custom client sends Accept-Encoding: identity, while Vector 0.57.0 advertises zstd,gzip,deflate,br. User-agent differences are expected because the binaries have different versions.

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. This is an internal HTTP client migration with no configuration change or intended user-facing behavior change.

Contributor Guidelines

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding the PR.
  • Some CI checks run only after we manually approve them. To catch issues early, add a pre-push hook (template) or run the following locally before pushing:
    • make fmt
    • make check-clippy (auto-fix with make clippy-fix)
    • make test
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run make build-licenses to regenerate the license inventory (https://github.com/vectordotdev/vrl/blob/main/LICENSE-3rdparty.csv) and commit the changes (if any). More details on the dd-rust-license-tool (https://crates.io/crates/dd-rust-license-tool).

@github-actions github-actions Bot added the domain: sinks Anything related to the Vector's sinks label Aug 19, 2026
@pront pront added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Aug 19, 2026
@pront pront changed the title chore(http): migrate Honeycomb sink to Hyper 1 client feat(http): migrate ClickHouse sink to Hyper 1 Aug 19, 2026
@pront
pront marked this pull request as ready for review August 19, 2026 19:29
@pront
pront requested a review from a team as a code owner August 19, 2026 19:29
@pront

pront commented Aug 19, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 2900b4bba2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…comb

# Conflicts:
#	src/sinks/clickhouse/config.rs
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-01T15:03:25.404871Z 0881fef New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@pront
pront enabled auto-merge August 31, 2026 20:50
@pront
pront added this pull request to the merge queue Aug 31, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 31, 2026
@pront
pront added this pull request to the merge queue Sep 2, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 2, 2026
@pront
pront added this pull request to the merge queue Sep 2, 2026
@pront
pront removed this pull request from the merge queue due to a manual request Sep 2, 2026
@pront
pront enabled auto-merge September 2, 2026 22:03
@pront
pront added this pull request to the merge queue Sep 2, 2026
Merged via the queue into master with commit 617ad6f Sep 2, 2026
60 checks passed
@pront
pront deleted the pront-http-v1-honeycomb branch September 2, 2026 22:50
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 2, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

domain: sinks Anything related to the Vector's sinks no-changelog Changes in this PR do not need user-facing explanations in the release changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants