docs: audit missing client report drop coverage#1185
Draft
szokeasaurusrex wants to merge 44 commits into
Draft
Conversation
This refactor should force us to make sure we add EnvelopeItemTypes corresponding to new EnvelopeItems: basically this sort of error should become impossible: #1144 (comment).
Add an `EnvelopeFilter` trait and callback container so callers can observe items removed by `Envelope::filter`. This includes items rejected directly by the filter and attachments removed later when their event or transaction is no longer present. Keep existing closure-based callers source-compatible through a blanket `FnMut(&EnvelopeItem) -> bool` implementation. Some closure sites may need an explicit item type annotation after this generalization. Closes [#1181](#1181) Closes [RUST-245](https://linear.app/getsentry/issue/RUST-245) # Conflicts: # CHANGELOG.md
This will allow us to add additional construction options, either to all transports or to individual transports, in the future without breaking the public API. This will be useful in #1004, as we will need to provide transports with a handle to record data losses. For users who use a custom transport/transport factory, this change may produce a minor behavior breakage, as the `&ClientOptions` received by the `create_transport` method (or the closure, in the case of transport factories which are just closures) will no longer contain all of the options set when init-ing the SDK, only those options which are also present in `TransportOptions`. As the API remains fully compatible, however, we are not considering this to be a public API breakage, and will release these changes in a minor/patch release. References [#1004](#1004) References [RUST-156](https://linear.app/getsentry/issue/RUST-156/record-transport-drops-and-attach-client-report-envelopes)
Added WIP client report protocol. Resolves [#1001](#1001) Resolves [RUST-153](https://linear.app/getsentry/issue/RUST-153/add-client-report-protocol-envelope-item-support-in-sentry-types)
Add `Envelope::losses_on_drop` so transports can inspect an envelope before dropping it and report the data categories and quantities that were lost through client reports. The new mapping covers all parsed envelope item types that contribute reportable losses: - events, sessions, transactions, spans, attachments, and monitor check-ins - log item counts and serialized log bytes - trace metric counts Expose `EnvelopeLossIter` from `sentry-types` and add the client report `Category` variants needed to represent the resulting losses. Client report items and raw envelope items intentionally do not produce additional losses. Add a changelog entry for the new dropped-envelope loss APIs. Resolves [#1171](#1171) Resolves [RUST-242](https://linear.app/getsentry/issue/RUST-242)
Create a client report aggregator in `sentry-core`. The aggregator lives on the `EnvelopeSender` next to the `Transport`. The `EnvelopeSender` itself is owned by a `Client`. The aggregator is entirely lock-free. Instead of the map shape suggested by the [SDK-wide client reports spec](https://develop.sentry.dev/sdk/telemetry/client-reports/#aggregation), the aggregator stores a counter for every discard reason/data category combination. There are currently [15 discard reasons](https://develop.sentry.dev/sdk/telemetry/client-reports/#discard-reasons-1) and [15 data categories](https://develop.sentry.dev/sdk/foundations/transport/rate-limiting/#definitions) allowed by the spec, some of which may be inapplicable to the Rust SDK. If we implement all of these in the SDK, the aggregator will consume approximately 1.8 KiB of fixed memory per instance (15 * 15 64-bit atomic counters). The alternative would be a HashMap keyed by `(category, reason)`. A HashMap would consume less memory while the aggregator is empty, but recording losses would require synchronization and could allocate memory. The fixed counter array is a reasonable tradeoff for now, and we can revisit the representation if this becomes a practical concern. Resolves [#1002](#1002) Resolves [RUST-154](https://linear.app/getsentry/issue/RUST-154/add-shared-client-report-aggregator-and-send-client-reports-option)
Add a public `ClientReportRecorder` handle for transports to record lost Sentry data without exposing the full client-report aggregator. Pass the recorder through `TransportOptions` when the SDK client builds a transport. Recorded losses are aggregated into future `client_report` envelope items, so transports can report drops without sending extra requests. Keep backwards-compatible transport construction paths working by using a no-op recorder when no client-report aggregator is available. Built-in transports will start calling the recorder in follow-up PRs. Resolves [#1148](#1148) Resolves [RUST-223](https://linear.app/getsentry/issue/RUST-223)
Record lost envelopes in the `reqwest` transport. Also, add necessary loss reason enum variants. Closes [#1151](#1151) Closes [RUST-226](https://linear.app/getsentry/issue/RUST-226)
Record lost envelopes in the `curl` transport after the transport thread accepts them for sending. Curl execution failures are recorded as `network_error`; response-code lookup failures and non-`429` HTTP `4xx`/`5xx` responses are recorded as `send_error`. Keep existing rate-limit handling for `429` responses and the existing payload-too-large debug message for `413` responses. Closes [#1152](#1152) Closes [RUST-227](https://linear.app/getsentry/issue/RUST-227)
Record lost envelopes in the `ureq` transport after the transport thread accepts them for sending. Ureq request failures are recorded as `network_error`; non-`429` HTTP `4xx`/`5xx` responses are recorded as `send_error`. Keep existing rate-limit handling for `429` responses and the existing payload-too-large debug message for `413` responses. Closes [#1153](#1153) Closes [RUST-228](https://linear.app/getsentry/issue/RUST-228)
Record envelopes dropped before the curl and ureq HTTP transports see them. The std transport thread now reports queue overflow, disconnected worker, and rate-limit drops through the client report recorder. Shutdown losses remain unreported because shutdown ends the opportunity to send another envelope carrying the aggregated client report. Fixes [#1149](#1149) Fixes [RUST-224](https://linear.app/getsentry/issue/RUST-224)
Co-authored-by: Daniel Szoke <szokeasaurusrex@users.noreply.github.com>
Co-authored-by: Daniel Szoke <szokeasaurusrex@users.noreply.github.com>
cursor
Bot
changed the title
szokeasaurusrex
force-pushed
the
szokeasaurusrex/transport-thread-client-reports
branch
4 times, most recently
from
96a15e5 to
11f9014
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a standalone markdown audit focused on drop paths that are not currently covered by the Rust SDK client reports implementation. The document explicitly excludes loss tracking in the tokio transport Thread, and avoids treating the absence of a client-report disable option or unused discard reasons as standalone gaps.
Issues
Reminders
feat:,fix:,ref:,meta:, etc.)