Repository navigation
RHINENG-33842: fix aggregator performance issues - #2395
MichaelMraka wants to merge 17 commits into
Conversation
Reviewer's GuideSchema and migrationsAdd the persisted recount timestamp and supporting account/advisory index, recording both in the schema snapshot and migration history.
Data structuresExpose the persisted aggregate update timestamp and retain subsecond precision in advisory update event timestamps.
APIChange the reader and handler contracts to support batches, retry them as a unit, and configure batching when spawning readers.
Add a keyed Kafka writer API so account-keyed advisory events are routed consistently.
Uses of the APIUpdate evaluator and listener consumers for the batch-handler contract, and publish advisory updates through keyed writers.
ImplementationReplace the in-memory advisory buffer with batch decoding and deterministic deduplication by account, workspace, and advisory, retaining the newest event timestamp.
Recount workspace-scoped advisory aggregates under a per-key transaction lock, skip stale events, remove empty rows, and propagate fatal processing or publishing errors.
Track prior inventory workspace state and emit advisory recount events for both old and new workspaces after a system moves.
Collect and commit Kafka messages in bounded batches, preserving message tracing and committing offsets only after successful handling.
ConfigurationAdd the emergency advisory-processing bypass, raise the default flush wait, and pass batch size and timeout into advisory readers.
DocumentationDocument the account advisory timestamp alongside the existing aggregate-table description.
TestsCover advisory event timestamp precision, deduplication, recount correctness, stale-event handling, locking, and workspace-specific aggregation.
Test keyed advisory event serialization and account-based Kafka keys.
Exercise batch-handler behavior, batching and timeout boundaries, trace propagation, cancellation, commits, and failures.
Adapt Kafka reader tests to batches and verify keyed routing and batch retry behavior.
Update evaluator event expectations and Kafka round-trip coverage for keyed publishing and batched reads.
Adapt upload tests to the expanded return value and verify recount events for both workspaces after a move.
Update mock readers to implement the batch reader interface.
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've found 11 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="aggregator/events.go" line_range="141-143" />
<code_context>
+ producedAt time.Time) error {
+ return database.DB(ctx).Transaction(func(tx *gorm.DB) error {
+ var row models.AccountAdvisory
+ err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).
+ Where("rh_account_id = ? AND workspace_id = ? AND advisory_id = ?", rhAccountID, workspaceID, advisoryID).
+ Take(&row).Error
+ if err != nil && !errors.Is(err, gorm.ErrRecordNotFound) {
+ return err
</code_context>
<issue_to_address>
**Concurrent recounts overwrite newer counts**
When multiple consumers recount the same key while its aggregate row is absent, including when a recount finds no matching systems, `SELECT ... FOR UPDATE` in `recountAdvisory` locks no row for an absent key, so the recounts can race: an older upsert can overwrite newer counts, or a delayed delete can remove a fresh row. The aggregate then stays wrong or missing until another update.
Serialize recounts by account, workspace, and advisory even when no aggregate row exists, covering both upserts and deletes.
Also at `aggregator/events.go:147`, `aggregator/events.go:165-167`, `aggregator/events.go:178-181`.
</issue_to_address>
### Comment 2
<location path="aggregator/events.go" line_range="86" />
<code_context>
- for _, id := range e.AdvisoryIDs {
- sets[e.RhAccountID][id] = struct{}{}
+ for _, advisoryID := range event.AdvisoryIDs {
+ key := advisoryRecountKey{event.RhAccountID, event.WorkspaceID, advisoryID}
+ if existing, ok := newest[key]; !ok || producedAt.After(existing) {
+ newest[key] = producedAt
</code_context>
<issue_to_address>
**Moved systems leave old workspace counts**
When a system changes workspaces and an advisory update is emitted for it, `groupAdvisoryUpdates` keys recounts only by the workspace in the event, and `advisoryCountSQL` recounts only that workspace. The moved system is therefore removed from neither the old workspace’s aggregate nor its counts, so that workspace continues showing stale totals.
Include the system’s previous workspace in the update or otherwise trigger a recount for affected old-workspace/advisory keys.
Also at `aggregator/events.go:37-38`.
</issue_to_address>
### Comment 3
<location path="aggregator/events.go" line_range="147" />
<code_context>
+ if err != nil && !errors.Is(err, gorm.ErrRecordNotFound) {
+ return err
+ }
+ if err == nil && row.Updated.After(producedAt) {
+ utils.LogWarn(
+ "rh_account_id", rhAccountID,
</code_context>
<issue_to_address>
**Later events are skipped after early recounts**
When an evaluator update is not yet visible when its event is recounted, or a later event is produced while an earlier recount is still running, `recountAdvisory` sets `Updated` to processing time, then compares it with event `ProducedAt`; the earlier recount can therefore make the later event look old even though its source changes were not visible. The later event is skipped and aggregate counts stay stale.
Do not use recount processing time as proof that an event's source changes were included; use a freshness mechanism that accounts for source transaction visibility.
</issue_to_address>
### Comment 4
<location path="aggregator/events.go" line_range="147" />
<code_context>
+ if err != nil && !errors.Is(err, gorm.ErrRecordNotFound) {
+ return err
+ }
+ if err == nil && row.Updated.After(producedAt) {
+ utils.LogWarn(
+ "rh_account_id", rhAccountID,
</code_context>
<issue_to_address>
**Same-second events are skipped**
When an aggregate is recounted and a later update for the same key is produced in the same second, `ProducedAt` is serialized to whole seconds while `row.Updated` retains subsecond precision, so the later event compares as older and `recountAdvisory` skips it. Callers see stale aggregate counts.
Preserve timestamp precision in `ProducedAt` or compare both timestamps at the same precision.
</issue_to_address>
### Comment 5
<location path="aggregator/events.go" line_range="147" />
<code_context>
+ if err != nil && !errors.Is(err, gorm.ErrRecordNotFound) {
+ return err
+ }
+ if err == nil && row.Updated.After(producedAt) {
+ utils.LogWarn(
+ "rh_account_id", rhAccountID,
</code_context>
<issue_to_address>
**Aggregate writes leave freshness marker stale**
When the SQL refresh, account-advisory backfill, or an older application version updates aggregate counts without updating `updated`, `recountAdvisory` treats the unchanged `Updated` value as evidence that the row is old, so a stale event can overwrite the refreshed counts and leave the aggregate incorrect.
Ensure every path that writes aggregate counts advances the freshness marker used by this guard, or stop treating that marker as proof of freshness.
Also at `base/models/models.go:312`.
</issue_to_address>
### Comment 6
<location path="database_admin/migrations/171_account_advisory_updated.up.sql" line_range="2" />
<code_context>
+ALTER TABLE account_advisory
+ ADD COLUMN IF NOT EXISTS updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW();
</code_context>
<issue_to_address>
**Pending updates leave stale counts**
When an existing aggregate is stale and its corresponding Kafka event was produced before the migration but is still pending, `recountAdvisory` treats the migration-assigned `updated` timestamp as proof that a row is newer than the event and skips the recount, leaving its old aggregate counts in place.
Recount existing rows during migration or initialize their timestamps so pending pre-migration events cannot be mistaken for already-applied updates.
Also at `aggregator/events.go:147-156`.
</issue_to_address>
### Comment 7
<location path="aggregator/events.go" line_range="147" />
<code_context>
+ if err != nil && !errors.Is(err, gorm.ErrRecordNotFound) {
+ return err
+ }
+ if err == nil && row.Updated.After(producedAt) {
+ utils.LogWarn(
+ "rh_account_id", rhAccountID,
</code_context>
<issue_to_address>
**Missing timestamps suppress recounts**
When an advisory update event omits `produced_at`, `groupAdvisoryUpdates` maps an absent `ProducedAt` to the zero time, so an existing row with a nonzero `Updated` always satisfies this guard. The event is acknowledged without recounting, leaving its aggregate stale.
Reject or explicitly handle events without `produced_at` rather than comparing them as zero time.
</issue_to_address>
### Comment 8
<location path="base/mqueue/mqueue_impl_gokafka.go" line_range="47" />
<code_context>
- utils.LogTrace("key", h.Key, "value", string(h.Value), "kafka message header")
- }
+
+ handlerCtx := ctx
+ if len(batch.app) == 1 {
+ handlerCtx = batch.singleCtx
</code_context>
<issue_to_address>
**Batched work loses message trace context**
When a Kafka fetch contains more than one message, `handleMessageBatches` passes the root `ctx` to handlers instead of a message-derived context, so downstream work loses the Kafka message trace parent and batch processing is disconnected in traces.
Pass a message-derived context to handlers for multi-message batches instead of the root context.
</issue_to_address>
### Comment 9
<location path="aggregator/events.go" line_range="132" />
<code_context>
+ if err := sonic.Unmarshal(m.Value, &event); err != nil {
+ utils.LogError("err", err, "could not deserialize advisory update event")
+ continue
+ }
+ events = append(events, event)
+ }
</code_context>
<issue_to_address>
**Notifications can be published twice**
When a notification publish succeeds but marking rows notified or committing the transaction fails, `MakeRetryingHandler` retries the batch after this fatal error, but `publishNewAdvisoryNotification` publishes to Kafka before marking rows notified and committing its transaction. If marking or commit fails after the publish succeeds, the retry finds the rows unnotified and publishes the same notification again.
Make notification delivery and the notified-state update idempotent, for example by using an outbox or a stable notification deduplication key.
</issue_to_address>
### Comment 10
<location path="aggregator/events.go" line_range="39" />
<code_context>
+ AND sa.rh_account_id = ?
+ AND si.workspace_id = ?
+ AND sa.advisory_id = ?
+ GROUP BY sa.advisory_id, sa.rh_account_id, si.workspace_id`
-func flushAdvisoryBuffer() {
</code_context>
<issue_to_address>
**Recounts repeatedly scan account data**
When a batch contains many distinct advisories for an account, `processAdvisoryBatch` calls `recountAdvisory` once per distinct advisory, and this filter has no supporting index: `system_advisories` is partitioned by account but its primary key is `(rh_account_id, system_id, advisory_id)`. Batches with many advisories repeatedly scan the account partition, increasing database work and delaying queue processing.
Recount multiple advisories in one grouped query per account, or add an index that supports the account-and-advisory lookup.
</issue_to_address>
### Comment 11
<location path="aggregator/events_test.go" line_range="182" />
<code_context>
+ Where("rh_account_id = 1 AND workspace_id = ? AND advisory_id = 1", recountWorkspace),
+ "systems_installable")
+ assert.Equal(t, expected, got)
+ assert.NotEqual(t, 99, got)
+
+ assert.Nil(t, database.TestDB().Create(&models.AccountAdvisory{
</code_context>
<issue_to_address>
**Deleted aggregate passes recount assertions**
When `recountAdvisory` deletes the row instead of updating it for an older or equal timestamp, `TestRecountRunsWhenUpdatedIsOlderOrEqual` uses `assert.NotEqual` at lines 182 and 189, so a recount that incorrectly deletes the row passes because `PluckInt` returns zero for no row. The test therefore does not establish that either recount updated the aggregate.
Assert the row still exists and its count equals the expected current count after each recount.
Also at `aggregator/events_test.go:189`.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 11 findings to address first, and with bypass enabled, the handler returns success without processing advisory events, so Kafka commits them; any resulting recounts or notifications are then skipped and cannot be recovered from those messages. Reverting stops future bypassing, but does not restore already committed events.
Blocking findings: aggregator/events.go:143, aggregator/events.go:86, aggregator/events.go:147, aggregator/events.go:147, aggregator/events.go:147, and 6 more
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2395 +/- ##
==========================================
+ Coverage 66.82% 67.22% +0.39%
==========================================
Files 157 157
Lines 9080 9210 +130
==========================================
+ Hits 6068 6191 +123
- Misses 3010 3017 +7
Partials 2 2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
cd4a403 to
e628378
Compare
|
@sourcery-ai review |
There was a problem hiding this comment.
Hey - I've found 8 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="aggregator/events.go" line_range="31-32" />
<code_context>
+SELECT count(sa.*) FILTER (WHERE sa.status_id = 0) AS systems_installable,
+ count(sa.*) AS systems_applicable
+ FROM system_advisories sa
+ JOIN system_inventory si
+ ON sa.rh_account_id = si.rh_account_id AND sa.system_id = si.id
+ WHERE si.stale = FALSE
+ AND sa.rh_account_id = ?
</code_context>
<issue_to_address>
**Unevaluated systems inflate totals**
When a non-stale system has advisory rows but no completed evaluation, `advisoryCountSQL` joins `system_advisories` to `system_inventory` without requiring a matching `system_patch` row with `last_evaluation IS NOT NULL`, so `recountAdvisory` includes unevaluated systems and writes inflated applicable and installable totals to `account_advisory`.
Join or filter on `system_patch` so only systems with a completed evaluation contribute to the recount.
Also at `aggregator/events.go:33`.
</issue_to_address>
### Comment 2
<location path="database_admin/migrations/171_account_advisory_updated.up.sql" line_range="2" />
<code_context>
+ALTER TABLE account_advisory
+ ADD COLUMN IF NOT EXISTS updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW();
</code_context>
<issue_to_address>
**Queued updates leave stale counts**
When an existing aggregate is stale and an event produced before the migration is still queued, the migration’s `DEFAULT NOW()` stamps existing `account_advisory` rows with migration time, not the time their counts were computed. `recountAdvisory` then skips the queued event because its `ProducedAt` is earlier, leaving stale counts visible.
Add the column without stamping existing rows as freshly updated, so queued events can recount them.
Also at `aggregator/events.go:161`.
</issue_to_address>
### Comment 3
<location path="aggregator/events.go" line_range="190" />
<code_context>
+ WorkspaceID: workspaceID,
+ SystemsApplicable: counts[0].SystemsApplicable,
+ SystemsInstallable: counts[0].SystemsInstallable,
+ Updated: time.Now(),
+ }
+ return database.OnConflictUpdateMulti(tx,
</code_context>
<issue_to_address>
**Fresh evaluations leave stale totals**
When a recount reads old committed data before an evaluator update commits, or the aggregator clock is ahead of the evaluator clock, `recountAdvisory` stamps the aggregate with the aggregator’s `time.Now()` rather than the event’s `ProducedAt`. The timestamp can be later than the event that should correct the stale count, so the timestamp guard skips that recount and leaves the aggregate stale.
Set `Updated` from the event’s `ProducedAt` instead of the aggregator’s wall-clock time.
Also at `aggregator/events.go:161`, `evaluator/advisory_update.go:53`.
</issue_to_address>
### Comment 4
<location path="aggregator/events.go" line_range="85" />
<code_context>
- for _, id := range e.AdvisoryIDs {
- sets[e.RhAccountID][id] = struct{}{}
+ for _, advisoryID := range event.AdvisoryIDs {
+ key := advisoryRecountKey{event.RhAccountID, event.WorkspaceID, advisoryID}
+ if existing, ok := newest[key]; !ok || producedAt.After(existing) {
+ newest[key] = producedAt
</code_context>
<issue_to_address>
**Former workspace retains moved systems**
When a system changes workspace and emits an advisory update, `groupAdvisoryUpdates` keys the recount by the event’s current workspace, and `recountAdvisory` only recounts that workspace. The former workspace’s cached advisory count is not recomputed and continues to include the moved system.
Include the former workspace in the update and recount its advisory totals as well as the current workspace’s.
Also at `aggregator/events.go:38`.
</issue_to_address>
### Comment 5
<location path="aggregator/events.go" line_range="130" />
<code_context>
+ for rhAccountID, advisoryIDs := range accountAdvisories {
if err := publishNewAdvisoryNotification(rhAccountID, advisoryIDs); err != nil {
utils.LogError("err", err, "rh_account_id", rhAccountID, "failed to publish new advisory notification")
+ return base.WrapFatalKafkaError(err, "publish new advisory notification")
}
}
</code_context>
<issue_to_address>
**Notifications can be published twice**
When the notification write succeeds but the database update or transaction commit fails, `processAdvisoryBatch` returns a fatal error after notification publication fails downstream, so the retry handler reruns the batch. If the notification was published but `markAdvisoriesNotified` or the transaction commit failed, the retry sees the advisory as unnotified and publishes it again.
Make notification publication idempotent or use an outbox so retrying the batch cannot emit a duplicate notification.
</issue_to_address>
### Comment 6
<location path="database_admin/migrations/171_account_advisory_updated.up.sql" line_range="2" />
<code_context>
+ALTER TABLE account_advisory
+ ADD COLUMN IF NOT EXISTS updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW();
</code_context>
<issue_to_address>
**Refreshes leave timestamps stale**
When the cache refresh or backfill SQL updates an existing account-advisory row, the refresh and backfill SQL upserts update aggregate counts without updating `updated`, while the new timestamp guard treats that column as the time the row was written. After one of those refreshes, an older event can pass the guard and recount the row instead of recognizing the newer aggregate write.
Update `updated` in every SQL path that changes account-advisory aggregate data, including the refresh function's conflict update.
</issue_to_address>
### Comment 7
<location path="aggregator/events.go" line_range="36" />
<code_context>
+ WHERE si.stale = FALSE
+ AND sa.rh_account_id = ?
+ AND sa.advisory_id = ?
+ AND si.rh_account_id = ?
+ AND si.workspace_id = ?
+ GROUP BY sa.advisory_id, sa.rh_account_id, si.workspace_id`
</code_context>
<issue_to_address>
**Per-advisory recounts scan account rows**
When an account has many system-advisory rows and a batch recounts multiple advisories, `advisoryCountSQL` filters by account and advisory, but the `system_advisories` primary key orders columns as `(rh_account_id, system_id, advisory_id)` and has no account/advisory index. PostgreSQL therefore scans that account’s advisory rows for each recount, multiplying work across the batch and degrading aggregation performance for large accounts.
Add an index on `system_advisories (rh_account_id, advisory_id)` to support the recount lookup.
</issue_to_address>
### Comment 8
<location path="aggregator/events_test.go" line_range="201" />
<code_context>
+ Where("rh_account_id = 1 AND workspace_id = ? AND advisory_id = 1", recountWorkspace),
+ "systems_installable")
+ assert.Equal(t, expected, got)
+ assert.NotEqual(t, 99, got)
+
+ assert.Nil(t, database.TestDB().Create(&models.AccountAdvisory{
</code_context>
<issue_to_address>
**Deleted recounts pass the test**
When a recount deletes the existing row even though its source systems still produce counts, when source systems still produce counts but a broken recount deletes the row, `TestRecountRunsWhenUpdatedIsOlderOrEqual` reads zero through `PluckInt` and its `assert.NotEqual(t, 99, got)` passes, so the test accepts missing aggregate data.
Assert that the row exists and that its count equals the expected source count after each recount.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 8 findings to address first, and a faulty recount can write or delete workspace aggregate rows, but those derived counts can be rebuilt. The less reversible risk is publishing an incorrect advisory notification to Kafka; reverting the code cannot retract a notification already consumed.
Blocking findings: aggregator/events.go:32, database_admin/migrations/171_account_advisory_updated.up.sql:2, aggregator/events.go:190, aggregator/events.go:85, aggregator/events.go:130, and 3 more
…of an in-memory buffer.
in case of emergency we can turn it on and just clear the queue without actually processing messages
…of the account-wide SQL refresh.
… is younger than the message.
…handles each account.
…t so a later same-second event is not skipped.
…lti-message batches.
…the source count.
…ounts avoid an account scan.
…stem moves so both aggregate counts are recounted.
Secure Coding Practices Checklist GitHub Link
Secure Coding Checklist
Summary by Sourcery
Improve advisory aggregation performance and consistency by processing Kafka updates in batches and safely refreshing workspace-scoped advisory counts.
New Features:
Bug Fixes:
Enhancements:
Deployment:
Documentation:
Tests:
Chores: