Skip to content

ROX-35909: Test_OutputQueue_ExpiringMessage fails in PubSub#21903

Merged
jschnath merged 2 commits into
masterfrom
jschnath/bugfix/ROX-35909_Test_OutputQueue_ExpiringMessage_fails_in_pubsub
Jul 24, 2026
Merged

ROX-35909: Test_OutputQueue_ExpiringMessage fails in PubSub#21903
jschnath merged 2 commits into
masterfrom
jschnath/bugfix/ROX-35909_Test_OutputQueue_ExpiringMessage_fails_in_pubsub

Conversation

@jschnath

@jschnath jschnath commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Description

Fixes test flakes in Test_OutputQueue_ExpiringMessages and Test_OutputQueue_ForwardMessages that caused intermittent CI failures.

Root Cause:
In pubsub mode, event processing is asynchronous (occurs in a separate goroutine via the lane's run() method). The tests set a mock expectation, send
an event, validate the assertion, then immediately finish without waiting for async processing to complete. This creates a race: if the test completes
before the lane's goroutine processes the event, gomock cleanup detects the missing call and fails the test.

Fix:
Rewrite all output queue tests to use testing/synctest, matching the pattern already established in other pubsub async tests (e.g. detector_serializer_test.go). synctest.Test provides deterministic goroutine scheduling, and synctest.Wait() ensures all spawned goroutines have settled before assertions run — eliminating both the race condition and wall-clock timeouts.

Validation:

  • Reproduced the flake on master with go test -count=200 -parallel=8 (failed immediately)
  • Fixed version passes 200 iterations with high parallelism (600 pubsub=true executions)
  • All package tests pass (go test -count=20 with zero flakes)

Introduced in PR #20898 (ROX-34880: Refactor resolver to use PubSub)

🤖 This PR was created with AI assistance.

User-facing documentation

  • CHANGELOG.md is updated OR update is not needed ← not needed (test-only fix)
  • documentation PR is created and is linked above OR is not needed ← not needed (test-only fix)

Testing and quality

  • the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag ← test-only change
  • CI results are inspected

Automated testing

  • modified existing tests ← replaced channel+timeout synchronization with synctest

How I validated my change

  1. Reproduced the flake on master: go test -count=200 -parallel=8 failed with missing call(s) to *mocks.MockDetector.ReprocessDeployments
  2. Verified fix: Same command on fixed version passes all 200 iterations (600 pubsub=true test executions)
  3. Full package tests: go test -count=20 passes with zero flakes

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Output queue tests replace timeout-based asynchronous waits with testing/synctest, immediate forwarding assertions, and simplified mock expectations across expiration, forwarding, detector, and stop-request scenarios.

Changes

Output queue test synchronization

Layer / File(s) Summary
Synchronization assertion foundation
sensor/kubernetes/eventpipeline/output/output_test.go
Adds testing/synctest, removes the timeout constant, and replaces time-based forwarding helpers with immediate channel assertions.
Deterministic output queue scenarios
sensor/kubernetes/eventpipeline/output/output_test.go
Expiration, forwarding, detector, and stop-request tests use synctest.Wait() instead of completion channels and timeout-based waits, with non-argument-specific reprocessing expectations.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is concise and matches the main test-flake fix, though it only names one affected test.
Description check ✅ Passed The description covers the required sections, explains the fix, and includes validation and testing details.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch jschnath/bugfix/ROX-35909_Test_OutputQueue_ExpiringMessage_fails_in_pubsub

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…eue_ForwardMessages

The tests had a race condition in pubsub mode where async event processing
might not complete before test cleanup, causing gomock to detect missing
ReprocessDeployments calls.

Add synchronization using done channel and .Do() callback on mock expectation,
matching the pattern in Test_OutputQueue_DetectorCalls.

Partially generated with AI assistance.
@jschnath
jschnath force-pushed the jschnath/bugfix/ROX-35909_Test_OutputQueue_ExpiringMessage_fails_in_pubsub branch from 5410249 to b371bb9 Compare July 23, 2026 11:19
@jschnath

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 51.36%. Comparing base (b3a080f) to head (978f957).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #21903      +/-   ##
==========================================
- Coverage   51.40%   51.36%   -0.04%     
==========================================
  Files        2857     2857              
  Lines      178776   178776              
==========================================
- Hits        91891    91837      -54     
- Misses      78865    78898      +33     
- Partials     8020     8041      +21     
Flag Coverage Δ
go-unit-tests 51.36% <ø> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🚀 Build Images Ready

Images are ready for commit 4d2381e. To use with deploy scripts:

export MAIN_IMAGE_TAG=4.12.x-583-g4d2381ef8f

@jschnath

Copy link
Copy Markdown
Contributor Author

/retest

@vikin91 vikin91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea for the fix! I am proposing to go one step further (sorry, I am a big synctest fan 😉 ).

Comment thread sensor/kubernetes/eventpipeline/output/output_test.go Outdated
Comment thread sensor/kubernetes/eventpipeline/output/output_test.go
@jschnath

Copy link
Copy Markdown
Contributor Author

Okay, I've switched them to use synctest 👍

@jschnath
jschnath requested a review from vikin91 July 24, 2026 08:13

@vikin91 vikin91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks for improving!
You may want to update the PR description before merging as it still suggests that a solution with a channel is used.

@jschnath
jschnath merged commit 4d2381e into master Jul 24, 2026
104 checks passed
@jschnath
jschnath deleted the jschnath/bugfix/ROX-35909_Test_OutputQueue_ExpiringMessage_fails_in_pubsub branch July 24, 2026 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants