Skip to content

fix: verify shared request queue is_finished against per-request reads - #1087

Open
vdusek wants to merge 3 commits into
masterfrom
worktree-fix-test-2
Open

fix: verify shared request queue is_finished against per-request reads#1087
vdusek wants to merge 3 commits into
masterfrom
worktree-fix-test-2

Conversation

@vdusek

@vdusek vdusek commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

test_request_queue_is_finished_and_is_empty[shared] failed on master
(run 31375518173)
with assert not True.

In shared access mode is_finished() answered solely from the queue head listing, which is eventually
consistent: right after a committed add_request the head can come back empty with no locked requests, so the
queue briefly reported itself as finished. That is a client defect, not just a test problem - a crawler polling
is_finished() in that window would shut down with work still in the queue.

is_finished() now confirms an empty-head verdict before reporting True: every request this client knows
about and has not yet seen handled is re-checked with a strongly consistent get_request read. A request that
is missing (not yet propagated) or unhandled (pending, or locked by another client) keeps the queue unfinished;
confirmed-handled requests are remembered, so each one is verified at most once. An in-flight
add_batch_of_requests call also keeps the queue unfinished. Only the shared client changes - the single
client already tracks added requests in its local head estimation, so it is consistent locally.

The test-side polling from the previous revision is reverted: the strict negative is_finished assertions are
back and serve as regression coverage for the client-side guarantee.

Verified by injecting the propagation lag deterministically (first head listing after each add returns an
empty, lock-free head without hitting the API): the unfixed client reproduces the exact CI failure, the fixed
one passes 10/10 runs with strict assertions. New unit tests cover the verification paths, and the whole
integration module passes under --numprocesses=auto.

✍️ Drafted by Claude Code

@vdusek vdusek added adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. labels Aug 11, 2026
@vdusek vdusek self-assigned this Aug 11, 2026
@github-actions github-actions Bot added this to the 147th sprint - Tooling team milestone Aug 11, 2026
@github-actions github-actions Bot added the tested Temporary label used only programatically for some analytics. label Aug 11, 2026
@codecov

codecov Bot commented Aug 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.33%. Comparing base (3212e1c) to head (e7cd6b4).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1087      +/-   ##
==========================================
+ Coverage   92.24%   92.33%   +0.08%     
==========================================
  Files          51       51              
  Lines        3249     3260      +11     
==========================================
+ Hits         2997     3010      +13     
+ Misses        252      250       -2     
Flag Coverage Δ
e2e 35.15% <7.14%> (-0.09%) ⬇️
integration 56.68% <64.28%> (+0.02%) ⬆️
unit 86.19% <100.00%> (+1.67%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek
vdusek requested a review from Pijukatel August 11, 2026 08:02
@vdusek
vdusek marked this pull request as ready for review August 11, 2026 08:02
@Pijukatel

Copy link
Copy Markdown
Contributor

Maybe the is_finished when returning True, should be double-checked(or inspected in other way) due to eventual consistency to prevent something like this:
https://apify.slack.com/archives/C0L33UM7Z/p1785322628993809

@vdusek vdusek changed the title test: deflake is_finished assertions in shared request queue mode fix: verify shared request queue is_finished against per-request reads Aug 14, 2026
Comment on lines +367 to +377
for request_id, cached_request in list(self._requests_cache.items()):
if cached_request.was_already_handled:
continue

request = await self._get_request_by_id(request_id)
if request is None or request.handled_at is None:
return False

cached_request.was_already_handled = True

return True

@Pijukatel Pijukatel Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussion points:

  1. In a normal scenario, are there many requests being double-checked like this, or will most of the requests already short-circuit this function due to cached_request.was_already_handled?
  2. Maybe it would make sense to check those requests in parallel instead of one by one?
    (not to be blocked by await self._get_request_by_id(request_id) in for loop)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants