Skip to content

cluster: report a timeout waiting for schema agreement, not a defunct connection - #1150

Open
nyh wants to merge 1 commit into
scylladb:masterfrom
nyh:fix-1317-timeout-message
Open

nyh wants to merge 1 commit into
scylladb:masterfrom
nyh:fix-1317-timeout-message

Conversation

@nyh

@nyh nyh commented Oct 26, 2026 •

Copy link
Copy Markdown

When a request times out while the driver waits for schema agreement after a schema change, the driver reports OperationTimedOut: errors={'Connection defunct by heartbeat': 'Client request timeout. ...'}. No connection failed, and the server did apply the schema change. This PR makes the driver report the existing message for this case, "Request timed out while waiting for schema agreement", which until now could not be reached.

The cause, found by Yauheni Khatsianevich in SCYLLADB-1317:

  • When a response arrives, the connection removes its request from _requests.
  • For a schema change result, ResponseFuture then waits for schema agreement in another thread, and the request's timer keeps running.
  • If the timer fires during that wait, _on_timeout() doesn't find the request in _requests. The KeyError branch for that case was added for PYTHON-1044, a connection defuncted by a heartbeat failure, which also empties _requests. So it always reported "Connection defunct by heartbeat".

The fix checks is_schema_agreed in that branch. That flag is false only while a schema change result waits for schema agreement, and in that case the branch now reports the schema agreement message. The heartbeat case keeps its message. The new unit test fails without the fix.

This message shows up in many Scylla CI failures, each with its own real cause, for example a DROP KEYSPACE that was slow under load in an SCT test. It sent their analysis looking for connection problems.

Fixes SCYLLADB-1317

… connection

This patch fixes the misleading error of a request that times out while
the driver waits for schema agreement after a schema change. The driver
reported "Connection defunct by heartbeat", although no connection failed
and the server had applied the schema change.

When a request's response arrives, the connection removes the request from
its queue. For a schema change result, the driver then waits for schema
agreement in another thread before it completes the request, and the
request's timer keeps running. When the timer fires during that wait,
_on_timeout() doesn't find the request in the connection's queue. The
KeyError branch for that case was added (PYTHON-1044) for a connection
that a heartbeat failure defuncted, which also empties the queue, so it
always reported "Connection defunct by heartbeat". The existing message
for this case, "Request timed out while waiting for schema agreement",
was never reached.

The fix checks is_schema_agreed in that branch, which is false only while
a schema change result waits for schema agreement, and then reports the
schema agreement message. The defunct connection case keeps its message.

This misleading message appeared in many Scylla CI failures, for example
a slow DROP KEYSPACE in SCT, and sent their analysis looking for
connection problems.

Fixes SCYLLADB-1317
@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: QUIET
  • Plan: Essentials
  • Run ID: 968d8d03-ea41-4423-ba93-8dad9114a750
Commits

Reviewing files that changed from the base of the PR and between 2ef43c8 and 53c479f.

Files selected for processing: 2
  • cassandra/cluster.py
  • tests/unit/test_response_future.py

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Walkthrough

ResponseFuture._on_timeout now reports schema-agreement timeout details when schema agreement is pending, including when the request has left the connection queue. Other defunct-connection timeouts retain their existing error details. A unit test checks the timeout message and configured timeout value.

Priority: Normal

Change: Bug fix

Merge Risk: Minimal · up to 53c47

No actionable merge-blocking risk was found in the timeout-reporting change.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed Title names the main change: a timeout while waiting for schema agreement is now reported instead of a defunct connection. Matches the raw summary and objectives.
Description check Passed Description states the problem, root cause (KeyError branch in _on_timeout reached after the response removes the request), the fix (check is_schema_agreed), the new test, and the Fixes annotation. Te…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

Description states the problem, root cause (KeyError branch in _on_timeout reached after the response removes the request), the fix (check is_schema_agreed), the new test, and the Fixes annotation. Template checklist items are not ticked, and the description does not mention commit splitting, docs, or docstrings, so those template items are unaddressed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@qodo-scylladb

qodo-scylladb Bot commented Oct 26, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Great, no issues found!

Qodo reviewed your code and found no material issues that require review
Tip of the day
💡 Did you know, you can type 'qodo, fix this' on a finding and the fix lands right on your PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗


Powered by Qodo

@codecov

codecov Bot commented Oct 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.71%. Comparing base (2ef43c8) to head (53c479f).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1150      +/-   ##
==========================================
- Coverage   86.72%   86.71%   -0.02%     
==========================================
  Files          44       44              
  Lines       19029    19033       +4     
  Branches     2968     2969       +1     
==========================================
+ Hits        16503    16504       +1     
- Misses       1858     1866       +8     
+ Partials      668      663       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant