Repository navigation
Conversation
… connection This patch fixes the misleading error of a request that times out while the driver waits for schema agreement after a schema change. The driver reported "Connection defunct by heartbeat", although no connection failed and the server had applied the schema change. When a request's response arrives, the connection removes the request from its queue. For a schema change result, the driver then waits for schema agreement in another thread before it completes the request, and the request's timer keeps running. When the timer fires during that wait, _on_timeout() doesn't find the request in the connection's queue. The KeyError branch for that case was added (PYTHON-1044) for a connection that a heartbeat failure defuncted, which also empties the queue, so it always reported "Connection defunct by heartbeat". The existing message for this case, "Request timed out while waiting for schema agreement", was never reached. The fix checks is_schema_agreed in that branch, which is false only while a schema change result waits for schema agreement, and then reports the schema agreement message. The defunct connection case keeps its message. This misleading message appeared in many Scylla CI failures, for example a slow DROP KEYSPACE in SCT, and sent their analysis looking for connection problems. Fixes SCYLLADB-1317
Code Review by Qodo🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)
Great, no issues found!Qodo reviewed your code and found no material issues that require reviewTip of the day💡 Did you know, you can type 'qodo, fix this' on a finding and the fix lands right on your PR Powered by Qodo |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1150 +/- ##
==========================================
- Coverage 86.72% 86.71% -0.02%
==========================================
Files 44 44
Lines 19029 19033 +4
Branches 2968 2969 +1
==========================================
+ Hits 16503 16504 +1
- Misses 1858 1866 +8
+ Partials 668 663 -5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a request times out while the driver waits for schema agreement after a schema change, the driver reports
OperationTimedOut: errors={'Connection defunct by heartbeat': 'Client request timeout. ...'}. No connection failed, and the server did apply the schema change. This PR makes the driver report the existing message for this case, "Request timed out while waiting for schema agreement", which until now could not be reached.The cause, found by Yauheni Khatsianevich in SCYLLADB-1317:
_requests.ResponseFuturethen waits for schema agreement in another thread, and the request's timer keeps running._on_timeout()doesn't find the request in_requests. TheKeyErrorbranch for that case was added for PYTHON-1044, a connection defuncted by a heartbeat failure, which also empties_requests. So it always reported "Connection defunct by heartbeat".The fix checks
is_schema_agreedin that branch. That flag is false only while a schema change result waits for schema agreement, and in that case the branch now reports the schema agreement message. The heartbeat case keeps its message. The new unit test fails without the fix.This message shows up in many Scylla CI failures, each with its own real cause, for example a DROP KEYSPACE that was slow under load in an SCT test. It sent their analysis looking for connection problems.
Fixes SCYLLADB-1317