Skip to content

connection: keep a partial v5 segment header across reads - #1141

Open
fruch wants to merge 1 commit into
scylladb:masterfrom
fruch:fix-v5-partial-segment-header
Open

fruch wants to merge 1 commit into
scylladb:masterfrom
fruch:fix-v5-partial-segment-header

Conversation

@fruch

@fruch fruch commented Oct 20, 2026

Copy link
Copy Markdown

With protocol v5, a read that ends 1-5 bytes into a segment header loses those bytes, and the connection is then defuncted with CRC mismatch on header.

Cause

Connection._process_segment_buffer() has three cases:

  1. A whole segment is buffered: decode it.
  2. A full header but an incomplete payload is buffered: rewind with seek(0) and wait for more.
  3. Less than one header (6 bytes) is buffered: return without rewinding.

process_io_buffer() then calls reset_io_buffer(), which keeps only the bytes after the current position. In case 3 the position is at the end, so the partial header is discarded. The next read is parsed from the wrong offset and its header fails the CRC24 check.

The fix adds the missing seek(0) to case 3.

How it was found

The new Cassandra CI job in #1135 failed once on Cassandra 5.0 + asyncio, in test_pool_management, during a large schema refresh:

cassandra.connection.ConnectionShutdown: CRC mismatch on header 85184f. Received 1f0000", computed 9e9992.

It needs a socket read to end inside a header, so it's rare. It's more likely with the asyncio reactor, which processes the buffer after every 4 KB read, than with libev, which reads until the socket is empty. It doesn't affect Scylla, which negotiates down to v4, but it affects default-configured clients on Cassandra 4.0+.

The same code is in apache/cassandra-python-driver.

Testing

  • New unit test test_v5_segment_header_split_across_reads feeds two segments split at every offset from 0 to 7 bytes into the second header. On master, offsets 1-5 fail. With the fix, all pass.
  • Against a 3-node Cassandra 5.0.9 cluster, with the driver forced to read 5 bytes at a time, test_pool_management on v5 failed 10/10 runs without the fix and passed 10/10 with it.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: QUIET
  • Plan: Essentials
  • Run ID: a85b8d71-2c84-4b5a-876c-f34b2c058266
📥 Commits

Reviewing files that changed from the base of the PR and between 7676d46 and 90f8d45.

📒 Files selected for processing: 2
  • cassandra/connection.py
  • tests/unit/test_connection.py

Included review availability: This review used your included allowance. 3 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.

Walkthrough

When a complete segment header is unavailable, _process_segment_buffer rewinds the buffer to retain partial header bytes. A protocol v5 regression test checks that two uncompressed segments are processed across split reads without defuncting the connection.

Suggested reviewers: dkropachev

Priority: Normal

Change: Bug fix

Merge Risk: Minimal · up to 90f8d

The partial-header fix and its regression test have no identified merge-blocking issue; merge after normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly identifies the affected connection behavior and the fix for partial protocol v5 segment headers.
Description check Passed The description explains the cause, fix, affected scenarios, regression test, and validation results. It does not include the template checklist or a Fixes annotation, but the required change and test…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description explains the cause, fix, affected scenarios, regression test, and validation results. It does not include the template checklist or a Fixes annotation, but the required change and testing information are substantially complete.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@qodo-scylladb

qodo-scylladb Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Great, no issues found!

Qodo reviewed your code and found no material issues that require review
Tip of the day
💡 Did you know, you can enable the Remediation agent and Qodo fixes findings in a dedicated fix PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗


Powered by Qodo

@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.76%. Comparing base (7676d46) to head (201b324).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1141      +/-   ##
==========================================
+ Coverage   86.63%   86.76%   +0.13%     
==========================================
  Files          44       44              
  Lines       18987    19006      +19     
  Branches     2963     2965       +2     
==========================================
+ Hits        16449    16491      +42     
+ Misses       1864     1844      -20     
+ Partials      674      671       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@fruch

fruch commented Oct 22, 2026

Copy link
Copy Markdown
Author

The same bug is in the Apache driver. I confirmed it with the same unit test on trunk (3.30.1) and sent the fix there: apache#1308.

(drafted with Claude Code)

With protocol v5 framing, _process_segment_buffer() returned without
rewinding the io buffer when fewer than 6 bytes (one segment header plus
its CRC24) had arrived. process_io_buffer() then called
reset_io_buffer(), which keeps only the bytes after the current
position, and the position was at the end, so those 1-5 header bytes
were dropped. The next read was parsed from the wrong offset and the
connection was defuncted with "CRC mismatch on header".

It needs a socket read to end inside a segment header, so it shows up
rarely and mostly with the asyncio reactor, which processes the buffer
after every 4 KB read. It was seen in CI against Cassandra 5.0 during a
large schema refresh.

Rewind in that branch too, like the "header but incomplete payload"
branch already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fruch
fruch force-pushed the fix-v5-partial-segment-header branch from 90f8d45 to 201b324 Compare October 25, 2026 07:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant