KAFKA-20845: Don't flush empty batch when appending large records in group coordinator (4.2) - #23207
Merged
Merged
Conversation
…group coordinator (4.2) When appending records, we create a new batch if no batch exists. KAFKA-19760 introduced a flush of any existing batch when appending large records, to maximize our chances of compressing the records under the max.message.bytes. When these two things happen in the same append operation, we flush an empty batch. Flushing an empty batch is written to fail and revert the coordinator state. This would be harmless, except some group coordinator operations update the coordinator state directly without replay. Upon appending their records and triggering the empty batch flush, their state changes are then reverted while their records are written. The group coordinator's in-memory state diverges from the on-disk state and subsequent writes for the group can be invalid for the on-disk state. eg. we may have a consumer group downgraded to a classic group, followed by consumer group records which is invalid. Do not flush empty batches when appending large records. Backport of apache#22969 to 4.2. The test is adapted to the `CoordinatorRuntime` API of this branch. (cherry picked from commit 475e0a1)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When appending records, we create a new batch if no batch exists.
KAFKA-19760 introduced a flush of any existing batch when appending
large records, to maximize our chances of compressing the records under
the max.message.bytes. When these two things happen in the same append
operation, we flush an empty batch.
Flushing an empty batch is written to fail and revert the coordinator
state. This would be harmless, except some group coordinator operations
update the coordinator state directly without replay. Upon appending
their records and triggering the empty batch flush, their state changes
are then reverted while their records are written. The group
coordinator's in-memory state diverges from the on-disk state and
subsequent writes for the group can be invalid for the on-disk state.
eg. we may have a consumer group downgraded to a classic group, followed
by consumer group records which is invalid.
Do not flush empty batches when appending large records.
Backport of #22969 to 4.2. The test is adapted to the
CoordinatorRuntimeAPI of this branch.Reviewers: Chia-Ping Tsai chia7712@gmail.com