Repository navigation
fix(native): zstd compression fails with NoSuchFieldError in the native image - #501
Merged
Merged
Conversation
…ve image zstd-jni's native code reads and writes Java fields (srcPos, dstPos, nativePtr, ...) through GetFieldID. jni-config.json registered only the methods, so every zstd producer died on its first batch with NoSuchFieldError: ZstdOutputStreamNoFinalizer.dstPos. Registering the fields was not enough on its own: the file sat at the root of META-INF/native-image/, where the native build never reads it. It now lives under META-INF/native-image/com.bmscomp/kates/. native-e2e.sh runs its LOAD test with zstd as well as lz4, so a codec that only breaks in the native image fails the script.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the native image every benchmark using
compression=zstdends FAILED. Each producer dies on its first batch:zstd-jni's native code reads and writes Java fields (
srcPos,dstPos,nativePtr, ...) throughGetFieldID. The native build kept the methods but not those fields. lz4 and snappy are unaffected: their native code never touches fields.Fix
jni-config.jsonregistersallDeclaredFieldson every zstd class, and addsBaseZstdBufferDecompressingStreamNoFinalizer, which holds the inheritedconsumed/producedfields.META-INF/native-image/toMETA-INF/native-image/com.bmscomp/kates/. At the root the native build never read it, so registering the fields alone changed nothing. A rebuild with only that change still threw the same error.scripts/native-e2e.shruns its 10k-record LOAD test with both lz4 and zstd.Verification
Built
kates:native-localfrom this branch on thepandakind cluster (SCRAM Kafkakrafter) and deployed it as a throwawaykates-e2erelease:b8f1b243771b7686The pod logged no
NoSuchFieldError. Before the move, the same zstd test failed with it.The full
make native-e2escript did not get as far as the LOAD test on that cluster. The chart'shelm testfails earlier on/q/openapi -> 401, which is unrelated to this change, so I ran the script's LOAD commands by hand.Not in this PR
reflect-config.jsonandresource-config.jsonsit in the same unread location, so they have never applied either. That explains the note inCompressionReflectionConfigthatreflect-config.json"did not take". Moving them changes what goes into the build, so it's left for a separate change.native-e2e.shprints logs from the postgres pod instead of kates, because its label selector matches both.