serde: support AWS Glue Schema Registry Avro deserialization - #2648
Open
cesarbruschetta wants to merge 5 commits into
Open
serde: support AWS Glue Schema Registry Avro deserialization#2648cesarbruschetta wants to merge 5 commits into
cesarbruschetta wants to merge 5 commits into
Conversation
Add PAYLOAD_ENCODING_AVRO_GLUE to the PayloadEncoding enum so that messages framed with the AWS Glue Schema Registry wire format can be requested and reported as a distinct encoding. The generated Go and TypeScript bindings, the serde constant and the mapper cases are included in the same commit because the exhaustive linter requires every switch over PayloadEncoding to handle the new value as soon as it exists.
Introduce a GlueSchemaRegistry configuration block that enables the AWS Glue Schema Registry integration and controls the region, an optional custom endpoint, the schema cache TTL and the client timeout. Credentials are resolved through the default AWS SDK credential chain, so no secrets are read from the Console configuration file.
Add a client that resolves Avro schemas from the AWS Glue Schema Registry by schema version ID and caches them in memory, so that a deserializer does not call the API once per record. The client honours an optional custom endpoint to allow pointing it at a local stub, and keeps failed lookups cached for a short period to avoid hammering the API for unknown schema versions. Name validation is relaxed when parsing a schema definition. Glue does not enforce the Avro strict name regex when a schema is registered, so names that the spec rejects are common in practice: CDC producers in particular derive namespaces from database identifiers and end up with hyphens. Rejecting those would make a schema unreadable even though AWS accepted it and the producer wrote records against it. Names are only identifiers for decoding and pass through verbatim. Schema resolution is logged, including whether the schema came from the cache, so that a failing lookup can be diagnosed from the logs.
Decode Avro records framed with the AWS Glue Schema Registry wire format: a version byte, a compression byte, the 128-bit schema version UUID and the Avro payload. Uncompressed and zlib-compressed payloads are supported. Deserialization only. Producing records in this format is not implemented, so SerializeObject reports that it is unsupported. The serde is only registered when a registry is configured. The client is held in an interface variable rather than a concrete pointer so that a disabled registry yields a nil interface: storing a nil pointer in an interface produces a non-nil interface holding a nil pointer, which would defeat the check.
Expose the new encoding in the deserializer selectors of the messages tab and map it to the avroGlue value used by the REST interfaces, so that a user can force it on a topic whose records are not detected automatically.
cesarbruschetta
force-pushed
the
feature/avro-glue-deserializer
branch
from
September 10, 2026 19:19
35bdae5 to
ba6b857
Compare
karguimaraes
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Console can already deserialize Avro records framed by the Confluent Schema
Registry, but records produced by the AWS Glue Schema Registry use a different
wire format and currently show up as raw bytes. This adds a deserializer for
that format.
The Glue framing is: a version byte, a compression byte, the 128-bit schema
version UUID in big-endian order, then the Avro payload. The schema is resolved
from the Glue API by that version ID and cached in memory, so the API is not
called once per record.
Scope is deliberately narrow:
SerializeObjectreturns an explicit "notsupported" error; producing in this format is not implemented.
0x00) and zlib (0x05).credential fields are read from the Console configuration.
notes below.
Configuration
regionis required when enabled. The block is documented indocs/config/console.yaml.Commits
Five commits, each building and linting clean on its own:
proto: add avroGlue payload encodingconfig: add serde.glueSchemaRegistryglue: add schema registry clientserde: add avroGlue deserializerfrontend: offer avroGlue deserializerCommit 1 bundles the proto enum, the generated Go/TS bindings, the serde
constant and the
mapper.gocases. They cannot be split: theexhaustivelinter requires every switch over
PayloadEncodingto handle a new value assoon as that value exists.
Testing
Unit tests cover the 18-byte header, UUID byte order, zlib payloads, every
error path (short payload, wrong version byte, unknown compression byte), the
Avro round-trip, cache hit/miss behaviour and config validation.
Beyond unit tests, this was validated end to end against a local Redpanda plus
a stub Glue endpoint, driving the real
ListMessagesConnect stream through acontainer built in the same mold as the published image:
{"a":27,"b":"uncompressed"}{"a":42,"b":"zlib compressed"}incorrect header version byte for avro glue7unknown compression byte 7 for avro gluepayload size is <= 18It was then run against a real MSK cluster whose topics carry Debezium CDC
records registered in Glue. That is where the name validation problem showed
up, and where the fix was confirmed.
Locally verified gates:
backend:fmt,backend:lint(0 issues),backend:test-unit(31/31),go vet -tags=integration,buf lint,buf format,buf breakingagainstmaster, frontendtype:check,build,test:unit(995),test:federation(1),test:integration(1425).Each of the five commits was also verified in isolation (build + lint clean)
using a detached worktree, so the series is bisectable.
Notes for reviewers
licenses/third_party_go.csvwas edited by hand rather than regenerated.Regenerating rewrites ~100 unrelated lines because the checked-in file is
stale (it still lists
hamba/avroandlinkedin/goavro, and is missinggithub.com/twmb/avro). This change touches only thegithub.com/aws/*lines. Happy to regenerate wholesale in a separate PR if you prefer.
go.modpromotesgithub.com/aws/aws-sdk-go-v2from indirect to direct andadds
service/glue. No new non-AWS dependency: the Avro codec isgithub.com/twmb/avro, already required.avro.WithLaxNames. Glue does notenforce the Avro strict name regex (
[A-Za-z_][A-Za-z0-9_]*) when a schemais registered, so definitions the spec rejects are common in practice. CDC
producers in particular derive namespaces from database identifiers and end
up with hyphens, e.g.
some-db.public.some-table-v0. Parsing strictly makessuch a schema unreadable even though AWS accepted it and the producer wrote
records against it, which is how this surfaced against a real cluster. Names
are only identifiers for decoding and pass through verbatim, so this does not
affect the wire format or the canonical form. Covered by a test case.
Worth flagging that the Confluent path has the same limitation today:
pkg/schema/client.goparses without that option, so a CDC schema with ahyphenated namespace would fail there too. Left alone as out of scope, but
happy to align it if you would like.
serde.GlueSchemaRegistryClientis held in an interface variable inconsole.NewService, not a*glue.Client, so that a disabled registryyields a nil interface rather than a non-nil interface holding a nil pointer,
which would defeat the nil check in
serde.NewService. This differs from thebsrClienta few lines above, which uses the concrete-pointer pattern and sohas the latent version of that problem. Not touched here to keep the diff
scoped.
avroGlueis deliberately absent fromencodingOptionsintopic-produce.tsx. That list drives serialization, and this change onlyimplements deserialization, so offering it there would surface an option that
always fails. It is registered in the three deserializer lists:
topics/messages/constants.ts(PAYLOAD_ENCODING_PAIRS),Tab.Messages/constants.tsandTab.Messages/modals/deserializers-modal.tsx.Open questions
SerializeObjectbe implemented? Producing would require pickinga schema version, which is a bigger design decision than deserialization.
deserializer already records it in
ExtraMetadata, but it never reaches thefrontend:
KafkaRecordPayload.schema_idis anint32and cannot hold aUUID, and there is no field for arbitrary metadata. Exposing it needs a new
proto field, so it is left out here.