Skip to content

serde: support AWS Glue Schema Registry Avro deserialization - #2648

Open
cesarbruschetta wants to merge 5 commits into
redpanda-data:masterfrom
cesarbruschetta:feature/avro-glue-deserializer
Open

serde: support AWS Glue Schema Registry Avro deserialization#2648
cesarbruschetta wants to merge 5 commits into
redpanda-data:masterfrom
cesarbruschetta:feature/avro-glue-deserializer

Conversation

@cesarbruschetta

@cesarbruschetta cesarbruschetta commented Sep 10, 2026

Copy link
Copy Markdown

Summary

Console can already deserialize Avro records framed by the Confluent Schema
Registry, but records produced by the AWS Glue Schema Registry use a different
wire format and currently show up as raw bytes. This adds a deserializer for
that format.

The Glue framing is: a version byte, a compression byte, the 128-bit schema
version UUID in big-endian order, then the Avro payload. The schema is resolved
from the Glue API by that version ID and cached in memory, so the API is not
called once per record.

Scope is deliberately narrow:

  • Deserialization only. SerializeObject returns an explicit "not
    supported" error; producing in this format is not implemented.
  • Compression: uncompressed (0x00) and zlib (0x05).
  • Credentials: resolved through the default AWS SDK credential chain. No
    credential fields are read from the Console configuration.
  • Name validation is relaxed when parsing a schema definition, see the
    notes below.

Configuration

serde:
  glueSchemaRegistry:
    enabled: true
    region: us-east-1
    # endpoint: ""      # optional, for a custom or local endpoint
    cacheTtl: 1h
    clientTimeout: 10s

region is required when enabled. The block is documented in
docs/config/console.yaml.

Commits

Five commits, each building and linting clean on its own:

  1. proto: add avroGlue payload encoding
  2. config: add serde.glueSchemaRegistry
  3. glue: add schema registry client
  4. serde: add avroGlue deserializer
  5. frontend: offer avroGlue deserializer

Commit 1 bundles the proto enum, the generated Go/TS bindings, the serde
constant and the mapper.go cases. They cannot be split: the exhaustive
linter requires every switch over PayloadEncoding to handle a new value as
soon as that value exists.

Testing

Unit tests cover the 18-byte header, UUID byte order, zlib payloads, every
error path (short payload, wrong version byte, unknown compression byte), the
Avro round-trip, cache hit/miss behaviour and config validation.

Beyond unit tests, this was validated end to end against a local Redpanda plus
a stub Glue endpoint, driving the real ListMessages Connect stream through a
container built in the same mold as the published image:

record result
uncompressed {"a":27,"b":"uncompressed"}
zlib {"a":42,"b":"zlib compressed"}
bad version byte incorrect header version byte for avro glue
compression byte 7 unknown compression byte 7 for avro glue
12-byte payload payload size is <= 18

It was then run against a real MSK cluster whose topics carry Debezium CDC
records registered in Glue. That is where the name validation problem showed
up, and where the fix was confirmed.

Locally verified gates: backend:fmt, backend:lint (0 issues),
backend:test-unit (31/31), go vet -tags=integration, buf lint,
buf format, buf breaking against master, frontend type:check, build,
test:unit (995), test:federation (1), test:integration (1425).

Each of the five commits was also verified in isolation (build + lint clean)
using a detached worktree, so the series is bisectable.

Notes for reviewers

  • licenses/third_party_go.csv was edited by hand rather than regenerated.
    Regenerating rewrites ~100 unrelated lines because the checked-in file is
    stale (it still lists hamba/avro and linkedin/goavro, and is missing
    github.com/twmb/avro). This change touches only the github.com/aws/*
    lines. Happy to regenerate wholesale in a separate PR if you prefer.
  • go.mod promotes github.com/aws/aws-sdk-go-v2 from indirect to direct and
    adds service/glue. No new non-AWS dependency: the Avro codec is
    github.com/twmb/avro, already required.
  • The Glue client parses schemas with avro.WithLaxNames. Glue does not
    enforce the Avro strict name regex ([A-Za-z_][A-Za-z0-9_]*) when a schema
    is registered, so definitions the spec rejects are common in practice. CDC
    producers in particular derive namespaces from database identifiers and end
    up with hyphens, e.g. some-db.public.some-table-v0. Parsing strictly makes
    such a schema unreadable even though AWS accepted it and the producer wrote
    records against it, which is how this surfaced against a real cluster. Names
    are only identifiers for decoding and pass through verbatim, so this does not
    affect the wire format or the canonical form. Covered by a test case.
    Worth flagging that the Confluent path has the same limitation today:
    pkg/schema/client.go parses without that option, so a CDC schema with a
    hyphenated namespace would fail there too. Left alone as out of scope, but
    happy to align it if you would like.
  • serde.GlueSchemaRegistryClient is held in an interface variable in
    console.NewService, not a *glue.Client, so that a disabled registry
    yields a nil interface rather than a non-nil interface holding a nil pointer,
    which would defeat the nil check in serde.NewService. This differs from the
    bsrClient a few lines above, which uses the concrete-pointer pattern and so
    has the latent version of that problem. Not touched here to keep the diff
    scoped.
  • avroGlue is deliberately absent from encodingOptions in
    topic-produce.tsx. That list drives serialization, and this change only
    implements deserialization, so offering it there would surface an option that
    always fails. It is registered in the three deserializer lists:
    topics/messages/constants.ts (PAYLOAD_ENCODING_PAIRS),
    Tab.Messages/constants.ts and Tab.Messages/modals/deserializers-modal.tsx.

Open questions

  1. Should SerializeObject be implemented? Producing would require picking
    a schema version, which is a bigger design decision than deserialization.
  2. Should the Glue schema version UUID be surfaced in the UI? The
    deserializer already records it in ExtraMetadata, but it never reaches the
    frontend: KafkaRecordPayload.schema_id is an int32 and cannot hold a
    UUID, and there is no field for arbitrary metadata. Exposing it needs a new
    proto field, so it is left out here.

Add PAYLOAD_ENCODING_AVRO_GLUE to the PayloadEncoding enum so that
messages framed with the AWS Glue Schema Registry wire format can be
requested and reported as a distinct encoding.

The generated Go and TypeScript bindings, the serde constant and the
mapper cases are included in the same commit because the exhaustive
linter requires every switch over PayloadEncoding to handle the new
value as soon as it exists.
Introduce a GlueSchemaRegistry configuration block that enables the
AWS Glue Schema Registry integration and controls the region, an
optional custom endpoint, the schema cache TTL and the client timeout.

Credentials are resolved through the default AWS SDK credential chain,
so no secrets are read from the Console configuration file.
Add a client that resolves Avro schemas from the AWS Glue Schema
Registry by schema version ID and caches them in memory, so that a
deserializer does not call the API once per record.

The client honours an optional custom endpoint to allow pointing it at
a local stub, and keeps failed lookups cached for a short period to
avoid hammering the API for unknown schema versions.

Name validation is relaxed when parsing a schema definition. Glue does
not enforce the Avro strict name regex when a schema is registered, so
names that the spec rejects are common in practice: CDC producers in
particular derive namespaces from database identifiers and end up with
hyphens. Rejecting those would make a schema unreadable even though
AWS accepted it and the producer wrote records against it. Names are
only identifiers for decoding and pass through verbatim.

Schema resolution is logged, including whether the schema came from
the cache, so that a failing lookup can be diagnosed from the logs.
Decode Avro records framed with the AWS Glue Schema Registry wire
format: a version byte, a compression byte, the 128-bit schema version
UUID and the Avro payload. Uncompressed and zlib-compressed payloads
are supported.

Deserialization only. Producing records in this format is not
implemented, so SerializeObject reports that it is unsupported.

The serde is only registered when a registry is configured. The client
is held in an interface variable rather than a concrete pointer so
that a disabled registry yields a nil interface: storing a nil pointer
in an interface produces a non-nil interface holding a nil pointer,
which would defeat the check.
Expose the new encoding in the deserializer selectors of the messages
tab and map it to the avroGlue value used by the REST interfaces, so
that a user can force it on a topic whose records are not detected
automatically.
@cesarbruschetta
cesarbruschetta force-pushed the feature/avro-glue-deserializer branch from 35bdae5 to ba6b857 Compare September 10, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants