Skip to content

DTLS 1.3 ClientHello reassembly before cookie, shared cookie secret, bounded DTLS receive list - #11989

Open
julek-wolfssl wants to merge 10 commits into
wolfSSL:masterfrom
julek-wolfssl:dtls13_frag_ticket_review
Open

julek-wolfssl wants to merge 10 commits into
wolfSSL:masterfrom
julek-wolfssl:dtls13_frag_ticket_review

Conversation

@julek-wolfssl

Copy link
Copy Markdown
Member

This series makes DTLS 1.3 servers reassemble a fragmented first ClientHello before verifying the cookie, lets servers of one context share a cookie secret, bounds the DTLS receive list by bytes, and fixes data races on the default ticket key context. It also adds two CI jobs.

ClientHello reassembly before the cookie

  • A stateless DTLS 1.3 server used to drop every fragment of a first ClientHello, so clients that fragment it (post-quantum key share, small MTU) failed silently. RFC 9147 Section 5.5 requires a receiver to buffer or process a fragment.
  • The first ClientHello is now reassembled and the cookie check runs on the complete message. No key, group or transcript state is kept until a cookie verifies.
  • Fragments are held in a pool owned by WOLFSSL_CTX, sharded by a hash of the peer address (WOLFSSL_DTLS_CH_POOL_SHARDS), so any stateless server of the context can complete a ClientHello another one started. The pool is keyed only on the remote address bytes reported by the I/O layer.
  • WOLFSSL_DTLS_RX_UNVERIFIED_MAX_BYTES (64 KiB by default) bounds all unverified bytes across shards. When short, the least recently progressed entry is evicted, never the storing peer's own. Entries expire after the server's retransmission maximum, and conflicting fragments replace the entry.
  • When a cookie verifies, the peer's held fragments become part of the connection's message, so the client need not resend them. The first fragment of a DTLS 1.2 ClientHello with a valid cookie is now stored as well.
  • New API: wolfSSL_dtls13_reassemble_ch_before_cookie(), wolfSSL_CTX_dtls13_reassemble_ch_before_cookie() and wolfSSL_CTX_dtls13_clear_ch_pool(). It is on by default. wolfSSL_dtls13_allow_ch_frag() and wolfSSL_dtls13_no_hrr_on_resume() now return BAD_FUNC_ARG for a NULL ssl. The example server gains --reassemble-before-cookie [n].

Shared cookie secret

  • Cookie secrets were per WOLFSSL, so a second ClientHello handed to a new listener failed. Each server context now holds a key per cookie type (HelloVerifyRequest and HelloRetryRequest) under a rwlock, and a context without a secret gets a random one when its first object enables cookies.
  • New API: wolfSSL_CTX_set_cookie_secret(), wolfSSL_CTX_set_cookie_secret_secondary() and wolfSSL_CTX_rotate_cookie_secret(). Rotation keeps earlier cookies valid; set does not. An object may still override with its own secret.
  • DoTls13ClientHello() checks the cookie before the SNI callback so it is checked with the right context's secret.

Bounded DTLS receive list

  • The receive list was bounded by 20 entries regardless of size, and a fragment it couldn't take ended the connection, even from a plaintext record anyone can send.
  • WOLFSSL_DTLS_RX_MAX_BYTES now bounds the retained bytes (default MAX_HANDSHAKE_SZ plus 16 KiB), tracked in dtls_rx_msg_list_bytes. The fragment bucket cap moves to DtlsMsgStore(), which checks before allocating.
  • DtlsMsgStore() distinguishes held, dropped (DTLS_MSG_STORE_DROPPED), contradictory and error results, and DtlsMsgStoreFatal() decides which end the connection. Plaintext fragments that don't fit are dropped without an ACK so the sender resends.
  • Dtls13ProcessBufferedMessages() drops stale entries and detaches the entry it processes. wolfSSL_clear() now frees the DTLS 1.3 receive list.

Ticket key context races

  • DefTicketEncCb() read and chose keys without the lock, racing with set-up and wolfSSL_CTX_set_tlsext_ticket_keys().
  • The lock is now a wolfSSL_RwLock. Readers pick and copy the key under the read lock and run the AEAD on the copy outside it. Writers take the write lock and re-check, since a read lock can't be upgraded. The copy is zeroized after use.
  • Tickets run in parallel only with --enable-rwlock; otherwise the lock is a mutex held for the copy.

CI

  • New DTLS 1.3 interop job against OpenSSL s_client and s_server (with and without cookies, post-quantum groups, small MTUs, KeyUpdate), built with --enable-curve25519.
  • New ThreadSanitizer job with --enable-rwlock running the dtls, dtls13, tls13 and session groups. One report fails the job.

Each change comes with tests covering its behavior, bounds and concurrency.

Copilot AI balanced review requested due to automatic review settings November 4, 2026 13:58
@julek-wolfssl julek-wolfssl self-assigned this Nov 4, 2026
@julek-wolfssl
julek-wolfssl requested a review from rizlik November 4, 2026 14:02
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

MemBrowse Memory Report · Explore what changed →

gcc-arm-cortex-m3

  • FLASH: .text +16 B (+0.0%, 132,229 B / 262,144 B, total: 50% used)

gcc-arm-cortex-m4-dtls13

  • FLASH: .rodata +72 B, .text +1,344 B (+0.7%, 198,432 B / 1,048,576 B, total: 19% used)

gcc-arm-cortex-m4-openssl-compat

  • FLASH: .rodata +24 B, .text +1,216 B (+0.2%, 803,932 B / 1,048,576 B, total: 77% used)

gcc-arm-cortex-m4-pkcs7

  • FLASH: .text +128 B (+0.1%, 224,020 B / 262,144 B, total: 85% used)

gcc-arm-cortex-m4-pq

  • FLASH: .text +128 B (+0.0%, 314,064 B / 1,048,576 B, total: 30% used)

gcc-arm-cortex-m4-rsa-only

  • FLASH: .text +128 B (+0.0%, 344,464 B / 1,048,576 B, total: 33% used)

gcc-arm-cortex-m4-tls13

  • FLASH: .text +128 B (+0.1%, 250,786 B / 262,144 B, total: 96% used)

gcc-arm-cortex-m7-pq

  • FLASH: .text +128 B (+0.0%, 314,896 B / 1,048,576 B, total: 30% used)

gcc-arm-cortex-m7-tls13

  • FLASH: .text +128 B (+0.1%, 250,786 B / 262,144 B, total: 96% used)

No memory changes detected for:


Catch memory regressions in CI with MemBrowse. Free for open source.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Pool handoffs bypass receive-list budget checks, allowing fragmented messages to exceed configured limits, and the digest fallback breaks supported builds without SHA-1 or SHA-256.

2 open issues
What changed in this PR

This PR improves wolfSSL’s DTLS handshake handling for fragmented ClientHellos and shared server contexts, bounds receive buffering, and addresses concurrent access to session-ticket keys.

Changes:

  • Add context-owned ClientHello reassembly and shared cookie-secret APIs.
  • Introduce byte-based DTLS receive limits and lock-protected ticket-key copies.
  • Expand API tests, documentation, example options, and interoperability/concurrency CI.
File Description
wolfssl/​ssl.h Declare cookie-secret and reassembly APIs.
wolfssl/​internal.h Define pool structures, budgets, and shared-key state.
wolfcrypt/​src/​misc.c Enable hashing helpers for pool sharding.
tests/​api/​test_tls13.h Register context cookie-secret coverage.
tests/​api/​test_tls13.c Test shared TLS 1.3 cookie secrets.
tests/​api/​test_ssl_cert.c Add NULL-argument API checks.
tests/​api/​test_session.h Register threaded ticket tests.
tests/​api/​test_session.c Test concurrent ticket use and key generation.
tests/​api/​test_dtls13.h Register reassembly, pool, and concurrency tests.
tests/​api/​test_dtls.h Register cookie and receive-budget tests.
src/​tls13.c Resolve shared cookie keys and reorder cookie validation.
src/​ssl.c Clear receive buffers and handle context-secret changes.
src/​ssl_api_ext.c Protect ticket-key access with read/write locks.
src/​ssl_api_dtls.c Implement context cookie-secret management.
src/​internal.c Implement bounded buffering, shared pools, and ticket locking.
src/​dtls13.c Reassemble unverified ClientHellos and expose pool controls.
src/​dtls.c Use shared cookie keys and transfer verified fragments.
examples/​server/​server.c Add pre-cookie reassembly option.
doc/​dox_comments/​header_files/​ssl.h Document APIs and behavioral changes.
configure.ac Clarify fragmentation help and configuration output.
CMakeLists.txt Clarify fragmentation option description.
ChangeLog.md Record handshake, buffering, and concurrency changes.
.github/​workflows/​tsan.yml Add ThreadSanitizer test job.
.github/​workflows/​openssl-dtls13.yml Add OpenSSL DTLS 1.3 interoperability workflow.
.github/​scripts/​openssl-dtls13.sh Exercise fragmentation, cookies, groups, and KeyUpdate.

Review effort: Balanced

Comment thread src/dtls13.c Resolved
Comment thread wolfssl/internal.h Resolved
@julek-wolfssl
julek-wolfssl force-pushed the dtls13_frag_ticket_review branch from 3bb1667 to cc99327 Compare November 4, 2026 17:31
@julek-wolfssl

Copy link
Copy Markdown
Member Author

retest this please

@julek-wolfssl
julek-wolfssl force-pushed the dtls13_frag_ticket_review branch 2 times, most recently from e96d7ad to bd2906b Compare November 5, 2026 11:00

@rizlik rizlik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work! I've done a first round of review and I have several suggestions:

  1. If reassembly before cookie is enabled (wolfSSL_dtls13_reassemble_ch_before_cookie()), always do reassembly of the full CH, without trying to check the cookie on the first fragment first. If reassembly before cookie is disabled, keep doing the first fragment cookie check with no reassembly.
    Checking the first fragment buys very little, if anything, but complicates the code considerably, because fragments need to be handed off between the shared list and the per-connection one. If the reassembly is done on the shared list, the stateful object can start with an empty list.

  2. Use just one mutex and one LRU list.

    • A flood that justifies the sharding will probably stall on eviction in the shared list rather than on lock contention. Having multiple listeners on the same socket is by itself a not very common use case (indeed not supported by other SSL stacks); a single mutex and a single LRU will already allow that, and I'm not convinced that the lock contention or the atomic operations can degrade performance.
    • Besides the unnecessary complexity, my other strong objection to this is the introduction of the dependency on atomics.
    • It will allow shared code for both lists. Together with this, I'd make the eviction and error policies uniform between the two lists:
      • Errors/conflicts on encrypted records -> error. This can happen only in the per-conne
      • On eviction, always drop the oldest one. In the attacking scenario this changes "first to arrive wins" into "fastest wins". I'm not sure that it's the riy new security guarantee (an ofsrupt the connection inthe pre-auth window), but it's not worse than what we have either, and it's a more sensible default.
  3. Drop the plaintext fragments stored in the list when receiving the first authenticated packet. This protects against a stale plaintext message sitting in the list breaking the connection much later on.

  4. Cookie secret: now the secret lives as long as the context and is never rotated unless the application calls wolfSSL_CTX_rotate_cookie_secret(). Before, the secret was per listener and was effectively rotated every time a new listener was created. Also wolfSSL_DTLS_SetCookieSecret()/wolfSSL_send_hrr_cookie() with a NULL secret now switch to the context secret instead of generating a random one for the object

DefTicketEncCb() checked whether the key context was set up, and chose
an encryption key, without holding the context's lock. A thread that
found the keys ready then used the name, keys and expiry with nothing
ordering it after the thread that set them up. The same reads raced
with wolfSSL_CTX_set_tlsext_ticket_keys(), which replaces them under the
lock. Only decryption held the lock, for the whole AEAD.

Make the lock a wolfSSL_RwLock. The IV is made before any lock.
Readers choose the key under the read lock, copy it, and run the AEAD
on the copy outside the lock. Set up, key generation and
wolfSSL_CTX_set_tlsext_ticket_keys() take the write lock. A read lock
can't be upgraded, so a reader that needs a change takes the write lock
and checks again: another thread may have done it. The copy is zeroized
after use.

Tickets only run in parallel with --enable-rwlock. Otherwise the lock is
a mutex, held just for the copy.

Tests: threads use one context's callback while another sets the keys,
and threads that find no key valid for encryption race for the write
lock: one key is made and every ticket decrypts.
The receive list of out-of-order and fragmented handshake messages was
bounded by DTLS_POOL_SZ entries, 20, whatever their size, and a
fragment the list could not take ended the connection, also from a
plaintext record anyone can send.

- WOLFSSL_DTLS_RX_MAX_BYTES bounds the bytes the list retains, default
  MAX_HANDSHAKE_SZ plus 16 KiB: message and fragment metadata and
  payload. The sum is kept in dtls_rx_msg_list_bytes and changed where
  the list changes, by DtlsMsgRxListInsert(), DtlsMsgRxListDetach(),
  DtlsMsgListRemove(), DtlsMsgListPurge() and DtlsMsgRxListFree(). An
  entry's size is computed from its counters, without walking its buckets.
- The per-message cap on fragment buckets, WOLFSSL_DTLS_FRAG_POOL_SZ,
  moves from DtlsMsgSet() to DtlsMsgStore(), which checks the projected
  size and bucket count before allocating anything.
- DtlsMsgStore() returns 0 when the fragment is held, a duplicate
  included, DTLS_MSG_STORE_DROPPED when it is not and that is no error,
  SEQUENCE_ERROR for a fragment that contradicts the message held, or
  an error. DtlsMsgStoreFatal() says which results end the connection:
  only running out of memory, or a peer whose records authenticate
  exceeding the budget. A plaintext fragment that does not fit, or
  reaches past its message, is dropped and not acknowledged, so its
  sender resends it.
- Dtls13ProcessBufferedMessages() drops an entry older than the next
  expected message instead of stalling on it, and detaches the entry it
  processes, so processing that empties the list frees nothing in use.
  Dtls13ProcessBufferedMsg() processes one message.
- wolfSSL_clear() frees the DTLS 1.3 receive list, which
  FreeHandshakeResources() leaves to the DTLS 1.3 code.
- Tests: the budget in DTLS 1.2 and 1.3, the counter against the list,
  which store results are fatal, wolfSSL_clear(), and the fragment
  flood tests sized to the budget.
…rified

A stateless DTLS 1.3 server dropped every fragment of a first
ClientHello, so clients that fragment it (a post-quantum key share, a
small MTU) failed in silence. RFC 9147 Section 5.5 says a receiver MUST
buffer or process a fragment.

- Reassemble the first ClientHello before its cookie is verified and run
  the cookie check on the complete message. No key, group or transcript
  state is kept until a cookie verifies.
- New wolfSSL_dtls13_reassemble_ch_before_cookie() and
  wolfSSL_CTX_dtls13_reassemble_ch_before_cookie() turn it off, and
  wolfSSL_CTX_dtls13_clear_ch_pool() frees what is held. On by default,
  and ClientHello reassembly is on for every WOLFSSL_DTLS_CH_FRAG
  server, not only with ML-KEM. Like wolfSSL_dtls13_allow_ch_frag(),
  they return WOLFSSL_FAILURE on a client.
- The fragments are held in a pool owned by WOLFSSL_CTX, so any
  stateless server made from the context can complete a ClientHello
  another one started: with one listener per handshake, a second
  client's partial ClientHello would otherwise be lost when the first
  client's cookie verified. The pool is split into
  WOLFSSL_DTLS_CH_POOL_SHARDS shards (default 8, 1 with
  SINGLE_THREADED) picked by a hash of the peer address, each with its
  own list and lock, so different peers rarely contend. A thread holds
  one shard lock at most. A shard costs a context 64 bytes on a 64-bit
  threaded build, 16 bytes single-threaded. Only a DTLS context
  initializes the pool, and the pool refuses stores and takes until it
  is, so an object moved to another kind of context by
  wolfSSL_set_SSL_CTX() holds nothing. The pool needs atomics in a
  threaded build and a clock in any (NO_ASN_TIME is not supported), and
  fails to compile without.
- Key the pool only on the remote address the I/O layer reports, as
  the bytes it reports: wolfSSL_dtls_set_peer() takes any bytes a
  custom transport chooses, so nothing in them is interpreted, and
  sockAddrEqual() is only for addresses our own I/O callbacks made. For
  those the bytes are stable: EmbedReceiveFrom() zeroes the buffer, and
  Linux and FreeBSD zero sin_zero and sin6_flowinfo and derive
  sin6_scope_id from the address and interface. An empty address never
  matches. A server that does not know the peer's address holds
  nothing: it drops such a fragment without an ACK. A ClientHello whose
  cookie is in its first fragment still completes without an address,
  statefully.
- When a cookie verifies, only that peer's entry leaves the pool. When
  the verified message is the first fragment of a ClientHello, the
  peer's held fragments of the same message (epoch, sequence number,
  type and size) become a message of the connection and the fragment
  completes it, so the server flight goes out without the client
  resending them. DoClientHelloStateless() takes the size of the
  fragment in hand and of the whole ClientHello for this, and the pool
  frees the peer's entry with DtlsChPoolDrop() or hands back the rest
  with DtlsChPoolTakeRest(). What the entry holds in the verified
  fragment's range is cut away, since the receive list keeps the bytes
  it already has when a fragment overlaps them: held bytes never
  replace verified ones, and the rest completes the message.
- Hold only a first fragment the cookie check could not judge: one with
  no cookie in it, or one that ends before the fields the check reads.
  One whose cookie failed gets the fatal illegal_parameter alert a
  whole ClientHello gets, and whatever the pool held for its peer goes.
- WOLFSSL_DTLS_RX_UNVERIFIED_MAX_BYTES bounds every unverified byte of
  a context, across all shards: 64 KiB by default, 20 to 30 ClientHellos
  with post-quantum key shares. A pool entry is bounded on its own by
  MAX_HANDSHAKE_SZ. The budget is an atomic counter. A store reserves
  the most its fragment can add, a new entry with one bucket, before it
  takes any lock, evicts for that bound when it passes the budget, then
  locks its shard once, stores, and settles the reservation to the bytes
  really added. The bound over-reserves, and so may over-evict, by the
  headers when the entry exists and by the fragment when it repeats
  bytes already held; genuine fragments are disjoint, so they come close
  to it. The largest message the pool completes is the budget less twice
  the entry overhead, since its last fragment is reserved as a new entry
  on top of the entry holding the rest. The counter may pass the budget
  by one bound per store evicting at once; a static assert leaves room
  for 64.
- When the budget is short, evict the least recently progressed entry
  of the whole pool, ordered by a context-wide progress value an entry
  takes under the shard lock whenever it gains bytes. Each shard
  publishes the progress of its two oldest entries at every unlock, so
  eviction finds the oldest shard without locks. The storing peer's own
  entry is never evicted: when the scan lands on it, the entry behind it
  in that shard goes, and the shard is judged by its second value from
  then on. The victim is rechecked under its shard's lock, where a
  stale value is a miss. An entry that cannot fit by itself evicts
  nothing. Frees happen outside the shard lock.
- An entry that gains no bytes for the storing server's retransmission
  maximum (wolfSSL_dtls_set_timeout_max()) expires, and a fragment
  whose bytes conflict with those held replaces the entry, so stale or
  forged bytes do not join a later ClientHello.
- Never pool a whole ClientHello. One the pool completes joins the
  receive list and is drained like any other message, which also covers
  one left pending by asynchronous crypto.
- Entries are allocated from the context's heap, which every pool call
  is given, and rebuilt on the server's heap only when the two differ.
  The pool allocates only while servers run, by when static memory
  loaded into the context is in place.
- The first fragment of a DTLS 1.2 ClientHello whose cookie verifies is
  stored, and the rest the pool holds for it taken, where before it was
  ignored and the client had to resend the message after its timeout.
- A second ClientHello carrying another server's cookie fails unless
  the servers share the cookie secret.
- wolfSSL_dtls13_allow_ch_frag() and wolfSSL_dtls13_no_hrr_on_resume()
  return BAD_FUNC_ARG for a NULL ssl.
- Example server: --reassemble-before-cookie [n].
- Document the fragmentation controls and WOLFSSL_DTLS_CH_FRAG.
- Tests: reassembly, interleaved peers, eviction, bounds, forged,
  planted and stale fragments, a foreign datagram mid-handshake, two
  listeners, second ClientHello hand-off with and without a shared
  secret, shared budget, expiry under the storing server's maximum,
  conflicting fragments, fragments without an address, addresses
  differing in one byte or in length or only in sin_zero being
  different peers, a whole ClientHello under asynchronous
  crypto, a first fragment with a corrupted cookie, a pooled tail
  reaching into the first fragment that verifies, freeing a
  listener or the context with entries held, wolfSSL_set_SSL_CTX(), a
  TLS context having no pool, static memory loaded before and after the
  context is made, a DTLS 1.2 second ClientHello in two fragments,
  entries in different shards sharing the budget and eviction across
  shards, the bound settling to the bytes added, a repeated fragment
  evicting for its bound, eviction of the entry behind the storing
  peer's own and of other shards' entries around it, a fragment refused
  by the bucket cap,
  clearing the pool, many concurrent peers checked against the budget
  by a monitor thread, concurrent stores, and two threads running full
  handshakes through one context's pool. Those threads start after one
  handshake, because wolfCrypt sets its CPU feature flags on first use
  without a lock.
With and without the cookie exchange, post-quantum groups, small MTUs
and KeyUpdate. wolfSSL is built with --enable-curve25519, which is off
by default, so X25519MLKEM768 is available. OpenSSL 4.1.0 is the first
release with DTLS 1.3.
…ride

Cookie secrets were per WOLFSSL, so a server could not verify a cookie
another server of the same context issued. A second ClientHello handed
to a new listener failed: a fatal alert in DTLS 1.3, a new
HelloVerifyRequest in DTLS 1.2.

- Every server context holds a cookie secret, as a key per cookie type,
  an HMAC of a label under the secret, so a HelloVerifyRequest cookie
  never verifies as a HelloRetryRequest cookie. It keeps only the keys,
  hvrKey and hrrKey, under a leaf rwlock: every cookie a server without
  a secret of its own makes or checks reads it, a few API calls write
  it. TLS 1.3 HelloRetryRequest cookies use it as the DTLS ones do.
- New wolfSSL_CTX_set_cookie_secret(),
  wolfSSL_CTX_set_cookie_secret_secondary() and
  wolfSSL_CTX_rotate_cookie_secret(). Rotate moves the primary to the
  secondary, so cookies issued before it keep verifying. Set discards
  the previous key: a set between a cookie's issue and its check fails
  that handshake and the client retries. A secret of any non-zero size,
  as the object setters take; a FIPS HMAC still needs 14 bytes or more.
  NULL generates a random secret as long as the keys.
- A context without a secret gets a random one the first time one of
  its objects enables cookies: accept, wolfSSL_enable_cookie(), or NULL
  to wolfSSL_send_hrr_cookie() or wolfSSL_DTLS_SetCookieSecret().
  Installed under the lock only if still empty, so of racing defaults
  the first stays. The servers of one context thus verify each other's
  cookies with no API call. NULL to those two functions drops the
  object's own secret and uses the context's.
- CookieSecretGet() resolves the secrets at each use: the object's own
  secret for the cookie type when the application set one, else the
  context's keys copied under its lock, primary and secondary. The
  object's buffers hold only application-supplied secrets, so nothing is
  copied ahead or kept in step. No secret anywhere fails creating and
  checking a cookie with COOKIE_ERROR.
- DoTls13ClientHello() checks the cookie before the SNI callback, where
  wolfSSL_set_SSL_CTX() usually runs: checked after the move it would
  be checked with the new context's secret. wolfSSL_set_SSL_CTX() gives
  a new context without a secret a default when the object is a server
  with cookies on.
- Tests: hand-off of a second ClientHello between listeners in DTLS 1.2
  and 1.3 and of a HelloRetryRequest cookie between two TLS 1.3 servers,
  rotation keeping cookies valid and set not, per-object override,
  objects created before the context's secret was set,
  wolfSSL_set_SSL_CTX(), no splicing between cookie types, device and
  RNG failures, no secret anywhere, rotation racing verification on
  threads, and a TLS context having the lock.
The ClientHello pool, the context cookie secret and the ticket key
context are shared between threads, and their threaded tests only prove
anything with a race detector. Build with -fsanitize=thread and
--enable-rwlock, so readers run in parallel, and run the dtls, dtls13,
tls13 and session groups. One report fails the job. Two minutes locally.

Checked that a report fails the run: with DtlsChShardLock() and
DtlsChShardUnlock() made no-ops, the same command reports a race in
DtlsChPoolFind() from test_dtls13_ch_pool_threads and exits 66.
@julek-wolfssl
julek-wolfssl force-pushed the dtls13_frag_ticket_review branch from bd2906b to b9b9ff0 Compare November 12, 2026 14:52
…nticate

A plaintext handshake fragment stored in the receive list, from the
peer or anyone else, stayed there after the peer's records started to
authenticate and could get in the way of the handshake later. Now a
DTLS 1.3 endpoint drops the epoch 0 messages of its receive list when
it processes an authenticated record while the list holds any. Protected
messages stay. test_dtls_stale_partial_head() now expects the plaintext
entry it plants far ahead in a client's receive list to be gone once the
server's first authenticated record arrives.
wolfSSL_CTX_dtls13_set_ch_pool_max_bytes() sets the bytes a context holds
for ClientHellos before their cookie verifies. The budget was the compile
time WOLFSSL_DTLS_RX_UNVERIFIED_MAX_BYTES alone, and a server facing the
internet sits at it, so sizing it meant rebuilding. The macro is now the
initial value, kept in the pool as an atomic that stores read lock-free.
A store reads the budget again after reserving, so one lowered meanwhile
is seen. Lowering it evicts down to the new budget at once: a store evicts
no more than its own bound, so stores alone would never get the pool back
under a lowered budget. The example server takes --ch-pool-max-bytes.
Built with NO_WOLFSSL_SERVER there is no pool to size or clear, so the
setter and wolfSSL_CTX_dtls13_clear_ch_pool() return WOLFSSL_FAILURE.
The budget is read with the atomic load directly and the receive list
reads WOLFSSL_DTLS_RX_MAX_BYTES directly: DtlsChPoolLimit and
DtlsMsgRxLimit are gone.
A server that pools ClientHellos, reassembly before the cookie on and a
peer address known, now sends every fragment of an unverified ClientHello
to the context's pool, the first included. DtlsChPoolStore() runs the
cookie check in place on the entry's contiguous head, with the shard
locked, once the head grew to a size that at least doubles each time, so
a message is parsed O(log n) times whatever its fragments. A verified
entry leaves the pool in the same step and moves whole, later fragments
included, to the connection's receive list; a bad cookie frees the entry
and the server answers illegal_parameter; no cookie, or a head that ends
before it, keeps pooling. A message that completes in the pool goes
through the whole-message path as before. A server that pools nothing,
reassembly off or no address, still checks the cookie on a first fragment
as it arrives and keeps it.

The stateless check is split: ClientHelloStatelessVerdict() reads the
object only and returns the verdict, ClientHelloStatelessCommit() turns
the object stateful and marks the replay window. DoClientHelloStateless()
does both for a whole message, with the stateless reply when there is no
cookie. DoClientHelloStatelessHead() is the verdict on a prefix, for the
pool: const object, no I/O, no pool access, and only the cookie secret's
leaf lock, so the lock order is shard lock, then cookie secret lock. The
caller commits once the pool has handed the entry out.
DtlsChPoolTakeRest and the trimming of the verified range go.
Put the issue time, 4 byte big endian seconds from LowResTimer(), under
the MAC of both stateless cookies: at the end of the HelloRetryRequest
cookie data and in front of the DTLS 1.2 HelloVerifyRequest MAC. After
the MAC verifies, a cookie older than WOLFSSL_COOKIE_MAX_AGE seconds
(60 by default, 0 disables the check) or from the future takes the
invalid-cookie path: illegal_parameter for TLS 1.3 and DTLS 1.3, a new
HelloVerifyRequest for DTLS 1.2. A captured cookie thus cannot be
replayed later (RFC 9147 Section 5.1). The DTLS 1.2 cookie stays 32
bytes on the wire, the time and the leftmost 28 bytes of the HMAC-SHA256
(SP 800-107 truncation), so deployed clients capped at WOLFSSL_COOKIE_LEN
keep working; the SHA-1 fallback keeps its full 20. Without a clock
(NO_ASN_TIME) no time is sent or checked. CreateDtls12Cookie() takes the
issue time and writes the whole cookie. Tests move the clock through
wc_SetTimeCb() and check that a DTLS 1.3, DTLS 1.2 and TLS 1.3 server
each reject a stale cookie and stay stateless, and verify it with the
check disabled.
@julek-wolfssl

Copy link
Copy Markdown
Member Author

1. Full reassembly in the pool lets a port-cycling flood evict a client that has already proven its address, so I kept an early exit instead: the pool checks the cookie in place on the contiguous head as it grows, and a verified entry leaves the pool whole in the same locked step. This prioritizes the peers that passed a routability check.

2. I am keeping the sharded pool for this PR. I benchmarked 8 shards against 1 on a 4-core box: the benchmarks showed substantial benefit in eviction heavy scenarios which is what I predict for real servers.

3. Done: the first authenticated record frees the plaintext entries of the receive list, with a test for the planted-Certificate case.

4. There is no cryptographic limit worth rotating for, so I bounded replay directly: both cookies carry their issue time under the MAC and expire after 60 seconds, the DTLS 1.2 cookie staying 32 bytes. NULL to the object setters still selects the context's secret.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants