fix tls_bench DTLS mode failures

[miyazakh]

Fix tls_bench DTLS (-u) mode failures

Summary

Running tls_bench -u (DTLS mode) failed immediately with the following errors:

Running TLS Benchmarks...
ERROR: failed to listen
error creating ctx
Client Error: -125 (out of memory error)

And after partial fixes, a third error appeared:

error on client write
Client Error: -439 (DTLS trying to send too much in single datagram error)

This PR fixes three independent bugs in examples/benchmark/tls_bench.c.

---

Bug 1: SetupSocketAndListen called listen() on a UDP socket

Root cause

When DTLS mode is active, SetupSocketAndListen() correctly created a SOCK_DGRAM
(UDP) socket instead of a SOCK_STREAM (TCP) socket. However, setsockopt(),
bind(), and listen() were all inside the TCP-only else branch, so:

• The UDP socket was never bound to the server address.
• listen() was called unconditionally after the socket-type branch, which always
  fails on a UDP socket (EOPNOTSUPP), producing ERROR: failed to listen.

Fix

• Move setsockopt(SO_REUSEADDR) and bind() outside the TCP-only block so they
  apply to both TCP and UDP sockets.
• Guard the listen() call with #ifdef WOLFSSL_DTLS if (!doDTLS) so it is only
  called for TCP connections.

---

Bug 2: wolfSSL_CTX_new() returned NULL for TLS 1.3 ciphers in DTLS mode (when WOLFSSL_DTLS13 is not compiled in)

Root cause

When iterating over the default cipher list in DTLS mode, TLS 1.3 cipher suites
(names starting with "TLS13") were encountered. The client/server setup code
attempted to use wolfDTLSv1_3_client/server_method() inside an
#ifdef WOLFSSL_DTLS13 guard. When WOLFSSL_DTLS13 is not compiled in, no
method was assigned, leaving ctx == NULL. The subsequent NULL check returned
MEMORY_E (-125), producing:

error creating ctx
Client Error: -125 (out of memory error)

Fix

Add an early-continue in the cipher iteration loop: when DTLS mode is active and
WOLFSSL_DTLS13 is not compiled in, silently skip any TLS 1.3 cipher suite.

#if defined(WOLFSSL_DTLS) && !defined(WOLFSSL_DTLS13)
        if (doDTLS && XSTRNCMP(cipher, "TLS13", 5) == 0) {
            /* DTLS 1.3 not compiled in; skip TLS 1.3 ciphers */
            cipher = (next_cipher != NULL) ? (next_cipher + 1) : NULL;
            continue;
        }
#endif

---

Bug 3: TEST_DTLS_PACKET_SIZE exceeded the default DTLS MTU

Root cause

TEST_DTLS_PACKET_SIZE was defined as 8092 bytes (referencing MAX_UDP_SIZE in
internal.h). However, wolfSSL's SendData() checks that the total DTLS record
size fits within MAX_MTU (default: 1400 bytes). Since 8092 bytes of payload
produces a record far larger than 1400 bytes, SendData() returned
DTLS_SIZE_ERROR (-439) on every write:

error on client write
Client Error: -439 (DTLS trying to send too much in single datagram error)

MAX_UDP_SIZE (8092) is the maximum UDP datagram size wolfSSL is prepared to
receive, not the send-side MTU limit. The two constants serve different purposes
and were conflated.

Fix

Change TEST_DTLS_PACKET_SIZE for non-embedded builds from 8092 to 1200,
which fits within MAX_MTU (1400) - MAX_MSG_EXTRA (~102). A comment documents
the two compile-time options that allow larger packets:

• WOLFSSL_DTLS_MTU + wolfSSL_dtls_set_mtu() — set a per-session MTU
• WOLFSSL_NO_DTLS_SIZE_CHECK — auto-fragment writes that exceed the MTU

The automatic fallback already present in tls_bench (argTestPacketSize is
reduced to TEST_DTLS_PACKET_SIZE when DTLS mode is detected and no -p flag
is given) now produces a value that actually works with the default MTU.

---

Improvement: startup message now reflects the active protocol

The banner printed at the start of the run was always "Running TLS Benchmarks...",
even when DTLS mode (-u) was selected. It now reads "Running DTLS Benchmarks..."
when DTLS is active and "Running TLS Benchmarks..." otherwise.

---

Files changed

• examples/benchmark/tls_bench.c

---

Testing

Verified on Linux (x86-64) with a wolfSSL build that has WOLFSSL_DTLS and
WOLFSSL_DTLS13 enabled.

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[Copilot]

Pull request overview

This PR fixes DTLS (-u) mode failures in examples/benchmark/tls_bench.c by correcting UDP socket setup, avoiding DTLS 1.3 method selection when not compiled in, and choosing a default DTLS payload size that fits within wolfSSL’s default MTU constraints.

Changes:

• Fix SetupSocketAndListen() to bind() UDP sockets and only call listen() for TCP.
• Skip TLS 1.3 cipher suites during DTLS runs when WOLFSSL_DTLS13 is not enabled to prevent wolfSSL_CTX_new() failures.
• Reduce the default DTLS test packet size for non-embedded builds to fit within WOLFSSL_MAX_MTU, and update the startup banner to reflect TLS vs DTLS mode.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10606

No scan targets match the changed files in this PR. Review skipped.