Add PKCS#11 provider support for ECDH and XDH key exchange

[olszomal]

Pull Request Type

• Bug fix
• New feature
• Code style / formatting / renaming
• Refactoring (no functional or API changes)
• Build / CI related changes
• Documentation
• Other (please describe):

Related Issue

Issue number: N/A

Current Behavior

The PKCS#11 provider does not implement OSSL_OP_KEYEXCH.

New Behavior

This change adds PKCS#11 provider support for key exchange operations.

The provider now implements OSSL_OP_KEYEXCH and routes EC, X25519, and X448 derive operations through PKCS#11. For EC keys, ECDH shared-secret derivation is performed with CKM_ECDH1_DERIVE or CKM_ECDH1_COFACTOR_DERIVE, including cofactor mode handling and peer public key import.

The change also adds X25519 and X448 key generation support using CKM_EC_MONTGOMERY_KEY_PAIR_GEN, detects XDH key types from CKA_EC_PARAMS, handles raw public key export/import for X25519 and X448, and wires XDH derive plumbing through the provider path.

Scope of Changes

• Add OSSL_OP_KEYEXCH dispatch and provider registration.
• Add PKCS11_evp_pkey_derive() as the common front-end derive helper.
• Add ECDH derive support through PKCS#11.
• Add X25519/X448 key generation through EC_MONTGOMERY.
• Add X25519/X448 key type detection and raw public key handling.
• Add legacy EVP_PKEY_METHOD derive hooks for X25519/X448 on OpenSSL versions before 4.0.
• Add X25519/X448 key generation examples.
• Add SoftHSM-based ECDH derive tests for both engine and provider paths.

Testing

• Existing tests
• New tests added
• Manual testing -> performed with Thales TCT Luna T-5000 Network HSM and YubiKey 5C

New SoftHSM tests generate an EC key pair with derive usage on the token and verify that:
ECDH(token_private, software_public) == ECDH(software_private, token_public)

Additional Notes

X25519/X448 derive support is wired through the provider path, but the actual hardware derive path remains untested for now because no available token/module advertises EC_MONTGOMERY derive support.

License Declaration

• I hereby agree to license my contribution under the project's license.

[mtrojnar]

[Medium] Provider ECDH derive fails when the peer key is also loaded from pkcs11prov

src/provider_helpers.c:1939

p11_keyexch_ctx_get_peer_pub() only reads keydata->pubkey, which is populated for imported cross-provider peers. Token-loaded provider keys store public bytes in pubdata / params, so using a PKCS#11 public key as -peerkey fails even though the public key was loaded successfully.

Impact: provider ECDH works with a software peer key, but fails for token-private + token-public derive flows.

Suggested fix: make p11_keyexch_ctx_get_peer_pub() fall back to pubdata.ec.pub / pubdata.raw.pub, or populate pubkey when creating P11_KEYDATA from an EVP_PKEY.

---

[Medium] X25519/X448 legacy wrappers can break non-PKCS#11 software derives

src/p11_eddsa.c:346

pkcs11_xdh_pmeth_derive() returns 0 when the key has no libp11 ex_data, but callers only delegate to OpenSSL when ret < 0 (src/p11_eddsa.c:406). After the global X25519/X448 method is registered, unrelated software XDH derives can fail instead of falling back to OpenSSL.

Suggested fix: return -1 for “foreign key” cases, or explicitly fall back whenever no PKCS#11 ex_data is present.

---

[Medium] X25519/X448 public-key equality is broken when GROUP_NAME is present

src/provider_helpers.c:869

The PR adds OSSL_PKEY_PARAM_GROUP_NAME values "X25519" / "X448", but p11_public_equal() treats any non-EdDSA group as classical EC and calls ec_point_equal_by_value() on raw XDH bytes. Identical X25519/X448 public keys with a group name therefore compare unequal.

Suggested fix: special-case "x25519" and "x448" alongside Ed25519/Ed448 and compare PUB_KEY as raw octets.

---

[Low] XDH keygen examples read past argv when PIN is omitted

examples/x25519keygen.c:92
examples/x448keygen.c:92

Both examples check argc < 5 but later use argv[5] for the PIN. Running the examples without a PIN can read past the argument array.

Suggested fix: require six arguments, e.g. argc < 6 or argc != 6.