From 6682ca8a9442d4b6cdc562b35e7c98bb8793abc9 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Marko=20Mili=C4=87?= <marko@tesobe.com>
Date: Tue, 21 Jul 2026 09:10:19 +0200
Subject: [PATCH 1/4] feat(mtls): make dev-mode mTLS a one-command toggle

Turning on the dev-only in-process mTLS termination previously meant editing
five props. Reduce it to a single switch, and stop the switch from silently
breaking local logins.

- Http4sMtls: only mtls.enabled is required now. The four store props fall back
  to the checked-in dev pair (server.jks / server.trust.jks), each fallback
  logged at WARN naming the prop and its OBP_* environment equivalent, so a
  default is never silent. A missing store now fails with the resolved absolute
  path and the working directory in the message -- the defaults are
  repo-relative, so "launched from the wrong directory" is the realistic
  failure and a bare FileNotFoundException did not say so.

- Http4sMtls.keyStoreTypeOf: load .p12/.pfx as PKCS12 and everything else as
  JKS, for keystore and truststore alike. The type cannot be sniffed cheaply
  and guessing wrong fails with an opaque parse error. This makes
  localhost_san_dns_ip.pfx usable -- the only local cert carrying
  DNS:localhost + IP:127.0.0.1, where the server.jks default has no SAN at all
  and relies on clients falling back to CN.

- scripts/mtls_env.sh: exports the OBP_MTLS_* overrides. No new machinery was
  needed -- APIUtil.getPropsValue already reads OBP_<NAME> from the environment
  ahead of the props file, so every prop is env-overridable. Pre-exported
  values win.

  It also pins local_identity_provider. Enabling TLS means hostname must become
  https://, but hostname is the default for local_identity_provider
  (constant.scala), which is the provider column every AuthUser / ResourceUser
  row is keyed on -- moving the scheme alone gives existing local users a
  different provider string and orphans them. The pin reads the hostname via
  Lift's own resolution order: Lift maps Development to an EMPTY modeName, so
  development.default.props is never loaded and default.props is. Reading the
  wrong file there reintroduces exactly the orphaning this pin prevents.

- scripts/java_env.sh: select a JDK >= 17 before any Maven work. The build
  compiles with -release 17 and a default JDK 11 fails with "'17' is not a
  valid choice for '-release'", which reads like a source error. An adequate
  JAVA_HOME is honoured; an inadequate one is replaced with an announcement.

- Both build_and_run scripts: --mtls flag following the existing
  --clean/--online/--no-flush/--background idiom. mtls_env.sh is sourced after
  the build so it can never affect compilation.

- Tests: extension-to-store-type mapping, and buildSslContext loading the .pfx.

Verified live: plain HTTP rejected (curl 52), certless handshake rejected under
client_auth=need (curl 56), 200 with a real client certificate, and a spoofed
PSD2-CERT header replaced by the handshake certificate (1252-char exact match).
Both scripts build and boot from a JDK 11 shell.
---
 docs/MTLS_DEV_MODE.md                         | 55 ++++++++++--
 flushall_build_and_run.sh                     | 15 ++++
 flushall_fast_build_and_run.sh                | 15 ++++
 .../resources/props/sample.props.template     |  4 +-
 .../scala/bootstrap/http4s/Http4sMtls.scala   | 71 ++++++++++++---
 .../bootstrap/http4s/Http4sMtlsTest.scala     | 23 +++++
 scripts/java_env.sh                           | 73 +++++++++++++++
 scripts/mtls_env.sh                           | 89 +++++++++++++++++++
 8 files changed, 325 insertions(+), 20 deletions(-)
 create mode 100755 scripts/java_env.sh
 create mode 100755 scripts/mtls_env.sh

diff --git a/docs/MTLS_DEV_MODE.md b/docs/MTLS_DEV_MODE.md
index 8febd8147a..867826bf6f 100644
--- a/docs/MTLS_DEV_MODE.md
+++ b/docs/MTLS_DEV_MODE.md
@@ -72,7 +72,22 @@ To accept a whole CA instead of individual client certs, import the CA certifica
 
 ### 2. Configure props
 
-In your props file (e.g. `obp-api/src/main/resources/props/default.props`):
+**The short way — no props edit at all:**
+
+```sh
+./flushall_fast_build_and_run.sh --mtls
+```
+
+The flag sources `scripts/mtls_env.sh`, which exports the `OBP_MTLS_*` environment overrides
+(every OBP prop is settable as `OBP_<NAME>` with dots replaced by underscores —
+`APIUtil.getPropsValue` reads the environment ahead of the props file) pointing at the dev
+keystore pair checked into the repo. It also flips `hostname` to `https://` **and** pins
+`local_identity_provider` to the pre-toggle hostname — see the note under "Props reference"
+for why that pinning matters. Anything you pre-export wins, so
+`OBP_MTLS_CLIENT_AUTH=want ./flushall_fast_build_and_run.sh --mtls` works.
+
+To make the mode permanent instead, put it in your props file
+(e.g. `obp-api/src/main/resources/props/default.props`):
 
 ```properties
 mtls.enabled=true
@@ -91,12 +106,20 @@ hostname=https://localhost:8080
 consumer_validation_method_for_consent=CONSUMER_CERTIFICATE
 ```
 
+Only `mtls.enabled` is actually required. The four store props default to the checked-in dev
+pair (`obp-api/src/test/resources/cert/server.jks` / `server.trust.jks`, password `123456`),
+resolved relative to the working directory — so they work when the server is launched from the
+repo root, which is what the run scripts do. Each fallback is logged at WARN naming the resolved
+absolute file, and a missing store fails at startup with the resolved path and working directory
+in the message rather than a bare `FileNotFoundException`.
+
 `run.mode` must be `development` (it is with the standard local run scripts).
 
 ### 3. Run
 
 ```sh
-./flushall_build_and_run.sh
+./flushall_build_and_run.sh --mtls        # or: ./flushall_fast_build_and_run.sh --mtls
+./flushall_build_and_run.sh               # props-driven, if you configured them above
 ```
 
 Boot log confirms the mode:
@@ -170,21 +193,35 @@ Consumer — presenting a different client certificate is rejected.
 
 | Prop | Default | Meaning |
 |---|---|---|
-| `mtls.enabled` | `false` | Master switch. Only honoured when `run.mode=development`. |
-| `mtls.keystore.path` | — (required) | JKS with the server's private key + certificate. |
-| `mtls.keystore.password` | — (required) | Password for keystore and key. |
-| `mtls.truststore.path` | — (required) | JKS with client certificates / CAs the server accepts. |
-| `mtls.truststore.password` | — (required) | Truststore password. |
+| `mtls.enabled` | `false` | Master switch. Only honoured when `run.mode=development`. The one genuinely required prop. |
+| `mtls.keystore.path` | `obp-api/src/test/resources/cert/server.jks` | Store with the server's private key + certificate. `.p12`/`.pfx` are read as PKCS12, anything else as JKS. |
+| `mtls.keystore.password` | `123456` | Password for keystore and key. |
+| `mtls.truststore.path` | `obp-api/src/test/resources/cert/server.trust.jks` | Store with client certificates / CAs the server accepts. Same type detection as the keystore. |
+| `mtls.truststore.password` | `123456` | Truststore password. |
 | `mtls.client_auth` | `need` | `need` rejects certless handshakes; `want` makes the client certificate optional. |
 
+Each of these is also settable from the environment as `OBP_MTLS_ENABLED`,
+`OBP_MTLS_KEYSTORE_PATH`, … which is what `--mtls` uses.
+
+> **`hostname` and `local_identity_provider` move together.** Turning mTLS on means generated
+> links should say `https://`, so `hostname` changes. But `hostname` is also the default for
+> `local_identity_provider` (`code/api/constant/constant.scala`), which is the `provider` column
+> every `AuthUser` / `ResourceUser` row is keyed on. Changing `hostname` alone gives your existing
+> local users a different provider string, orphaning them — logins start failing after a toggle.
+> `scripts/mtls_env.sh` handles this by pinning `local_identity_provider` to the pre-toggle
+> hostname. If you configure mTLS through props instead, set `local_identity_provider` explicitly
+> to whatever `hostname` was before you added the `https://`.
+
 ## Troubleshooting
 
 | Symptom | Cause |
 |---|---|
 | Boot warning `mtls.enabled=true is ignored` | `run.mode` is not `development`. |
-| Boot fails with `requires the props value 'mtls.…'` | One of the four keystore/truststore props is missing. |
+| Boot fails with `points at '…', which does not exist` | The store file isn't there. If you relied on the defaults, the server was launched from somewhere other than the repo root — the message prints the resolved path and the working directory. Use `--mtls` (absolute paths) or set the props absolutely. |
+| Logins that worked over plain HTTP now fail | `hostname` changed scheme without `local_identity_provider` being pinned — see the note under "Props reference". |
 | `curl: (35)` / `alert certificate required` | No (or untrusted) client certificate in `need` mode — check the cert is in `server.trust.jks`. |
 | `curl: (60) SSL certificate problem` | curl doesn't trust the server cert — pass `--cacert server.crt` (or `-k` for a quick look). |
+| Client rejects the server cert with "no alternative certificate subject name matches" | The default `server.jks` leaf has **no SAN**. curl/OpenSSL and Java's `DefaultHostnameVerifier` fall back to CN (`localhost`) so both accept it, but stricter clients won't. Point `mtls.keystore.path` at `obp-api/src/test/resources/cert/localhost_san_dns_ip.pfx` (password `123456`), which carries `DNS:localhost, IP:127.0.0.1`. |
 | Plain `http://` requests hang or error | The port serves HTTPS when mTLS is enabled — use `https://`. |
 | Cert reaches OBP but consumer lookup fails | The Consumer's Client Certificate field doesn't contain the client cert's PEM (lookup is by PEM match, with a whitespace-normalized fallback). |
 | DirectLogin returns `OBP-20073: The user email has not been validated` | A freshly created user must validate their email first. In local dev, either click the link from the validation email (see the `mail.*` props) or set the flag directly in the DB: `update authuser set validated=true where username='YOUR_USER';` |
@@ -208,6 +245,8 @@ header. Two important rules for any proxy config:
 |---|---|
 | `obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala` | Props, SSLContext/TLSContext construction, `PSD2-CERT` injection middleware. |
 | `obp-api/src/main/scala/bootstrap/http4s/Http4sServer.scala` | `mtls.enabled` branch on the Ember builder. |
+| `scripts/mtls_env.sh` | The `OBP_MTLS_*` environment overrides behind the `--mtls` flag of both `flushall_*build_and_run.sh` scripts. Sourceable on its own. |
+| `scripts/java_env.sh` | Selects a JDK >= 17 for the run scripts. The build compiles with `-release 17`; on a default JDK 11 it otherwise fails with the misleading `'17' is not a valid choice for '-release'`. |
 | `obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsTest.scala` | Unit tests: PEM encoding, header injection/stripping, SSLContext from the checked-in keystores. |
 | `obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsHandshakeTest.scala` | End-to-end: real Ember server + real mTLS handshake; proves the verified client cert surfaces as `PSD2-CERT` and certless handshakes are rejected. |
 
diff --git a/flushall_build_and_run.sh b/flushall_build_and_run.sh
index 13a8bfa6c5..3843dc9fa9 100755
--- a/flushall_build_and_run.sh
+++ b/flushall_build_and_run.sh
@@ -11,6 +11,7 @@
 #   ./build_and_run.sh              - Build and run with Redis flush
 #   ./build_and_run.sh --no-flush   - Build and run without Redis flush
 #   ./build_and_run.sh --background - Run server in background
+#   ./build_and_run.sh --mtls       - Serve HTTPS with mutual TLS (dev only)
 #
 # The HTTP4S server runs on the port configured in your props file
 # (default: 8080 for dev.port, or 8086 for hostname port)
@@ -21,6 +22,7 @@ set -e  # Exit on error
 # Parse arguments
 FLUSH_REDIS=true
 RUN_BACKGROUND=false
+USE_MTLS=false
 
 for arg in "$@"; do
     case $arg in
@@ -32,9 +34,16 @@ for arg in "$@"; do
             RUN_BACKGROUND=true
             echo ">>> Server will run in background"
             ;;
+        --mtls)
+            USE_MTLS=true
+            echo ">>> mTLS mode requested (dev-only in-process TLS termination)"
+            ;;
     esac
 done
 
+# Select a JDK >= 17 before any Maven work (the build compiles with -release 17).
+. "$(dirname "${BASH_SOURCE[0]}")/scripts/java_env.sh"
+
 ################################################################################
 # FLUSH REDIS CACHE (OPTIONAL)
 ################################################################################
@@ -132,6 +141,12 @@ echo ""
 # RUN HTTP4S SERVER
 ################################################################################
 
+# Applied after the build so it only affects the running server, never compilation.
+if [ "$USE_MTLS" = true ]; then
+    . "$(dirname "${BASH_SOURCE[0]}")/scripts/mtls_env.sh"
+    echo ""
+fi
+
 echo "=========================================="
 if [ "$RUN_BACKGROUND" = true ]; then
     echo "Starting HTTP4S server (background)..."
diff --git a/flushall_fast_build_and_run.sh b/flushall_fast_build_and_run.sh
index 88669ab6e9..d3c5874b78 100755
--- a/flushall_fast_build_and_run.sh
+++ b/flushall_fast_build_and_run.sh
@@ -17,6 +17,7 @@
 #   ./fast_build_and_run.sh --online       - Check remote repositories
 #   ./fast_build_and_run.sh --no-flush     - Skip Redis flush
 #   ./fast_build_and_run.sh --background   - Run server in background
+#   ./fast_build_and_run.sh --mtls         - Serve HTTPS with mutual TLS (dev only)
 #
 # Typical speedup: 2-5x faster than regular build for incremental changes
 ################################################################################
@@ -28,6 +29,7 @@ DO_CLEAN=""
 OFFLINE_FLAG="-o"  # Default to offline mode for faster builds
 FLUSH_REDIS=true
 RUN_BACKGROUND=false
+USE_MTLS=false
 
 for arg in "$@"; do
     case $arg in
@@ -47,9 +49,16 @@ for arg in "$@"; do
             RUN_BACKGROUND=true
             echo ">>> Server will run in background"
             ;;
+        --mtls)
+            USE_MTLS=true
+            echo ">>> mTLS mode requested (dev-only in-process TLS termination)"
+            ;;
     esac
 done
 
+# Select a JDK >= 17 before any Maven work (the build compiles with -release 17).
+. "$(dirname "${BASH_SOURCE[0]}")/scripts/java_env.sh"
+
 # Show offline mode status if not explicitly set
 if [ "$OFFLINE_FLAG" = "-o" ]; then
     echo ">>> Using offline mode (default) - use --online to check remote repos"
@@ -300,6 +309,12 @@ echo ""
 # RUN HTTP4S SERVER
 ################################################################################
 
+# Applied after the build so it only affects the running server, never compilation.
+if [ "$USE_MTLS" = true ]; then
+    . "$(dirname "${BASH_SOURCE[0]}")/scripts/mtls_env.sh"
+    echo ""
+fi
+
 echo "=========================================="
 if [ "$RUN_BACKGROUND" = true ]; then
     echo "Starting HTTP4S server (background)..."
diff --git a/obp-api/src/main/resources/props/sample.props.template b/obp-api/src/main/resources/props/sample.props.template
index c5ac75d4c4..0409a9dd82 100644
--- a/obp-api/src/main/resources/props/sample.props.template
+++ b/obp-api/src/main/resources/props/sample.props.template
@@ -166,7 +166,9 @@ jwt.use.ssl=false
 ## that forwards the verified client certificate as the PSD2-CERT request header.
 ## When enabled, dev.port serves HTTPS with mutual TLS and the verified client certificate is
 ## injected as the PSD2-CERT header (any client-supplied PSD2-CERT header is stripped).
-## The paths below point at the JKS pair checked into the repo for local development.
+## Only mtls.enabled is required: the four store props below default to exactly these values,
+## the JKS pair checked into the repo for local development. Set them to use your own certificates.
+## Easiest switch is not to edit this file at all: ./flushall_fast_build_and_run.sh --mtls
 # mtls.enabled=true
 # mtls.keystore.path=obp-api/src/test/resources/cert/server.jks
 # mtls.keystore.password=123456
diff --git a/obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala b/obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala
index 6060652d63..56fc785810 100644
--- a/obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala
+++ b/obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala
@@ -1,6 +1,6 @@
 package bootstrap.http4s
 
-import java.io.FileInputStream
+import java.io.{File, FileInputStream}
 import java.nio.charset.StandardCharsets
 import java.security.KeyStore
 import java.security.cert.X509Certificate
@@ -32,6 +32,12 @@ import org.typelevel.ci.CIString
  *   mtls.truststore.password=...
  *   mtls.client_auth=need                           (need = reject certless handshakes; want = optional)
  *
+ * Only `mtls.enabled` is mandatory: the four store props fall back to the checked-in dev keystore
+ * pair (see DevKeystorePath below) so that switching a local server to mTLS is a single toggle.
+ * Every fallback is logged at WARN naming the resolved file, so the default is never silent.
+ * Like every OBP prop these are also settable from the environment as OBP_MTLS_ENABLED,
+ * OBP_MTLS_KEYSTORE_PATH, ... (see APIUtil.getPropsValue, which reads the environment first).
+ *
  * The prop is honoured ONLY in Development run mode. In production OBP must sit behind a reverse
  * proxy that terminates mTLS and forwards the client certificate as the PSD2-CERT header — this
  * feature exists so local development does not need that proxy.
@@ -57,29 +63,72 @@ object Http4sMtls extends MdcLoggable {
     } else propEnabled
   }
 
+  // The checked-in dev pair: a CN=localhost server keypair and the TESOBE CA that signs the dev
+  // client certificates. Repo-relative, so they resolve when the server is launched from the repo
+  // root (which is what the build_and_run scripts do). Only ever reached in Development run mode.
+  private val DevKeystorePath = "obp-api/src/test/resources/cert/server.jks"
+  private val DevTruststorePath = "obp-api/src/test/resources/cert/server.trust.jks"
+  private val DevStorePassword = "123456"
+
+  /** The OBP_-prefixed environment variable APIUtil.getPropsValue consults ahead of the props file. */
+  private def envVarOf(propName: String): String = "OBP_" + propName.replace('.', '_').toUpperCase
+
   lazy val config: MtlsConfig = {
-    def required(name: String): String = APIUtil.getPropsValue(name)
-      .openOrThrowException(s"mtls.enabled=true requires the props value '$name' to be set")
+    def prop(name: String): Option[String] =
+      APIUtil.getPropsValue(name).toOption.map(_.trim).filter(_.nonEmpty)
+
+    // Returns the absolute store path plus its password, defaulting both to the dev pair.
+    def store(pathProp: String, devPath: String, passwordProp: String): (String, String) = {
+      val path = prop(pathProp).getOrElse {
+        logger.warn(s"'$pathProp' is not set — falling back to the checked-in dev store '$devPath'. " +
+          s"Set '$pathProp' (or ${envVarOf(pathProp)}) to use your own certificates.")
+        devPath
+      }
+      val file = new File(path)
+      if (!file.isFile) throw new RuntimeException(
+        s"mtls.enabled=true but '$pathProp' points at '$path', which does not exist " +
+          s"(resolved to ${file.getAbsolutePath} from working directory ${new File(".").getAbsolutePath}). " +
+          s"Launch from the repo root or set '$pathProp' to an absolute path.")
+      val password = prop(passwordProp).getOrElse {
+        logger.warn(s"'$passwordProp' is not set — falling back to the checked-in dev store password.")
+        DevStorePassword
+      }
+      (file.getAbsolutePath, password)
+    }
+
+    val (keystorePath, keystorePassword) = store("mtls.keystore.path", DevKeystorePath, "mtls.keystore.password")
+    val (truststorePath, truststorePassword) = store("mtls.truststore.path", DevTruststorePath, "mtls.truststore.password")
     MtlsConfig(
-      keystorePath = required("mtls.keystore.path"),
-      keystorePassword = required("mtls.keystore.password"),
-      truststorePath = required("mtls.truststore.path"),
-      truststorePassword = required("mtls.truststore.password"),
+      keystorePath = keystorePath,
+      keystorePassword = keystorePassword,
+      truststorePath = truststorePath,
+      truststorePassword = truststorePassword,
       needClientAuth = APIUtil.getPropsValue("mtls.client_auth", "need").toLowerCase != "want"
     )
   }
 
+  /**
+   * KeyStore type for a store file, by extension: `.p12` / `.pfx` are PKCS12, everything else JKS.
+   * Both formats are common for the certificates a TPP hands over — `obp-api/src/test/resources/cert`
+   * carries examples of each — and guessing wrong fails at load with an opaque parse error.
+   */
+  private[http4s] def keyStoreTypeOf(path: String): String =
+    path.toLowerCase match {
+      case p if p.endsWith(".p12") || p.endsWith(".pfx") => "PKCS12"
+      case _ => "JKS"
+    }
+
   def buildSslContext(config: MtlsConfig): SSLContext = {
-    def loadJks(path: String, password: String): KeyStore = {
-      val keyStore = KeyStore.getInstance("JKS")
+    def loadStore(path: String, password: String): KeyStore = {
+      val keyStore = KeyStore.getInstance(keyStoreTypeOf(path))
       val in = new FileInputStream(path)
       try keyStore.load(in, password.toCharArray) finally in.close()
       keyStore
     }
     val keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm)
-    keyManagerFactory.init(loadJks(config.keystorePath, config.keystorePassword), config.keystorePassword.toCharArray)
+    keyManagerFactory.init(loadStore(config.keystorePath, config.keystorePassword), config.keystorePassword.toCharArray)
     val trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm)
-    trustManagerFactory.init(loadJks(config.truststorePath, config.truststorePassword))
+    trustManagerFactory.init(loadStore(config.truststorePath, config.truststorePassword))
     val sslContext = SSLContext.getInstance("TLS")
     sslContext.init(keyManagerFactory.getKeyManagers, trustManagerFactory.getTrustManagers, null)
     sslContext
diff --git a/obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsTest.scala b/obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsTest.scala
index 781346e2de..f38ac157f5 100644
--- a/obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsTest.scala
+++ b/obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsTest.scala
@@ -96,4 +96,27 @@ class Http4sMtlsTest extends FlatSpec with Matchers {
     sslContext should not be null
     sslContext.getProtocol shouldEqual "TLS"
   }
+
+  // A TPP's certificates arrive as often in PKCS12 as in JKS, and the store type cannot be
+  // sniffed from the bytes cheaply — getting it wrong fails at load with an opaque parse error.
+  "keyStoreTypeOf" should "select PKCS12 for .p12/.pfx and JKS otherwise" in {
+    Http4sMtls.keyStoreTypeOf("/path/server.p12") shouldEqual "PKCS12"
+    Http4sMtls.keyStoreTypeOf("/path/server.PFX") shouldEqual "PKCS12"
+    Http4sMtls.keyStoreTypeOf("/path/server.jks") shouldEqual "JKS"
+    Http4sMtls.keyStoreTypeOf("/path/server") shouldEqual "JKS"
+  }
+
+  it should "let buildSslContext load a PKCS12 keystore" in {
+    val certDir = new java.io.File(getClass.getResource("/cert/server.jks").toURI).getParent
+    // localhost_san_dns_ip.pfx carries SAN DNS:localhost + IP:127.0.0.1, which JKS server.jks
+    // lacks — clients that do not fall back to CN need this one.
+    val config = Http4sMtls.MtlsConfig(
+      keystorePath = s"$certDir/localhost_san_dns_ip.pfx",
+      keystorePassword = "123456",
+      truststorePath = s"$certDir/server.trust.jks",
+      truststorePassword = "123456",
+      needClientAuth = true
+    )
+    Http4sMtls.buildSslContext(config) should not be null
+  }
 }
diff --git a/scripts/java_env.sh b/scripts/java_env.sh
new file mode 100755
index 0000000000..d99b4e04d0
--- /dev/null
+++ b/scripts/java_env.sh
@@ -0,0 +1,73 @@
+#!/bin/bash
+################################################################################
+# OBP-API JDK selection
+#
+# The build compiles with `-release 17`. A JDK older than 17 fails the build with
+#
+#   ERROR '17' is not a valid choice for '-release'
+#   ERROR bad option: '-release'
+#
+# which is easy to misread as a source error. Distributions where the default
+# `java` is still 11 hit this on every build, so the run scripts source this file
+# to select a >= 17 JDK before invoking Maven.
+#
+# An existing JAVA_HOME is honoured whenever it is new enough; it is only replaced
+# when it would fail the build, and the replacement is announced. Sets both
+# JAVA_HOME and PATH, because the run scripts invoke a bare `java` for the server.
+################################################################################
+
+# Echoes the feature version (e.g. 17) of the java binary passed in, or nothing.
+java_major_of() {
+    local java_bin="$1"
+    [ -x "$java_bin" ] || return 0
+    "$java_bin" -version 2>&1 \
+        | head -1 \
+        | sed -nE 's/.*version "([0-9]+)(\.[0-9]+)*.*/\1/p'
+}
+
+java_env_required=17
+java_env_selected=""
+
+# 1. An adequate JAVA_HOME already in the environment wins — do not second-guess it.
+if [ -n "$JAVA_HOME" ]; then
+    java_env_current="$(java_major_of "$JAVA_HOME/bin/java")"
+    if [ -n "$java_env_current" ] && [ "$java_env_current" -ge "$java_env_required" ] 2>/dev/null; then
+        java_env_selected="$JAVA_HOME"
+    else
+        echo ">>> JAVA_HOME points at Java ${java_env_current:-unknown}, but the build needs >= $java_env_required — looking for another JDK"
+    fi
+fi
+
+# 2. Otherwise search the usual locations (Linux distro paths, then macOS).
+if [ -z "$java_env_selected" ]; then
+    for java_env_candidate in \
+        /usr/lib/jvm/java-17-openjdk-amd64 \
+        /usr/lib/jvm/java-17-openjdk \
+        /usr/lib/jvm/java-21-openjdk-amd64 \
+        /usr/lib/jvm/java-21-openjdk \
+        "$(/usr/libexec/java_home -v 17 2>/dev/null)" \
+        "$(/usr/libexec/java_home -v 21 2>/dev/null)"; do
+        [ -n "$java_env_candidate" ] || continue
+        java_env_candidate_major="$(java_major_of "$java_env_candidate/bin/java")"
+        if [ -n "$java_env_candidate_major" ] && [ "$java_env_candidate_major" -ge "$java_env_required" ] 2>/dev/null; then
+            java_env_selected="$java_env_candidate"
+            break
+        fi
+    done
+fi
+
+# 3. Fall back to whatever `java` is on PATH, warning if it cannot build.
+if [ -n "$java_env_selected" ]; then
+    export JAVA_HOME="$java_env_selected"
+    export PATH="$JAVA_HOME/bin:$PATH"
+    echo ">>> Using JDK $(java_major_of "$JAVA_HOME/bin/java") at $JAVA_HOME"
+else
+    java_env_path_major="$(java_major_of "$(command -v java 2>/dev/null)")"
+    echo ">>> WARNING: no JDK >= $java_env_required found (java on PATH is ${java_env_path_major:-missing})."
+    echo "    The build will fail with \"'17' is not a valid choice for '-release'\"."
+    echo "    Install a JDK $java_env_required or set JAVA_HOME to one."
+fi
+
+# Sourced under `set -e`: end on a command that always succeeds so a false test
+# in the block above can never take down the caller.
+true
diff --git a/scripts/mtls_env.sh b/scripts/mtls_env.sh
new file mode 100755
index 0000000000..aff7ed8ef2
--- /dev/null
+++ b/scripts/mtls_env.sh
@@ -0,0 +1,89 @@
+#!/bin/bash
+################################################################################
+# OBP-API dev-only mTLS environment
+#
+# Exports the OBP_* overrides that switch the http4s server into in-process mTLS
+# termination (see obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala and
+# docs/MTLS_DEV_MODE.md). Sourced by the --mtls flag of the build_and_run
+# scripts; can also be sourced by hand for a jar that is already built:
+#
+#   . scripts/mtls_env.sh && java -jar obp-api/target/obp-api.jar
+#
+# Every OBP prop is overridable from the environment as OBP_<NAME> with dots
+# replaced by underscores (APIUtil.getPropsValue reads the environment ahead of
+# the props file), which is what makes this a no-edit toggle.
+#
+# All values below are defaults only: anything already exported wins, so
+#   OBP_MTLS_CLIENT_AUTH=want ./flushall_fast_build_and_run.sh --mtls
+# behaves as expected.
+################################################################################
+
+MTLS_REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+# Mirror Lift's own props resolution, in its order. In Development, Lift's `modeName`
+# is the EMPTY string, so its candidate list collapses to the user/host-specific files
+# and then default.props. A file named `development.default.props` is therefore never
+# loaded in Development — despite the name, it is inert, and reading the hostname from
+# it silently pins local_identity_provider to a value the running app does not use.
+# (`test.default.props` does work, because in Test mode modeName is "test".)
+mtls_props_dir="$MTLS_REPO_ROOT/obp-api/src/main/resources/props"
+mtls_user="$(id -un 2>/dev/null)"
+mtls_host="$(hostname 2>/dev/null)"
+mtls_props_file=""
+for mtls_candidate in \
+    "$mtls_props_dir/$mtls_user.$mtls_host.props" \
+    "$mtls_props_dir/$mtls_user.props" \
+    "$mtls_props_dir/$mtls_host.props" \
+    "$mtls_props_dir/default.props"; do
+    if [ -f "$mtls_candidate" ]; then mtls_props_file="$mtls_candidate"; break; fi
+done
+
+# Echoes the value of a prop from that file, or nothing when it is absent/commented out.
+mtls_prop() {
+    [ -n "$mtls_props_file" ] || return 0
+    grep -E "^[[:space:]]*$1[[:space:]]*=" "$mtls_props_file" \
+        | tail -1 | cut -d= -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'
+}
+
+# --- the toggle itself -------------------------------------------------------
+# Http4sMtls defaults the four store props to this same checked-in dev pair, so
+# strictly only OBP_MTLS_ENABLED is required. They are set explicitly here as
+# absolute paths so that the server also starts correctly from another working
+# directory, where the repo-relative defaults would not resolve.
+export OBP_MTLS_ENABLED=true
+: "${OBP_MTLS_KEYSTORE_PATH:=$MTLS_REPO_ROOT/obp-api/src/test/resources/cert/server.jks}"
+: "${OBP_MTLS_KEYSTORE_PASSWORD:=123456}"
+: "${OBP_MTLS_TRUSTSTORE_PATH:=$MTLS_REPO_ROOT/obp-api/src/test/resources/cert/server.trust.jks}"
+: "${OBP_MTLS_TRUSTSTORE_PASSWORD:=123456}"
+: "${OBP_MTLS_CLIENT_AUTH:=need}"
+export OBP_MTLS_KEYSTORE_PATH OBP_MTLS_KEYSTORE_PASSWORD
+export OBP_MTLS_TRUSTSTORE_PATH OBP_MTLS_TRUSTSTORE_PASSWORD OBP_MTLS_CLIENT_AUTH
+
+# --- hostname, and why local_identity_provider has to be pinned with it -------
+# The listener now speaks TLS, so generated links must say https://. But hostname
+# is also the default for local_identity_provider (code/api/constant/constant.scala),
+# which is the `provider` column every AuthUser / ResourceUser row is keyed on.
+# Flipping the scheme alone would give the same local users a different provider
+# string on every toggle, orphaning them. So pin local_identity_provider to the
+# pre-toggle hostname and let only the scheme move. Skipped when the props file
+# already sets local_identity_provider explicitly — then there is nothing to pin.
+mtls_props_hostname="$(mtls_prop hostname)"
+if [ -n "$mtls_props_hostname" ]; then
+    if [ -z "$(mtls_prop local_identity_provider)" ]; then
+        : "${OBP_LOCAL_IDENTITY_PROVIDER:=$mtls_props_hostname}"
+        export OBP_LOCAL_IDENTITY_PROVIDER
+    fi
+    : "${OBP_HOSTNAME:=$(echo "$mtls_props_hostname" | sed 's|^http://|https://|')}"
+    export OBP_HOSTNAME
+fi
+
+echo ">>> mTLS enabled (dev-only in-process TLS termination)"
+echo "      keystore   : $OBP_MTLS_KEYSTORE_PATH"
+echo "      truststore : $OBP_MTLS_TRUSTSTORE_PATH"
+echo "      client_auth: $OBP_MTLS_CLIENT_AUTH"
+echo "      hostname   : ${OBP_HOSTNAME:-<unchanged, no hostname in props>}"
+# Plain `if`, not `[ … ] && echo`: this is the last command of a sourced file, and the callers
+# run under `set -e` — a false test would take its exit status and abort the caller.
+if [ -n "$OBP_LOCAL_IDENTITY_PROVIDER" ]; then
+    echo "      local_identity_provider pinned to: $OBP_LOCAL_IDENTITY_PROVIDER"
+fi

From be9c226dc1a67cbd4bfd0b872794b9b50acf624d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Marko=20Mili=C4=87?= <marko@tesobe.com>
Date: Tue, 21 Jul 2026 09:10:53 +0200
Subject: [PATCH 2/4] fix(build): stop set -e from swallowing build failures in
 the run scripts

Both build_and_run scripts run under `set -e`, and both call mvn without
guarding it. A failing build therefore aborted the script at the mvn line, so
everything written to handle that failure never ran:

- flushall_fast_build_and_run.sh captured BUILD_EXIT_CODE=$? and had a whole
  "detect incremental compilation cache issue, retry with a clean build" block
  keyed off it. None of it was reachable. The script exited silently with no
  diagnosis and no retry.
- flushall_build_and_run.sh could not reach its BUILD_EXIT check, so it never
  printed the log tail it promises on failure.

Bracket each mvn invocation with set +e / set -e so the exit code is captured
and the existing handling runs.

The symptom is easy to misread: the scripts pipe their output, so the exit code
a caller sees is the pipeline's, not the script's -- a failed build could look
like a successful run that simply produced no server.

Verified by putting a failing mvn on PATH and running the real script: it now
reports the failure, detects the cache-issue signature, retries with a clean
build, tails the log and exits 1.
---
 flushall_build_and_run.sh      |  4 ++++
 flushall_fast_build_and_run.sh | 11 ++++++++---
 2 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/flushall_build_and_run.sh b/flushall_build_and_run.sh
index 3843dc9fa9..ae6485bdca 100755
--- a/flushall_build_and_run.sh
+++ b/flushall_build_and_run.sh
@@ -118,8 +118,12 @@ echo "Build output will be saved to: build.log"
 # Show last 3 lines of build output in real-time
 tail -n 3 -f build.log &
 TAIL_PID=$!
+# set +e: this script runs under `set -e`, which would abort at a failing build
+# before the BUILD_EXIT check below could print the log tail.
+set +e
 mvn -pl obp-api -am clean package -DskipTests=true -Dmaven.test.skip=true -T 4 > build.log 2>&1
 BUILD_EXIT=$?
+set -e
 kill $TAIL_PID 2>/dev/null || true
 wait $TAIL_PID 2>/dev/null || true
 
diff --git a/flushall_fast_build_and_run.sh b/flushall_fast_build_and_run.sh
index d3c5874b78..7483e2ce7b 100755
--- a/flushall_fast_build_and_run.sh
+++ b/flushall_fast_build_and_run.sh
@@ -174,6 +174,10 @@ fi
 } > fast_build.log
 
 # Run Maven build and append to log
+# set +e around the build: this script runs under `set -e`, which would abort here
+# on a failing build and make BUILD_EXIT_CODE — and the auto-retry-with-clean block
+# below — unreachable. We want to inspect the failure, not die at it.
+set +e
 mvn -pl obp-api -am \
     $DO_CLEAN \
     package \
@@ -184,8 +188,8 @@ mvn -pl obp-api -am \
     -Dcheckstyle.skip=true \
     -Dspotbugs.skip=true \
     -Dpmd.skip=true >> fast_build.log 2>&1
-
 BUILD_EXIT_CODE=$?
+set -e
 
 # Record build end time and calculate duration
 if command -v gdate &> /dev/null; then
@@ -246,7 +250,8 @@ if [ $BUILD_EXIT_CODE -ne 0 ] && [ -z "$DO_CLEAN" ]; then
             echo ""
         } > fast_build.log
         
-        # Run clean build
+        # Run clean build (set +e for the same reason as the first invocation)
+        set +e
         mvn -pl obp-api -am \
             clean \
             package \
@@ -257,8 +262,8 @@ if [ $BUILD_EXIT_CODE -ne 0 ] && [ -z "$DO_CLEAN" ]; then
             -Dcheckstyle.skip=true \
             -Dspotbugs.skip=true \
             -Dpmd.skip=true >> fast_build.log 2>&1
-        
         BUILD_EXIT_CODE=$?
+        set -e
         
         # Record retry end time and calculate duration
         if command -v gdate &> /dev/null; then

From 3c863d318095e1e35b14f6994aad5721a819e08b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Marko=20Mili=C4=87?= <marko@tesobe.com>
Date: Tue, 21 Jul 2026 09:18:19 +0200
Subject: [PATCH 3/4] fix(psd2): return 401, not 500, when
 TPP-Signature-Certificate is unusable

With requirePsd2Certificates=ONLINE, any request reaching passesPsd2Aisp
without a TPP-Signature-Certificate header produced

    500 OBP-50000: Unknown Error.: Illegal base64 character 2d

getCertificateFromTppSignatureCertificate read the header via getHeaderValue,
which substitutes SecureRandomUtil.csprng.nextLong().toString when a header is
absent. That is a serviceable "never matches" sentinel for the string
comparisons it was written for (Digest, Signature), but the value here goes
straight into Base64.getDecoder: a negative Long renders with a leading '-',
which is 0x2d, and the decoder throws.

The bug was also flaky by construction. Only about half of calls got a negative
Long; the rest base64-decoded to garbage and failed later in getCertificatePem
with a different error. Same missing header, two symptoms.

Make getCertificateFromTppSignatureCertificate return Box[X509Certificate]
instead of throwing, and look the header up directly rather than through
getHeaderValue. Each failure mode is now a Failure: header absent, not base64,
base64 of something that is not a PEM, or a PEM that is not a certificate.

Call sites:
- APIUtil.passesPsd2ServiceProviderCommon: fail closed. passesPsd2ServiceProvider
  already maps a Failure to 401 -- this is the path that produced the 500.
- BerlinGroupSigning.verifySignedRequest: checkRequestIsSigned only proves the
  header is present, not usable, so an unusable one is a 401.
- BerlinGroupSigning.getOrCreateConsumer: leave the caller's result untouched.
  Consumer creation is a side effect of a signed request, not the thing being
  authorised; verifySignedRequest is what rejects a bad certificate.
- BerlinGroupCheck: without a certificate there is no serial number to compare
  keyId.SN against, so it is an invalid signature header (400).

Found by running a real TPP (OBP-Hola) through the UK Open Banking consent
journey. Verified: the identical POST to
/open-banking/v3.1/account-access-consents now returns 401 with no "Illegal
base64" in the log; RegulatedEntityTest passes.

Note this changes only the failure mode, not the gate: a caller with no
TPP-Signature-Certificate is still rejected, just with the correct status.
---
 .../main/scala/code/api/util/APIUtil.scala    |  50 ++++---
 .../code/api/util/BerlinGroupCheck.scala      |  15 ++-
 .../code/api/util/BerlinGroupSigning.scala    | 124 ++++++++++++------
 3 files changed, 120 insertions(+), 69 deletions(-)

diff --git a/obp-api/src/main/scala/code/api/util/APIUtil.scala b/obp-api/src/main/scala/code/api/util/APIUtil.scala
index cd283759a1..c17eb54dcc 100644
--- a/obp-api/src/main/scala/code/api/util/APIUtil.scala
+++ b/obp-api/src/main/scala/code/api/util/APIUtil.scala
@@ -3854,27 +3854,37 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{
       case value if value.toUpperCase == "ONLINE" =>
         val requestHeaders = cc.map(_.requestHeaders).getOrElse(Nil)
         val consumerName = cc.flatMap(_.consumer.map(_.name.get)).getOrElse("")
-        val certificate = getCertificateFromTppSignatureCertificate(requestHeaders)
-        for {
-          tpps <- BerlinGroupSigning.getRegulatedEntityByCertificate(certificate, cc)
-        } yield {
-          tpps match {
-            case Nil =>
-              ObpApiFailure(RegulatedEntityNotFoundByCertificate, 401, cc)
-            case single :: Nil =>
-              logger.debug(s"Regulated entity by certificate: $single")
-              // Only one match, proceed to role check
-              if (single.services.contains(serviceProvider)) {
-                logger.debug(s"Regulated entity by certificate (single.services: ${single.services}, serviceProvider: $serviceProvider): ")
-                Full(true)
-              } else {
-                ObpApiFailure(X509ActionIsNotAllowed, 403, cc)
+        getCertificateFromTppSignatureCertificate(requestHeaders) match {
+          // No usable TPP-Signature-Certificate: fail closed. passesPsd2ServiceProvider maps a
+          // Failure to a 401 -- this used to throw out of the base64 decode and become a 500.
+          case failure: Failure =>
+            logger.debug(s"passesPsd2ServiceProvider: no usable TPP-Signature-Certificate: $failure")
+            Future(failure)
+          case Empty =>
+            logger.debug("passesPsd2ServiceProvider: no TPP-Signature-Certificate header")
+            Future(Failure(X509CannotGetCertificate))
+          case Full(certificate) =>
+            for {
+              tpps <- BerlinGroupSigning.getRegulatedEntityByCertificate(certificate, cc)
+            } yield {
+              tpps match {
+                case Nil =>
+                  ObpApiFailure(RegulatedEntityNotFoundByCertificate, 401, cc)
+                case single :: Nil =>
+                  logger.debug(s"Regulated entity by certificate: $single")
+                  // Only one match, proceed to role check
+                  if (single.services.contains(serviceProvider)) {
+                    logger.debug(s"Regulated entity by certificate (single.services: ${single.services}, serviceProvider: $serviceProvider): ")
+                    Full(true)
+                  } else {
+                    ObpApiFailure(X509ActionIsNotAllowed, 403, cc)
+                  }
+                case multiple =>
+                  // Ambiguity detected: more than one TPP matches the certificate
+                  val names = multiple.map(e => s"'${e.entityName}' (Code: ${e.entityCode})").mkString(", ")
+                  ObpApiFailure(s"$RegulatedEntityAmbiguityByCertificate: multiple TPPs found: $names", 401, cc)
               }
-            case multiple =>
-              // Ambiguity detected: more than one TPP matches the certificate
-              val names = multiple.map(e => s"'${e.entityName}' (Code: ${e.entityCode})").mkString(", ")
-              ObpApiFailure(s"$RegulatedEntityAmbiguityByCertificate: multiple TPPs found: $names", 401, cc)
-          }
+            }
         }
       case value if value.toUpperCase == "CERTIFICATE" => Future {
         `getPSD2-CERT`(cc.map(_.requestHeaders).getOrElse(Nil)) match {
diff --git a/obp-api/src/main/scala/code/api/util/BerlinGroupCheck.scala b/obp-api/src/main/scala/code/api/util/BerlinGroupCheck.scala
index 98026bf4ff..95706d8f3a 100644
--- a/obp-api/src/main/scala/code/api/util/BerlinGroupCheck.scala
+++ b/obp-api/src/main/scala/code/api/util/BerlinGroupCheck.scala
@@ -141,16 +141,19 @@ object BerlinGroupCheck extends MdcLoggable {
             logger.debug(s"  Algorithm: ${parsed.algorithm}")
             logger.debug(s"  Signature: ${parsed.signature}")
 
-            val certificate = getCertificateFromTppSignatureCertificate(reqHeaders)
-            val certSerialNumber = certificate.getSerialNumber
+            // A Signature header without a usable TPP-Signature-Certificate cannot have its keyId.SN
+            // checked against anything, so it is an invalid signature header (400) — not a 500.
+            val certSerialNumber = getCertificateFromTppSignatureCertificate(reqHeaders).map(_.getSerialNumber)
 
-            logger.debug(s"Certificate serial number (decimal): ${certSerialNumber.toString}")
-            logger.debug(s"Certificate serial number (hex): ${certSerialNumber.toString(16).toUpperCase}")
+            logger.debug(s"Certificate serial number (decimal): ${certSerialNumber.map(_.toString)}")
+            logger.debug(s"Certificate serial number (hex): ${certSerialNumber.map(sn => sn.toString(16).toUpperCase)}")
 
-            val snMatches = BerlinGroupSignatureHeaderParser.doesSerialNumberMatch(parsed.keyId.sn, certSerialNumber)
+            val snMatches = certSerialNumber
+              .map(BerlinGroupSignatureHeaderParser.doesSerialNumberMatch(parsed.keyId.sn, _))
+              .getOrElse(false)
 
             if (!snMatches) {
-              logger.debug(s"Serial number mismatch. Parsed SN: ${parsed.keyId.sn}, Certificate decimal: ${certSerialNumber.toString}, Certificate hex: ${certSerialNumber.toString(16).toUpperCase}")
+              logger.debug(s"Serial number mismatch (or no usable certificate). Parsed SN: ${parsed.keyId.sn}, Certificate decimal: ${certSerialNumber.map(_.toString)}, Certificate hex: ${certSerialNumber.map(sn => sn.toString(16).toUpperCase)}")
               Some(
                 (
                   fullBoxOrException(
diff --git a/obp-api/src/main/scala/code/api/util/BerlinGroupSigning.scala b/obp-api/src/main/scala/code/api/util/BerlinGroupSigning.scala
index 0ea72011a7..f183de602e 100644
--- a/obp-api/src/main/scala/code/api/util/BerlinGroupSigning.scala
+++ b/obp-api/src/main/scala/code/api/util/BerlinGroupSigning.scala
@@ -168,40 +168,46 @@ object BerlinGroupSigning extends MdcLoggable {
         forwardResult
       case true =>
         val requestHeaders = forwardResult._2.map(_.requestHeaders).getOrElse(Nil)
-        val certificate = getCertificateFromTppSignatureCertificate(requestHeaders)
-        X509.validateCertificate(certificate) match {
-          case Full(true) => // PEM certificate is ok
-            val generatedDigest = generateDigest(body.getOrElse(""))
-            val requestHeaderDigest = getHeaderValue(RequestHeader.Digest, requestHeaders)
-            if(generatedDigest == requestHeaderDigest) { // Verifying the Hash in the Digest Field
-              val signatureHeaderValue = getHeaderValue(RequestHeader.Signature, requestHeaders)
-              val signature = parseSignatureHeader(signatureHeaderValue).getOrElse("signature", "NONE")
-              val headersToSign = parseSignatureHeader(signatureHeaderValue).getOrElse("headers", "").split(" ").toList
-              val sn = parseSignatureHeader(signatureHeaderValue).getOrElse("keyId", "").split(" ").toList
-              val headers = headersToSign.map(h =>
-                if (h.toLowerCase() == RequestHeader.Digest.toLowerCase()) {
-                  s"$h: $generatedDigest"
-                } else {
-                  s"$h: ${getHeaderValue(h, requestHeaders)}"
+        // checkRequestIsSigned only proves the header is present, not that it holds a usable
+        // certificate — an unparseable one is a 401, not a 500.
+        getCertificateFromTppSignatureCertificate(requestHeaders) match {
+          case Full(certificate) =>
+            X509.validateCertificate(certificate) match {
+              case Full(true) => // PEM certificate is ok
+                val generatedDigest = generateDigest(body.getOrElse(""))
+                val requestHeaderDigest = getHeaderValue(RequestHeader.Digest, requestHeaders)
+                if(generatedDigest == requestHeaderDigest) { // Verifying the Hash in the Digest Field
+                  val signatureHeaderValue = getHeaderValue(RequestHeader.Signature, requestHeaders)
+                  val signature = parseSignatureHeader(signatureHeaderValue).getOrElse("signature", "NONE")
+                  val headersToSign = parseSignatureHeader(signatureHeaderValue).getOrElse("headers", "").split(" ").toList
+                  val sn = parseSignatureHeader(signatureHeaderValue).getOrElse("keyId", "").split(" ").toList
+                  val headers = headersToSign.map(h =>
+                    if (h.toLowerCase() == RequestHeader.Digest.toLowerCase()) {
+                      s"$h: $generatedDigest"
+                    } else {
+                      s"$h: ${getHeaderValue(h, requestHeaders)}"
+                    }
+                  )
+                  val signingString = headers.mkString("\n")
+                  val isVerified = verifySignature(signingString, signature, certificate.getPublicKey)
+                  val isValidated = CertificateVerifier.validateCertificate(certificate)
+                  val bypassValidation = APIUtil.getPropsAsBoolValue("bypass_tpp_signature_validation", defaultValue = false)
+                  (isVerified, isValidated) match {
+                    case (true, true) => forwardResult
+                    case (true, false) if bypassValidation => forwardResult
+                    case (true, false) => apiFailure(ErrorMessages.X509PublicKeyCannotBeValidated, 401)(forwardResult)
+                    case (false, _) => apiFailure(ErrorMessages.X509PublicKeyCannotVerify, 401)(forwardResult)
+                  }
+                } else { // The two DIGEST hashes do NOT match, the integrity of the request body is NOT confirmed.
+                  logger.debug(s"Generated digest: $generatedDigest")
+                  logger.debug(s"Request header digest: $requestHeaderDigest")
+                  apiFailure(ErrorMessages.X509PublicKeyCannotVerify, 401)(forwardResult)
                 }
-              )
-              val signingString = headers.mkString("\n")
-              val isVerified = verifySignature(signingString, signature, certificate.getPublicKey)
-              val isValidated = CertificateVerifier.validateCertificate(certificate)
-              val bypassValidation = APIUtil.getPropsAsBoolValue("bypass_tpp_signature_validation", defaultValue = false)
-              (isVerified, isValidated) match {
-                case (true, true) => forwardResult
-                case (true, false) if bypassValidation => forwardResult
-                case (true, false) => apiFailure(ErrorMessages.X509PublicKeyCannotBeValidated, 401)(forwardResult)
-                case (false, _) => apiFailure(ErrorMessages.X509PublicKeyCannotVerify, 401)(forwardResult)
-              }
-            } else { // The two DIGEST hashes do NOT match, the integrity of the request body is NOT confirmed.
-              logger.debug(s"Generated digest: $generatedDigest")
-              logger.debug(s"Request header digest: $requestHeaderDigest")
-              apiFailure(ErrorMessages.X509PublicKeyCannotVerify, 401)(forwardResult)
+              case Failure(msg, t, c) => (Failure(msg, t, c), forwardResult._2) // PEM certificate is not valid
+              case _ => apiFailure(ErrorMessages.X509GeneralError, 401)(forwardResult) // PEM certificate cannot be validated
             }
-          case Failure(msg, t, c) => (Failure(msg, t, c), forwardResult._2) // PEM certificate is not valid
-          case _ => apiFailure(ErrorMessages.X509GeneralError, 401)(forwardResult) // PEM certificate cannot be validated
+          case Failure(msg, t, c) => (Failure(msg, t, c), forwardResult._2) // certificate header unusable
+          case _ => apiFailure(ErrorMessages.X509CannotGetCertificate, 401)(forwardResult)
         }
     }
   }
@@ -210,15 +216,40 @@ object BerlinGroupSigning extends MdcLoggable {
     requestHeaders.find(_.name.toLowerCase() == name.toLowerCase()).map(_.values.mkString)
       .getOrElse(SecureRandomUtil.csprng.nextLong().toString)
   }
-  def getCertificateFromTppSignatureCertificate(requestHeaders: List[HTTPParam]): X509Certificate = {
-    val certificate = getHeaderValue(RequestHeader.`TPP-Signature-Certificate`, requestHeaders)
-    // Decode the Base64 string
-    val decodedBytes = Base64.getDecoder.decode(certificate)
-    // Convert the bytes to a string (it could be PEM format for public key)
-    val decodedString = new String(decodedBytes, StandardCharsets.UTF_8)
-
-    val certificatePemString = getCertificatePem(decodedString)
-    parseCertificate(certificatePemString)
+  /**
+   * The `TPP-Signature-Certificate` header parsed into an X509 certificate, or a Failure saying why
+   * it could not be.
+   *
+   * Returns a Box rather than throwing. Every step here consumes caller-supplied input and can fail
+   * on data we do not control: the header may be absent, not base64, base64 of something that is
+   * not a PEM, or a PEM that is not a certificate. Callers map the Failure to a 401; an exception
+   * would surface as a 500 instead.
+   *
+   * In particular this must NOT use `getHeaderValue`, which substitutes a random Long for an absent
+   * header. That is a serviceable "never matches" sentinel for the string comparisons it was
+   * written for (Digest, Signature), but not for a value about to be Base64-decoded: a negative
+   * Long renders with a leading '-', and `Base64.getDecoder` rejects it with
+   * "Illegal base64 character 2d" — so a merely missing header became a 500, and only for the
+   * ~half of calls where the random Long happened to be negative.
+   */
+  def getCertificateFromTppSignatureCertificate(requestHeaders: List[HTTPParam]): Box[X509Certificate] = {
+    requestHeaders
+      .find(_.name.equalsIgnoreCase(RequestHeader.`TPP-Signature-Certificate`))
+      .map(_.values.mkString.trim)
+      .filter(_.nonEmpty) match {
+      case None =>
+        Failure(ErrorMessages.X509CannotGetCertificate)
+      case Some(headerValue) =>
+        // Decode the Base64 string, then pull the PEM certificate out of whatever it decoded to.
+        Helpers.tryo(new String(Base64.getDecoder.decode(headerValue), StandardCharsets.UTF_8)) match {
+          case Full(decodedString) =>
+            getCertificatePem(decodedString) match {
+              case "" => Failure(ErrorMessages.X509CannotGetCertificate)
+              case certificatePemString => Helpers.tryo(parseCertificate(certificatePemString)) ?~ ErrorMessages.X509GeneralError
+            }
+          case _ => Failure(ErrorMessages.X509GeneralError)
+        }
+    }
   }
 
   private def getCertificatePem(decodedString: String) = {
@@ -276,8 +307,15 @@ object BerlinGroupSigning extends MdcLoggable {
     val tppSignatureCert: String = APIUtil.getRequestHeader(RequestHeader.`TPP-Signature-Certificate`, requestHeaders)
     if (tppSignatureCert.isEmpty) {
       Future(forwardResult)
-    } else { // Dynamic consumer creation/update works in case that RequestHeader.`TPP-Signature-Certificate` is present
-      val certificate = getCertificateFromTppSignatureCertificate(requestHeaders)
+    } else getCertificateFromTppSignatureCertificate(requestHeaders) match {
+      // Header present but unusable: leave the caller's result untouched rather than throwing.
+      // Consumer creation is a side effect of a signed request, not the thing being authorised —
+      // verifySignedRequest is what rejects a bad certificate.
+      case unusable if unusable.isEmpty =>
+        logger.debug(s"getOrCreateConsumer: unusable TPP-Signature-Certificate: $unusable")
+        Future(forwardResult)
+      // Dynamic consumer creation/update, when TPP-Signature-Certificate holds a real certificate
+      case Full(certificate) =>
 
       val extractedEmail = emailPattern.findFirstMatchIn(certificate.getSubjectDN.getName).map(_.group(1))
       val extractOrganisation = organisationlPattern.findFirstMatchIn(certificate.getSubjectDN.getName).map(_.group(1))

From bc08fc098524b07c347fd06724e3e6e3cbef451b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Marko=20Mili=C4=87?= <marko@tesobe.com>
Date: Tue, 21 Jul 2026 14:15:18 +0200
Subject: [PATCH 4/4] fix(psd2): identify the TPP by the certificate its API
 standard actually uses

BEHAVIOUR CHANGE for UK Open Banking endpoints when requirePsd2Certificates=ONLINE.
Read the note at the end before upgrading.

The PSD2 gate (passesPsd2Aisp / passesPsd2Pisp) identified the TPP from the
TPP-Signature-Certificate header regardless of which API standard was being
served. That header is Berlin Group NextGenPSD2. UK Open Banking does not use it
at all: OBIE identifies the TPP by the mTLS transport certificate, which reaches
OBP as PSD2-CERT -- set by the reverse proxy that terminates mTLS, or by
bootstrap.http4s.Http4sMtls in development. A correctly behaving OBIE client was
therefore rejected for omitting a header its specification never mentions.

BerlinGroupCheck.validate already scopes every OTHER piece of Berlin Group
machinery to Berlin Group URLs -- the mandatory headers, the request-signature
verification, the on-the-fly consumer creation. The PSD2 gate was the single
place that crossed that boundary. tppCertificateForStandard brings it in line:
UK Open Banking URLs read PSD2-CERT, everything else keeps the previous code
path byte for byte.

One call site changes, which covers all 10 UK enforcement sites (5 in v3.1.0,
5 in v4.0.1) and any added later. Berlin Group (37 sites) and the OBP-native
endpoints (4) are untouched.

Only the certificate SOURCE changes, not what is done with it: both branches feed
the same regulated-entity lookup, which matches on issuer CN + serial number and
works with any X509 certificate. No eIDAS QCStatement is required -- that is only
needed by the CERTIFICATE mode, which is unaffected.

Verified end to end against a real TPP (OBP-Hola) over mutual TLS:
- before registering the entity: 401 OBP-34102 (regulated entity not found),
  proving the mTLS certificate is now the one being looked up, where previously
  the same call returned OBP-20306 (no certificate in the request at all);
- after registering it: the UK v4.0.1 consent is created, Status
  AWAITINGAUTHORISATION, with the gate logging the matched entity and PSP_AI;
- /berlin-group/v1.3/consents still rejects on missing tpp-signature-certificate,
  unchanged.
RegulatedEntityTest and the mTLS suites pass (13 tests). No test in this repo
sends TPP-Signature-Certificate to a UK endpoint -- only the Berlin Group suites
reference that header.

UPGRADE NOTE. Under eIDAS a TPP holds two different certificates: the QWAC, used
for TLS and therefore surfacing as PSD2-CERT, and the QSEAL, used to sign and
therefore carried in TPP-Signature-Certificate. They have different serial
numbers and often different issuing CAs. Since regulated entities are matched on
issuer CN + serial, a deployment that runs UK endpoints with
requirePsd2Certificates=ONLINE and onboarded its TPPs by QSEAL will now match a
different entity, or none:
- no PSD2-CERT present (no mTLS proxy) -> previously working TPPs get 401;
- QWAC registered as a different entity -> that entity's services decide the call.
This cannot authorise an unregistered party: both paths require a CA + serial
match and fail closed. Deployments on requirePsd2Certificates=NONE are unaffected
because the gate does not run. Register the QWAC alongside the QSEAL to migrate.
---
 .../main/scala/code/api/util/APIUtil.scala    | 47 +++++++++++++++++--
 1 file changed, 42 insertions(+), 5 deletions(-)

diff --git a/obp-api/src/main/scala/code/api/util/APIUtil.scala b/obp-api/src/main/scala/code/api/util/APIUtil.scala
index c17eb54dcc..685f90979a 100644
--- a/obp-api/src/main/scala/code/api/util/APIUtil.scala
+++ b/obp-api/src/main/scala/code/api/util/APIUtil.scala
@@ -3849,19 +3849,56 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{
     currentSupportFormats.toStream.map(_.parse(date, parsePosition)).find(null.!=)
   }
 
+  /**
+   * The certificate identifying the TPP, taken from the channel the request's API standard
+   * actually uses.
+   *
+   * Berlin Group signs requests and carries the signing certificate in TPP-Signature-Certificate.
+   * UK Open Banking does not use that header at all: OBIE identifies the TPP by the mTLS transport
+   * certificate, which reaches us as PSD2-CERT (set by the reverse proxy that terminates mTLS, or
+   * by bootstrap.http4s.Http4sMtls in development). Reading the Berlin Group header on a UK
+   * endpoint therefore rejects a correctly-behaving OBIE client for missing a header its
+   * specification never mentions.
+   *
+   * BerlinGroupCheck.validate already scopes every OTHER piece of Berlin Group machinery -- the
+   * mandatory headers, the request-signature verification, the on-the-fly consumer creation -- to
+   * Berlin Group URLs. The PSD2 gate was the one place that crossed that boundary; this brings it
+   * in line. Non-UK standards keep the previous behaviour byte for byte.
+   *
+   * Note this changes only WHERE the certificate comes from, not what is then done with it: both
+   * paths feed the same regulated-entity lookup, which matches on issuer CN + serial number and so
+   * works with any X509 certificate.
+   */
+  private def tppCertificateForStandard(cc: Option[CallContext]): Box[java.security.cert.X509Certificate] = {
+    val requestHeaders = cc.map(_.requestHeaders).getOrElse(Nil)
+    val url = cc.map(_.url).getOrElse("")
+    val isUkOpenBanking = url.contains(s"/${ApiVersion.ukOpenBankingV31.urlPrefix}/")
+    if (isUkOpenBanking) {
+      `getPSD2-CERT`(requestHeaders) match {
+        case Some(pem) =>
+          tryo(BerlinGroupSigning.parseCertificate(pem)) ?~ X509GeneralError
+        case None =>
+          logger.debug(s"tppCertificateForStandard: UK Open Banking request with no ${RequestHeader.`PSD2-CERT`} header")
+          Failure(X509CannotGetCertificate)
+      }
+    } else {
+      BerlinGroupSigning.getCertificateFromTppSignatureCertificate(requestHeaders)
+    }
+  }
+
   private def passesPsd2ServiceProviderCommon(cc: Option[CallContext], serviceProvider: String) = {
     val result = getPropsValue("requirePsd2Certificates", "NONE") match {
       case value if value.toUpperCase == "ONLINE" =>
         val requestHeaders = cc.map(_.requestHeaders).getOrElse(Nil)
         val consumerName = cc.flatMap(_.consumer.map(_.name.get)).getOrElse("")
-        getCertificateFromTppSignatureCertificate(requestHeaders) match {
-          // No usable TPP-Signature-Certificate: fail closed. passesPsd2ServiceProvider maps a
-          // Failure to a 401 -- this used to throw out of the base64 decode and become a 500.
+        tppCertificateForStandard(cc) match {
+          // No usable certificate: fail closed. passesPsd2ServiceProvider maps a Failure to a 401
+          // -- this used to throw out of the base64 decode and become a 500.
           case failure: Failure =>
-            logger.debug(s"passesPsd2ServiceProvider: no usable TPP-Signature-Certificate: $failure")
+            logger.debug(s"passesPsd2ServiceProvider: no usable TPP certificate: $failure")
             Future(failure)
           case Empty =>
-            logger.debug("passesPsd2ServiceProvider: no TPP-Signature-Certificate header")
+            logger.debug("passesPsd2ServiceProvider: no TPP certificate header")
             Future(Failure(X509CannotGetCertificate))
           case Full(certificate) =>
             for {