http4s mtls dev toggle

docs/MTLS_DEV_MODE.md

@@ -72,7 +72,22 @@ To accept a whole CA instead of individual client certs, import the CA certifica
 
 ### 2. Configure props
 
-In your props file (e.g. `obp-api/src/main/resources/props/default.props`):
+**The short way — no props edit at all:**
+
+```sh
+./flushall_fast_build_and_run.sh --mtls
+```
+
+The flag sources `scripts/mtls_env.sh`, which exports the `OBP_MTLS_*` environment overrides
+(every OBP prop is settable as `OBP_<NAME>` with dots replaced by underscores —
+`APIUtil.getPropsValue` reads the environment ahead of the props file) pointing at the dev
+keystore pair checked into the repo. It also flips `hostname` to `https://` **and** pins
+`local_identity_provider` to the pre-toggle hostname — see the note under "Props reference"
+for why that pinning matters. Anything you pre-export wins, so
+`OBP_MTLS_CLIENT_AUTH=want ./flushall_fast_build_and_run.sh --mtls` works.
+
+To make the mode permanent instead, put it in your props file
+(e.g. `obp-api/src/main/resources/props/default.props`):
 
 ```properties
 mtls.enabled=true
@@ -91,12 +106,20 @@ hostname=https://localhost:8080
 consumer_validation_method_for_consent=CONSUMER_CERTIFICATE
 ```
 
+Only `mtls.enabled` is actually required. The four store props default to the checked-in dev
+pair (`obp-api/src/test/resources/cert/server.jks` / `server.trust.jks`, password `123456`),
+resolved relative to the working directory — so they work when the server is launched from the
+repo root, which is what the run scripts do. Each fallback is logged at WARN naming the resolved
+absolute file, and a missing store fails at startup with the resolved path and working directory
+in the message rather than a bare `FileNotFoundException`.
+
 `run.mode` must be `development` (it is with the standard local run scripts).
 
 ### 3. Run
 
 ```sh
-./flushall_build_and_run.sh
+./flushall_build_and_run.sh --mtls        # or: ./flushall_fast_build_and_run.sh --mtls
+./flushall_build_and_run.sh               # props-driven, if you configured them above
 ```
 
 Boot log confirms the mode:
@@ -170,21 +193,35 @@ Consumer — presenting a different client certificate is rejected.
 
 | Prop | Default | Meaning |
 |---|---|---|
-| `mtls.enabled` | `false` | Master switch. Only honoured when `run.mode=development`. |
-| `mtls.keystore.path` | — (required) | JKS with the server's private key + certificate. |
-| `mtls.keystore.password` | — (required) | Password for keystore and key. |
-| `mtls.truststore.path` | — (required) | JKS with client certificates / CAs the server accepts. |
-| `mtls.truststore.password` | — (required) | Truststore password. |
+| `mtls.enabled` | `false` | Master switch. Only honoured when `run.mode=development`. The one genuinely required prop. |
+| `mtls.keystore.path` | `obp-api/src/test/resources/cert/server.jks` | Store with the server's private key + certificate. `.p12`/`.pfx` are read as PKCS12, anything else as JKS. |
+| `mtls.keystore.password` | `123456` | Password for keystore and key. |
+| `mtls.truststore.path` | `obp-api/src/test/resources/cert/server.trust.jks` | Store with client certificates / CAs the server accepts. Same type detection as the keystore. |
+| `mtls.truststore.password` | `123456` | Truststore password. |
 | `mtls.client_auth` | `need` | `need` rejects certless handshakes; `want` makes the client certificate optional. |
 
+Each of these is also settable from the environment as `OBP_MTLS_ENABLED`,
+`OBP_MTLS_KEYSTORE_PATH`, … which is what `--mtls` uses.
+
+> **`hostname` and `local_identity_provider` move together.** Turning mTLS on means generated
+> links should say `https://`, so `hostname` changes. But `hostname` is also the default for
+> `local_identity_provider` (`code/api/constant/constant.scala`), which is the `provider` column
+> every `AuthUser` / `ResourceUser` row is keyed on. Changing `hostname` alone gives your existing
+> local users a different provider string, orphaning them — logins start failing after a toggle.
+> `scripts/mtls_env.sh` handles this by pinning `local_identity_provider` to the pre-toggle
+> hostname. If you configure mTLS through props instead, set `local_identity_provider` explicitly
+> to whatever `hostname` was before you added the `https://`.
+
 ## Troubleshooting
 
 | Symptom | Cause |
 |---|---|
 | Boot warning `mtls.enabled=true is ignored` | `run.mode` is not `development`. |
-| Boot fails with `requires the props value 'mtls.…'` | One of the four keystore/truststore props is missing. |
+| Boot fails with `points at '…', which does not exist` | The store file isn't there. If you relied on the defaults, the server was launched from somewhere other than the repo root — the message prints the resolved path and the working directory. Use `--mtls` (absolute paths) or set the props absolutely. |
+| Logins that worked over plain HTTP now fail | `hostname` changed scheme without `local_identity_provider` being pinned — see the note under "Props reference". |
 | `curl: (35)` / `alert certificate required` | No (or untrusted) client certificate in `need` mode — check the cert is in `server.trust.jks`. |
 | `curl: (60) SSL certificate problem` | curl doesn't trust the server cert — pass `--cacert server.crt` (or `-k` for a quick look). |
+| Client rejects the server cert with "no alternative certificate subject name matches" | The default `server.jks` leaf has **no SAN**. curl/OpenSSL and Java's `DefaultHostnameVerifier` fall back to CN (`localhost`) so both accept it, but stricter clients won't. Point `mtls.keystore.path` at `obp-api/src/test/resources/cert/localhost_san_dns_ip.pfx` (password `123456`), which carries `DNS:localhost, IP:127.0.0.1`. |
 | Plain `http://` requests hang or error | The port serves HTTPS when mTLS is enabled — use `https://`. |
 | Cert reaches OBP but consumer lookup fails | The Consumer's Client Certificate field doesn't contain the client cert's PEM (lookup is by PEM match, with a whitespace-normalized fallback). |
 | DirectLogin returns `OBP-20073: The user email has not been validated` | A freshly created user must validate their email first. In local dev, either click the link from the validation email (see the `mail.*` props) or set the flag directly in the DB: `update authuser set validated=true where username='YOUR_USER';` |
@@ -208,6 +245,8 @@ header. Two important rules for any proxy config:
 |---|---|
 | `obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala` | Props, SSLContext/TLSContext construction, `PSD2-CERT` injection middleware. |
 | `obp-api/src/main/scala/bootstrap/http4s/Http4sServer.scala` | `mtls.enabled` branch on the Ember builder. |
+| `scripts/mtls_env.sh` | The `OBP_MTLS_*` environment overrides behind the `--mtls` flag of both `flushall_*build_and_run.sh` scripts. Sourceable on its own. |
+| `scripts/java_env.sh` | Selects a JDK >= 17 for the run scripts. The build compiles with `-release 17`; on a default JDK 11 it otherwise fails with the misleading `'17' is not a valid choice for '-release'`. |
 | `obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsTest.scala` | Unit tests: PEM encoding, header injection/stripping, SSLContext from the checked-in keystores. |
 | `obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsHandshakeTest.scala` | End-to-end: real Ember server + real mTLS handshake; proves the verified client cert surfaces as `PSD2-CERT` and certless handshakes are rejected. |
 

flushall_build_and_run.sh

@@ -11,6 +11,7 @@
 #   ./build_and_run.sh              - Build and run with Redis flush
 #   ./build_and_run.sh --no-flush   - Build and run without Redis flush
 #   ./build_and_run.sh --background - Run server in background
+#   ./build_and_run.sh --mtls       - Serve HTTPS with mutual TLS (dev only)
 #
 # The HTTP4S server runs on the port configured in your props file
 # (default: 8080 for dev.port, or 8086 for hostname port)
@@ -21,6 +22,7 @@
 # Parse arguments
 FLUSH_REDIS=true
 RUN_BACKGROUND=false
+USE_MTLS=false
 
 for arg in "$@"; do
     case $arg in
@@ -32,9 +34,16 @@
             RUN_BACKGROUND=true
             echo ">>> Server will run in background"
             ;;
+        --mtls)
+            USE_MTLS=true
+            echo ">>> mTLS mode requested (dev-only in-process TLS termination)"
+            ;;
     esac
 done
 
+# Select a JDK >= 17 before any Maven work (the build compiles with -release 17).
+. "$(dirname "${BASH_SOURCE[0]}")/scripts/java_env.sh"
+
 ################################################################################
 # FLUSH REDIS CACHE (OPTIONAL)
 ################################################################################
@@ -109,8 +118,12 @@
 # Show last 3 lines of build output in real-time
 tail -n 3 -f build.log &
 TAIL_PID=$!
+# set +e: this script runs under `set -e`, which would abort at a failing build
+# before the BUILD_EXIT check below could print the log tail.
+set +e
 mvn -pl obp-api -am clean package -DskipTests=true -Dmaven.test.skip=true -T 4 > build.log 2>&1
 BUILD_EXIT=$?
+set -e
 kill $TAIL_PID 2>/dev/null || true
 wait $TAIL_PID 2>/dev/null || true
 
@@ -132,6 +145,12 @@
 # RUN HTTP4S SERVER
 ################################################################################
 
+# Applied after the build so it only affects the running server, never compilation.
+if [ "$USE_MTLS" = true ]; then
+    . "$(dirname "${BASH_SOURCE[0]}")/scripts/mtls_env.sh"
+    echo ""
+fi
+
 echo "=========================================="
 if [ "$RUN_BACKGROUND" = true ]; then
     echo "Starting HTTP4S server (background)..."

flushall_fast_build_and_run.sh

@@ -17,6 +17,7 @@
 #   ./fast_build_and_run.sh --online       - Check remote repositories
 #   ./fast_build_and_run.sh --no-flush     - Skip Redis flush
 #   ./fast_build_and_run.sh --background   - Run server in background
+#   ./fast_build_and_run.sh --mtls         - Serve HTTPS with mutual TLS (dev only)
 #
 # Typical speedup: 2-5x faster than regular build for incremental changes
 ################################################################################
@@ -28,6 +29,7 @@
 OFFLINE_FLAG="-o"  # Default to offline mode for faster builds
 FLUSH_REDIS=true
 RUN_BACKGROUND=false
+USE_MTLS=false
 
 for arg in "$@"; do
     case $arg in
@@ -47,9 +49,16 @@
             RUN_BACKGROUND=true
             echo ">>> Server will run in background"
             ;;
+        --mtls)
+            USE_MTLS=true
+            echo ">>> mTLS mode requested (dev-only in-process TLS termination)"
+            ;;
     esac
 done
 
+# Select a JDK >= 17 before any Maven work (the build compiles with -release 17).
+. "$(dirname "${BASH_SOURCE[0]}")/scripts/java_env.sh"
+
 # Show offline mode status if not explicitly set
 if [ "$OFFLINE_FLAG" = "-o" ]; then
     echo ">>> Using offline mode (default) - use --online to check remote repos"
@@ -165,6 +174,10 @@
 } > fast_build.log
 
 # Run Maven build and append to log
+# set +e around the build: this script runs under `set -e`, which would abort here
+# on a failing build and make BUILD_EXIT_CODE — and the auto-retry-with-clean block
+# below — unreachable. We want to inspect the failure, not die at it.
+set +e
 mvn -pl obp-api -am \
     $DO_CLEAN \
     package \
@@ -175,8 +188,8 @@
     -Dcheckstyle.skip=true \
     -Dspotbugs.skip=true \
     -Dpmd.skip=true >> fast_build.log 2>&1
-
 BUILD_EXIT_CODE=$?
+set -e
 
 # Record build end time and calculate duration
 if command -v gdate &> /dev/null; then
@@ -237,7 +250,8 @@
             echo ""
         } > fast_build.log
 
-        # Run clean build
+        # Run clean build (set +e for the same reason as the first invocation)
+        set +e
         mvn -pl obp-api -am \
             clean \
             package \
@@ -248,8 +262,8 @@
             -Dcheckstyle.skip=true \
             -Dspotbugs.skip=true \
             -Dpmd.skip=true >> fast_build.log 2>&1
-
         BUILD_EXIT_CODE=$?
+        set -e
 
         # Record retry end time and calculate duration
         if command -v gdate &> /dev/null; then
@@ -300,6 +314,12 @@
 # RUN HTTP4S SERVER
 ################################################################################
 
+# Applied after the build so it only affects the running server, never compilation.
+if [ "$USE_MTLS" = true ]; then
+    . "$(dirname "${BASH_SOURCE[0]}")/scripts/mtls_env.sh"
+    echo ""
+fi
+
 echo "=========================================="
 if [ "$RUN_BACKGROUND" = true ]; then
     echo "Starting HTTP4S server (background)..."

obp-api/src/main/resources/props/sample.props.template

@@ -166,7 +166,9 @@ jwt.use.ssl=false
 ## that forwards the verified client certificate as the PSD2-CERT request header.
 ## When enabled, dev.port serves HTTPS with mutual TLS and the verified client certificate is
 ## injected as the PSD2-CERT header (any client-supplied PSD2-CERT header is stripped).
-## The paths below point at the JKS pair checked into the repo for local development.
+## Only mtls.enabled is required: the four store props below default to exactly these values,
+## the JKS pair checked into the repo for local development. Set them to use your own certificates.
+## Easiest switch is not to edit this file at all: ./flushall_fast_build_and_run.sh --mtls
 # mtls.enabled=true
 # mtls.keystore.path=obp-api/src/test/resources/cert/server.jks
 # mtls.keystore.password=123456

obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala

@@ -1,6 +1,6 @@
 package bootstrap.http4s
 
-import java.io.FileInputStream
+import java.io.{File, FileInputStream}
 import java.nio.charset.StandardCharsets
 import java.security.KeyStore
 import java.security.cert.X509Certificate
@@ -32,6 +32,12 @@ import org.typelevel.ci.CIString
  *   mtls.truststore.password=...
  *   mtls.client_auth=need                           (need = reject certless handshakes; want = optional)
  *
+ * Only `mtls.enabled` is mandatory: the four store props fall back to the checked-in dev keystore
+ * pair (see DevKeystorePath below) so that switching a local server to mTLS is a single toggle.
+ * Every fallback is logged at WARN naming the resolved file, so the default is never silent.
+ * Like every OBP prop these are also settable from the environment as OBP_MTLS_ENABLED,
+ * OBP_MTLS_KEYSTORE_PATH, ... (see APIUtil.getPropsValue, which reads the environment first).
+ *
  * The prop is honoured ONLY in Development run mode. In production OBP must sit behind a reverse
  * proxy that terminates mTLS and forwards the client certificate as the PSD2-CERT header — this
  * feature exists so local development does not need that proxy.
@@ -57,29 +63,72 @@ object Http4sMtls extends MdcLoggable {
     } else propEnabled
   }
 
+  // The checked-in dev pair: a CN=localhost server keypair and the TESOBE CA that signs the dev
+  // client certificates. Repo-relative, so they resolve when the server is launched from the repo
+  // root (which is what the build_and_run scripts do). Only ever reached in Development run mode.
+  private val DevKeystorePath = "obp-api/src/test/resources/cert/server.jks"
+  private val DevTruststorePath = "obp-api/src/test/resources/cert/server.trust.jks"
+  private val DevStorePassword = "123456"
+
+  /** The OBP_-prefixed environment variable APIUtil.getPropsValue consults ahead of the props file. */
+  private def envVarOf(propName: String): String = "OBP_" + propName.replace('.', '_').toUpperCase
+
   lazy val config: MtlsConfig = {
-    def required(name: String): String = APIUtil.getPropsValue(name)
-      .openOrThrowException(s"mtls.enabled=true requires the props value '$name' to be set")
+    def prop(name: String): Option[String] =
+      APIUtil.getPropsValue(name).toOption.map(_.trim).filter(_.nonEmpty)
+
+    // Returns the absolute store path plus its password, defaulting both to the dev pair.
+    def store(pathProp: String, devPath: String, passwordProp: String): (String, String) = {
+      val path = prop(pathProp).getOrElse {
+        logger.warn(s"'$pathProp' is not set — falling back to the checked-in dev store '$devPath'. " +
+          s"Set '$pathProp' (or ${envVarOf(pathProp)}) to use your own certificates.")
+        devPath
+      }
+      val file = new File(path)
+      if (!file.isFile) throw new RuntimeException(
+        s"mtls.enabled=true but '$pathProp' points at '$path', which does not exist " +
+          s"(resolved to ${file.getAbsolutePath} from working directory ${new File(".").getAbsolutePath}). " +
+          s"Launch from the repo root or set '$pathProp' to an absolute path.")
+      val password = prop(passwordProp).getOrElse {
+        logger.warn(s"'$passwordProp' is not set — falling back to the checked-in dev store password.")
+        DevStorePassword
+      }
+      (file.getAbsolutePath, password)
+    }
+
+    val (keystorePath, keystorePassword) = store("mtls.keystore.path", DevKeystorePath, "mtls.keystore.password")
+    val (truststorePath, truststorePassword) = store("mtls.truststore.path", DevTruststorePath, "mtls.truststore.password")
     MtlsConfig(
-      keystorePath = required("mtls.keystore.path"),
-      keystorePassword = required("mtls.keystore.password"),
-      truststorePath = required("mtls.truststore.path"),
-      truststorePassword = required("mtls.truststore.password"),
+      keystorePath = keystorePath,
+      keystorePassword = keystorePassword,
+      truststorePath = truststorePath,
+      truststorePassword = truststorePassword,
       needClientAuth = APIUtil.getPropsValue("mtls.client_auth", "need").toLowerCase != "want"
     )
   }
 
+  /**
+   * KeyStore type for a store file, by extension: `.p12` / `.pfx` are PKCS12, everything else JKS.
+   * Both formats are common for the certificates a TPP hands over — `obp-api/src/test/resources/cert`
+   * carries examples of each — and guessing wrong fails at load with an opaque parse error.
+   */
+  private[http4s] def keyStoreTypeOf(path: String): String =
+    path.toLowerCase match {
+      case p if p.endsWith(".p12") || p.endsWith(".pfx") => "PKCS12"
+      case _ => "JKS"
+    }
+
   def buildSslContext(config: MtlsConfig): SSLContext = {
-    def loadJks(path: String, password: String): KeyStore = {
-      val keyStore = KeyStore.getInstance("JKS")
+    def loadStore(path: String, password: String): KeyStore = {
+      val keyStore = KeyStore.getInstance(keyStoreTypeOf(path))
       val in = new FileInputStream(path)
       try keyStore.load(in, password.toCharArray) finally in.close()
       keyStore
     }
     val keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm)
-    keyManagerFactory.init(loadJks(config.keystorePath, config.keystorePassword), config.keystorePassword.toCharArray)
+    keyManagerFactory.init(loadStore(config.keystorePath, config.keystorePassword), config.keystorePassword.toCharArray)
     val trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm)
-    trustManagerFactory.init(loadJks(config.truststorePath, config.truststorePassword))
+    trustManagerFactory.init(loadStore(config.truststorePath, config.truststorePassword))
     val sslContext = SSLContext.getInstance("TLS")
     sslContext.init(keyManagerFactory.getKeyManagers, trustManagerFactory.getTrustManagers, null)
     sslContext