diff --git a/docs/MTLS_DEV_MODE.md b/docs/MTLS_DEV_MODE.md index 8febd8147a..867826bf6f 100644 --- a/docs/MTLS_DEV_MODE.md +++ b/docs/MTLS_DEV_MODE.md @@ -72,7 +72,22 @@ To accept a whole CA instead of individual client certs, import the CA certifica ### 2. Configure props -In your props file (e.g. `obp-api/src/main/resources/props/default.props`): +**The short way — no props edit at all:** + +```sh +./flushall_fast_build_and_run.sh --mtls +``` + +The flag sources `scripts/mtls_env.sh`, which exports the `OBP_MTLS_*` environment overrides +(every OBP prop is settable as `OBP_` with dots replaced by underscores — +`APIUtil.getPropsValue` reads the environment ahead of the props file) pointing at the dev +keystore pair checked into the repo. It also flips `hostname` to `https://` **and** pins +`local_identity_provider` to the pre-toggle hostname — see the note under "Props reference" +for why that pinning matters. Anything you pre-export wins, so +`OBP_MTLS_CLIENT_AUTH=want ./flushall_fast_build_and_run.sh --mtls` works. + +To make the mode permanent instead, put it in your props file +(e.g. `obp-api/src/main/resources/props/default.props`): ```properties mtls.enabled=true @@ -91,12 +106,20 @@ hostname=https://localhost:8080 consumer_validation_method_for_consent=CONSUMER_CERTIFICATE ``` +Only `mtls.enabled` is actually required. The four store props default to the checked-in dev +pair (`obp-api/src/test/resources/cert/server.jks` / `server.trust.jks`, password `123456`), +resolved relative to the working directory — so they work when the server is launched from the +repo root, which is what the run scripts do. Each fallback is logged at WARN naming the resolved +absolute file, and a missing store fails at startup with the resolved path and working directory +in the message rather than a bare `FileNotFoundException`. + `run.mode` must be `development` (it is with the standard local run scripts). ### 3. Run ```sh -./flushall_build_and_run.sh +./flushall_build_and_run.sh --mtls # or: ./flushall_fast_build_and_run.sh --mtls +./flushall_build_and_run.sh # props-driven, if you configured them above ``` Boot log confirms the mode: @@ -170,21 +193,35 @@ Consumer — presenting a different client certificate is rejected. | Prop | Default | Meaning | |---|---|---| -| `mtls.enabled` | `false` | Master switch. Only honoured when `run.mode=development`. | -| `mtls.keystore.path` | — (required) | JKS with the server's private key + certificate. | -| `mtls.keystore.password` | — (required) | Password for keystore and key. | -| `mtls.truststore.path` | — (required) | JKS with client certificates / CAs the server accepts. | -| `mtls.truststore.password` | — (required) | Truststore password. | +| `mtls.enabled` | `false` | Master switch. Only honoured when `run.mode=development`. The one genuinely required prop. | +| `mtls.keystore.path` | `obp-api/src/test/resources/cert/server.jks` | Store with the server's private key + certificate. `.p12`/`.pfx` are read as PKCS12, anything else as JKS. | +| `mtls.keystore.password` | `123456` | Password for keystore and key. | +| `mtls.truststore.path` | `obp-api/src/test/resources/cert/server.trust.jks` | Store with client certificates / CAs the server accepts. Same type detection as the keystore. | +| `mtls.truststore.password` | `123456` | Truststore password. | | `mtls.client_auth` | `need` | `need` rejects certless handshakes; `want` makes the client certificate optional. | +Each of these is also settable from the environment as `OBP_MTLS_ENABLED`, +`OBP_MTLS_KEYSTORE_PATH`, … which is what `--mtls` uses. + +> **`hostname` and `local_identity_provider` move together.** Turning mTLS on means generated +> links should say `https://`, so `hostname` changes. But `hostname` is also the default for +> `local_identity_provider` (`code/api/constant/constant.scala`), which is the `provider` column +> every `AuthUser` / `ResourceUser` row is keyed on. Changing `hostname` alone gives your existing +> local users a different provider string, orphaning them — logins start failing after a toggle. +> `scripts/mtls_env.sh` handles this by pinning `local_identity_provider` to the pre-toggle +> hostname. If you configure mTLS through props instead, set `local_identity_provider` explicitly +> to whatever `hostname` was before you added the `https://`. + ## Troubleshooting | Symptom | Cause | |---|---| | Boot warning `mtls.enabled=true is ignored` | `run.mode` is not `development`. | -| Boot fails with `requires the props value 'mtls.…'` | One of the four keystore/truststore props is missing. | +| Boot fails with `points at '…', which does not exist` | The store file isn't there. If you relied on the defaults, the server was launched from somewhere other than the repo root — the message prints the resolved path and the working directory. Use `--mtls` (absolute paths) or set the props absolutely. | +| Logins that worked over plain HTTP now fail | `hostname` changed scheme without `local_identity_provider` being pinned — see the note under "Props reference". | | `curl: (35)` / `alert certificate required` | No (or untrusted) client certificate in `need` mode — check the cert is in `server.trust.jks`. | | `curl: (60) SSL certificate problem` | curl doesn't trust the server cert — pass `--cacert server.crt` (or `-k` for a quick look). | +| Client rejects the server cert with "no alternative certificate subject name matches" | The default `server.jks` leaf has **no SAN**. curl/OpenSSL and Java's `DefaultHostnameVerifier` fall back to CN (`localhost`) so both accept it, but stricter clients won't. Point `mtls.keystore.path` at `obp-api/src/test/resources/cert/localhost_san_dns_ip.pfx` (password `123456`), which carries `DNS:localhost, IP:127.0.0.1`. | | Plain `http://` requests hang or error | The port serves HTTPS when mTLS is enabled — use `https://`. | | Cert reaches OBP but consumer lookup fails | The Consumer's Client Certificate field doesn't contain the client cert's PEM (lookup is by PEM match, with a whitespace-normalized fallback). | | DirectLogin returns `OBP-20073: The user email has not been validated` | A freshly created user must validate their email first. In local dev, either click the link from the validation email (see the `mail.*` props) or set the flag directly in the DB: `update authuser set validated=true where username='YOUR_USER';` | @@ -208,6 +245,8 @@ header. Two important rules for any proxy config: |---|---| | `obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala` | Props, SSLContext/TLSContext construction, `PSD2-CERT` injection middleware. | | `obp-api/src/main/scala/bootstrap/http4s/Http4sServer.scala` | `mtls.enabled` branch on the Ember builder. | +| `scripts/mtls_env.sh` | The `OBP_MTLS_*` environment overrides behind the `--mtls` flag of both `flushall_*build_and_run.sh` scripts. Sourceable on its own. | +| `scripts/java_env.sh` | Selects a JDK >= 17 for the run scripts. The build compiles with `-release 17`; on a default JDK 11 it otherwise fails with the misleading `'17' is not a valid choice for '-release'`. | | `obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsTest.scala` | Unit tests: PEM encoding, header injection/stripping, SSLContext from the checked-in keystores. | | `obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsHandshakeTest.scala` | End-to-end: real Ember server + real mTLS handshake; proves the verified client cert surfaces as `PSD2-CERT` and certless handshakes are rejected. | diff --git a/flushall_build_and_run.sh b/flushall_build_and_run.sh index 13a8bfa6c5..ae6485bdca 100755 --- a/flushall_build_and_run.sh +++ b/flushall_build_and_run.sh @@ -11,6 +11,7 @@ # ./build_and_run.sh - Build and run with Redis flush # ./build_and_run.sh --no-flush - Build and run without Redis flush # ./build_and_run.sh --background - Run server in background +# ./build_and_run.sh --mtls - Serve HTTPS with mutual TLS (dev only) # # The HTTP4S server runs on the port configured in your props file # (default: 8080 for dev.port, or 8086 for hostname port) @@ -21,6 +22,7 @@ set -e # Exit on error # Parse arguments FLUSH_REDIS=true RUN_BACKGROUND=false +USE_MTLS=false for arg in "$@"; do case $arg in @@ -32,9 +34,16 @@ for arg in "$@"; do RUN_BACKGROUND=true echo ">>> Server will run in background" ;; + --mtls) + USE_MTLS=true + echo ">>> mTLS mode requested (dev-only in-process TLS termination)" + ;; esac done +# Select a JDK >= 17 before any Maven work (the build compiles with -release 17). +. "$(dirname "${BASH_SOURCE[0]}")/scripts/java_env.sh" + ################################################################################ # FLUSH REDIS CACHE (OPTIONAL) ################################################################################ @@ -109,8 +118,12 @@ echo "Build output will be saved to: build.log" # Show last 3 lines of build output in real-time tail -n 3 -f build.log & TAIL_PID=$! +# set +e: this script runs under `set -e`, which would abort at a failing build +# before the BUILD_EXIT check below could print the log tail. +set +e mvn -pl obp-api -am clean package -DskipTests=true -Dmaven.test.skip=true -T 4 > build.log 2>&1 BUILD_EXIT=$? +set -e kill $TAIL_PID 2>/dev/null || true wait $TAIL_PID 2>/dev/null || true @@ -132,6 +145,12 @@ echo "" # RUN HTTP4S SERVER ################################################################################ +# Applied after the build so it only affects the running server, never compilation. +if [ "$USE_MTLS" = true ]; then + . "$(dirname "${BASH_SOURCE[0]}")/scripts/mtls_env.sh" + echo "" +fi + echo "==========================================" if [ "$RUN_BACKGROUND" = true ]; then echo "Starting HTTP4S server (background)..." diff --git a/flushall_fast_build_and_run.sh b/flushall_fast_build_and_run.sh index 88669ab6e9..7483e2ce7b 100755 --- a/flushall_fast_build_and_run.sh +++ b/flushall_fast_build_and_run.sh @@ -17,6 +17,7 @@ # ./fast_build_and_run.sh --online - Check remote repositories # ./fast_build_and_run.sh --no-flush - Skip Redis flush # ./fast_build_and_run.sh --background - Run server in background +# ./fast_build_and_run.sh --mtls - Serve HTTPS with mutual TLS (dev only) # # Typical speedup: 2-5x faster than regular build for incremental changes ################################################################################ @@ -28,6 +29,7 @@ DO_CLEAN="" OFFLINE_FLAG="-o" # Default to offline mode for faster builds FLUSH_REDIS=true RUN_BACKGROUND=false +USE_MTLS=false for arg in "$@"; do case $arg in @@ -47,9 +49,16 @@ for arg in "$@"; do RUN_BACKGROUND=true echo ">>> Server will run in background" ;; + --mtls) + USE_MTLS=true + echo ">>> mTLS mode requested (dev-only in-process TLS termination)" + ;; esac done +# Select a JDK >= 17 before any Maven work (the build compiles with -release 17). +. "$(dirname "${BASH_SOURCE[0]}")/scripts/java_env.sh" + # Show offline mode status if not explicitly set if [ "$OFFLINE_FLAG" = "-o" ]; then echo ">>> Using offline mode (default) - use --online to check remote repos" @@ -165,6 +174,10 @@ fi } > fast_build.log # Run Maven build and append to log +# set +e around the build: this script runs under `set -e`, which would abort here +# on a failing build and make BUILD_EXIT_CODE — and the auto-retry-with-clean block +# below — unreachable. We want to inspect the failure, not die at it. +set +e mvn -pl obp-api -am \ $DO_CLEAN \ package \ @@ -175,8 +188,8 @@ mvn -pl obp-api -am \ -Dcheckstyle.skip=true \ -Dspotbugs.skip=true \ -Dpmd.skip=true >> fast_build.log 2>&1 - BUILD_EXIT_CODE=$? +set -e # Record build end time and calculate duration if command -v gdate &> /dev/null; then @@ -237,7 +250,8 @@ if [ $BUILD_EXIT_CODE -ne 0 ] && [ -z "$DO_CLEAN" ]; then echo "" } > fast_build.log - # Run clean build + # Run clean build (set +e for the same reason as the first invocation) + set +e mvn -pl obp-api -am \ clean \ package \ @@ -248,8 +262,8 @@ if [ $BUILD_EXIT_CODE -ne 0 ] && [ -z "$DO_CLEAN" ]; then -Dcheckstyle.skip=true \ -Dspotbugs.skip=true \ -Dpmd.skip=true >> fast_build.log 2>&1 - BUILD_EXIT_CODE=$? + set -e # Record retry end time and calculate duration if command -v gdate &> /dev/null; then @@ -300,6 +314,12 @@ echo "" # RUN HTTP4S SERVER ################################################################################ +# Applied after the build so it only affects the running server, never compilation. +if [ "$USE_MTLS" = true ]; then + . "$(dirname "${BASH_SOURCE[0]}")/scripts/mtls_env.sh" + echo "" +fi + echo "==========================================" if [ "$RUN_BACKGROUND" = true ]; then echo "Starting HTTP4S server (background)..." diff --git a/obp-api/src/main/resources/props/sample.props.template b/obp-api/src/main/resources/props/sample.props.template index c5ac75d4c4..0409a9dd82 100644 --- a/obp-api/src/main/resources/props/sample.props.template +++ b/obp-api/src/main/resources/props/sample.props.template @@ -166,7 +166,9 @@ jwt.use.ssl=false ## that forwards the verified client certificate as the PSD2-CERT request header. ## When enabled, dev.port serves HTTPS with mutual TLS and the verified client certificate is ## injected as the PSD2-CERT header (any client-supplied PSD2-CERT header is stripped). -## The paths below point at the JKS pair checked into the repo for local development. +## Only mtls.enabled is required: the four store props below default to exactly these values, +## the JKS pair checked into the repo for local development. Set them to use your own certificates. +## Easiest switch is not to edit this file at all: ./flushall_fast_build_and_run.sh --mtls # mtls.enabled=true # mtls.keystore.path=obp-api/src/test/resources/cert/server.jks # mtls.keystore.password=123456 diff --git a/obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala b/obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala index 6060652d63..56fc785810 100644 --- a/obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala +++ b/obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala @@ -1,6 +1,6 @@ package bootstrap.http4s -import java.io.FileInputStream +import java.io.{File, FileInputStream} import java.nio.charset.StandardCharsets import java.security.KeyStore import java.security.cert.X509Certificate @@ -32,6 +32,12 @@ import org.typelevel.ci.CIString * mtls.truststore.password=... * mtls.client_auth=need (need = reject certless handshakes; want = optional) * + * Only `mtls.enabled` is mandatory: the four store props fall back to the checked-in dev keystore + * pair (see DevKeystorePath below) so that switching a local server to mTLS is a single toggle. + * Every fallback is logged at WARN naming the resolved file, so the default is never silent. + * Like every OBP prop these are also settable from the environment as OBP_MTLS_ENABLED, + * OBP_MTLS_KEYSTORE_PATH, ... (see APIUtil.getPropsValue, which reads the environment first). + * * The prop is honoured ONLY in Development run mode. In production OBP must sit behind a reverse * proxy that terminates mTLS and forwards the client certificate as the PSD2-CERT header — this * feature exists so local development does not need that proxy. @@ -57,29 +63,72 @@ object Http4sMtls extends MdcLoggable { } else propEnabled } + // The checked-in dev pair: a CN=localhost server keypair and the TESOBE CA that signs the dev + // client certificates. Repo-relative, so they resolve when the server is launched from the repo + // root (which is what the build_and_run scripts do). Only ever reached in Development run mode. + private val DevKeystorePath = "obp-api/src/test/resources/cert/server.jks" + private val DevTruststorePath = "obp-api/src/test/resources/cert/server.trust.jks" + private val DevStorePassword = "123456" + + /** The OBP_-prefixed environment variable APIUtil.getPropsValue consults ahead of the props file. */ + private def envVarOf(propName: String): String = "OBP_" + propName.replace('.', '_').toUpperCase + lazy val config: MtlsConfig = { - def required(name: String): String = APIUtil.getPropsValue(name) - .openOrThrowException(s"mtls.enabled=true requires the props value '$name' to be set") + def prop(name: String): Option[String] = + APIUtil.getPropsValue(name).toOption.map(_.trim).filter(_.nonEmpty) + + // Returns the absolute store path plus its password, defaulting both to the dev pair. + def store(pathProp: String, devPath: String, passwordProp: String): (String, String) = { + val path = prop(pathProp).getOrElse { + logger.warn(s"'$pathProp' is not set — falling back to the checked-in dev store '$devPath'. " + + s"Set '$pathProp' (or ${envVarOf(pathProp)}) to use your own certificates.") + devPath + } + val file = new File(path) + if (!file.isFile) throw new RuntimeException( + s"mtls.enabled=true but '$pathProp' points at '$path', which does not exist " + + s"(resolved to ${file.getAbsolutePath} from working directory ${new File(".").getAbsolutePath}). " + + s"Launch from the repo root or set '$pathProp' to an absolute path.") + val password = prop(passwordProp).getOrElse { + logger.warn(s"'$passwordProp' is not set — falling back to the checked-in dev store password.") + DevStorePassword + } + (file.getAbsolutePath, password) + } + + val (keystorePath, keystorePassword) = store("mtls.keystore.path", DevKeystorePath, "mtls.keystore.password") + val (truststorePath, truststorePassword) = store("mtls.truststore.path", DevTruststorePath, "mtls.truststore.password") MtlsConfig( - keystorePath = required("mtls.keystore.path"), - keystorePassword = required("mtls.keystore.password"), - truststorePath = required("mtls.truststore.path"), - truststorePassword = required("mtls.truststore.password"), + keystorePath = keystorePath, + keystorePassword = keystorePassword, + truststorePath = truststorePath, + truststorePassword = truststorePassword, needClientAuth = APIUtil.getPropsValue("mtls.client_auth", "need").toLowerCase != "want" ) } + /** + * KeyStore type for a store file, by extension: `.p12` / `.pfx` are PKCS12, everything else JKS. + * Both formats are common for the certificates a TPP hands over — `obp-api/src/test/resources/cert` + * carries examples of each — and guessing wrong fails at load with an opaque parse error. + */ + private[http4s] def keyStoreTypeOf(path: String): String = + path.toLowerCase match { + case p if p.endsWith(".p12") || p.endsWith(".pfx") => "PKCS12" + case _ => "JKS" + } + def buildSslContext(config: MtlsConfig): SSLContext = { - def loadJks(path: String, password: String): KeyStore = { - val keyStore = KeyStore.getInstance("JKS") + def loadStore(path: String, password: String): KeyStore = { + val keyStore = KeyStore.getInstance(keyStoreTypeOf(path)) val in = new FileInputStream(path) try keyStore.load(in, password.toCharArray) finally in.close() keyStore } val keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm) - keyManagerFactory.init(loadJks(config.keystorePath, config.keystorePassword), config.keystorePassword.toCharArray) + keyManagerFactory.init(loadStore(config.keystorePath, config.keystorePassword), config.keystorePassword.toCharArray) val trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm) - trustManagerFactory.init(loadJks(config.truststorePath, config.truststorePassword)) + trustManagerFactory.init(loadStore(config.truststorePath, config.truststorePassword)) val sslContext = SSLContext.getInstance("TLS") sslContext.init(keyManagerFactory.getKeyManagers, trustManagerFactory.getTrustManagers, null) sslContext diff --git a/obp-api/src/main/scala/code/api/util/APIUtil.scala b/obp-api/src/main/scala/code/api/util/APIUtil.scala index cd283759a1..685f90979a 100644 --- a/obp-api/src/main/scala/code/api/util/APIUtil.scala +++ b/obp-api/src/main/scala/code/api/util/APIUtil.scala @@ -3849,32 +3849,79 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{ currentSupportFormats.toStream.map(_.parse(date, parsePosition)).find(null.!=) } + /** + * The certificate identifying the TPP, taken from the channel the request's API standard + * actually uses. + * + * Berlin Group signs requests and carries the signing certificate in TPP-Signature-Certificate. + * UK Open Banking does not use that header at all: OBIE identifies the TPP by the mTLS transport + * certificate, which reaches us as PSD2-CERT (set by the reverse proxy that terminates mTLS, or + * by bootstrap.http4s.Http4sMtls in development). Reading the Berlin Group header on a UK + * endpoint therefore rejects a correctly-behaving OBIE client for missing a header its + * specification never mentions. + * + * BerlinGroupCheck.validate already scopes every OTHER piece of Berlin Group machinery -- the + * mandatory headers, the request-signature verification, the on-the-fly consumer creation -- to + * Berlin Group URLs. The PSD2 gate was the one place that crossed that boundary; this brings it + * in line. Non-UK standards keep the previous behaviour byte for byte. + * + * Note this changes only WHERE the certificate comes from, not what is then done with it: both + * paths feed the same regulated-entity lookup, which matches on issuer CN + serial number and so + * works with any X509 certificate. + */ + private def tppCertificateForStandard(cc: Option[CallContext]): Box[java.security.cert.X509Certificate] = { + val requestHeaders = cc.map(_.requestHeaders).getOrElse(Nil) + val url = cc.map(_.url).getOrElse("") + val isUkOpenBanking = url.contains(s"/${ApiVersion.ukOpenBankingV31.urlPrefix}/") + if (isUkOpenBanking) { + `getPSD2-CERT`(requestHeaders) match { + case Some(pem) => + tryo(BerlinGroupSigning.parseCertificate(pem)) ?~ X509GeneralError + case None => + logger.debug(s"tppCertificateForStandard: UK Open Banking request with no ${RequestHeader.`PSD2-CERT`} header") + Failure(X509CannotGetCertificate) + } + } else { + BerlinGroupSigning.getCertificateFromTppSignatureCertificate(requestHeaders) + } + } + private def passesPsd2ServiceProviderCommon(cc: Option[CallContext], serviceProvider: String) = { val result = getPropsValue("requirePsd2Certificates", "NONE") match { case value if value.toUpperCase == "ONLINE" => val requestHeaders = cc.map(_.requestHeaders).getOrElse(Nil) val consumerName = cc.flatMap(_.consumer.map(_.name.get)).getOrElse("") - val certificate = getCertificateFromTppSignatureCertificate(requestHeaders) - for { - tpps <- BerlinGroupSigning.getRegulatedEntityByCertificate(certificate, cc) - } yield { - tpps match { - case Nil => - ObpApiFailure(RegulatedEntityNotFoundByCertificate, 401, cc) - case single :: Nil => - logger.debug(s"Regulated entity by certificate: $single") - // Only one match, proceed to role check - if (single.services.contains(serviceProvider)) { - logger.debug(s"Regulated entity by certificate (single.services: ${single.services}, serviceProvider: $serviceProvider): ") - Full(true) - } else { - ObpApiFailure(X509ActionIsNotAllowed, 403, cc) + tppCertificateForStandard(cc) match { + // No usable certificate: fail closed. passesPsd2ServiceProvider maps a Failure to a 401 + // -- this used to throw out of the base64 decode and become a 500. + case failure: Failure => + logger.debug(s"passesPsd2ServiceProvider: no usable TPP certificate: $failure") + Future(failure) + case Empty => + logger.debug("passesPsd2ServiceProvider: no TPP certificate header") + Future(Failure(X509CannotGetCertificate)) + case Full(certificate) => + for { + tpps <- BerlinGroupSigning.getRegulatedEntityByCertificate(certificate, cc) + } yield { + tpps match { + case Nil => + ObpApiFailure(RegulatedEntityNotFoundByCertificate, 401, cc) + case single :: Nil => + logger.debug(s"Regulated entity by certificate: $single") + // Only one match, proceed to role check + if (single.services.contains(serviceProvider)) { + logger.debug(s"Regulated entity by certificate (single.services: ${single.services}, serviceProvider: $serviceProvider): ") + Full(true) + } else { + ObpApiFailure(X509ActionIsNotAllowed, 403, cc) + } + case multiple => + // Ambiguity detected: more than one TPP matches the certificate + val names = multiple.map(e => s"'${e.entityName}' (Code: ${e.entityCode})").mkString(", ") + ObpApiFailure(s"$RegulatedEntityAmbiguityByCertificate: multiple TPPs found: $names", 401, cc) } - case multiple => - // Ambiguity detected: more than one TPP matches the certificate - val names = multiple.map(e => s"'${e.entityName}' (Code: ${e.entityCode})").mkString(", ") - ObpApiFailure(s"$RegulatedEntityAmbiguityByCertificate: multiple TPPs found: $names", 401, cc) - } + } } case value if value.toUpperCase == "CERTIFICATE" => Future { `getPSD2-CERT`(cc.map(_.requestHeaders).getOrElse(Nil)) match { diff --git a/obp-api/src/main/scala/code/api/util/BerlinGroupCheck.scala b/obp-api/src/main/scala/code/api/util/BerlinGroupCheck.scala index 98026bf4ff..95706d8f3a 100644 --- a/obp-api/src/main/scala/code/api/util/BerlinGroupCheck.scala +++ b/obp-api/src/main/scala/code/api/util/BerlinGroupCheck.scala @@ -141,16 +141,19 @@ object BerlinGroupCheck extends MdcLoggable { logger.debug(s" Algorithm: ${parsed.algorithm}") logger.debug(s" Signature: ${parsed.signature}") - val certificate = getCertificateFromTppSignatureCertificate(reqHeaders) - val certSerialNumber = certificate.getSerialNumber + // A Signature header without a usable TPP-Signature-Certificate cannot have its keyId.SN + // checked against anything, so it is an invalid signature header (400) — not a 500. + val certSerialNumber = getCertificateFromTppSignatureCertificate(reqHeaders).map(_.getSerialNumber) - logger.debug(s"Certificate serial number (decimal): ${certSerialNumber.toString}") - logger.debug(s"Certificate serial number (hex): ${certSerialNumber.toString(16).toUpperCase}") + logger.debug(s"Certificate serial number (decimal): ${certSerialNumber.map(_.toString)}") + logger.debug(s"Certificate serial number (hex): ${certSerialNumber.map(sn => sn.toString(16).toUpperCase)}") - val snMatches = BerlinGroupSignatureHeaderParser.doesSerialNumberMatch(parsed.keyId.sn, certSerialNumber) + val snMatches = certSerialNumber + .map(BerlinGroupSignatureHeaderParser.doesSerialNumberMatch(parsed.keyId.sn, _)) + .getOrElse(false) if (!snMatches) { - logger.debug(s"Serial number mismatch. Parsed SN: ${parsed.keyId.sn}, Certificate decimal: ${certSerialNumber.toString}, Certificate hex: ${certSerialNumber.toString(16).toUpperCase}") + logger.debug(s"Serial number mismatch (or no usable certificate). Parsed SN: ${parsed.keyId.sn}, Certificate decimal: ${certSerialNumber.map(_.toString)}, Certificate hex: ${certSerialNumber.map(sn => sn.toString(16).toUpperCase)}") Some( ( fullBoxOrException( diff --git a/obp-api/src/main/scala/code/api/util/BerlinGroupSigning.scala b/obp-api/src/main/scala/code/api/util/BerlinGroupSigning.scala index 0ea72011a7..f183de602e 100644 --- a/obp-api/src/main/scala/code/api/util/BerlinGroupSigning.scala +++ b/obp-api/src/main/scala/code/api/util/BerlinGroupSigning.scala @@ -168,40 +168,46 @@ object BerlinGroupSigning extends MdcLoggable { forwardResult case true => val requestHeaders = forwardResult._2.map(_.requestHeaders).getOrElse(Nil) - val certificate = getCertificateFromTppSignatureCertificate(requestHeaders) - X509.validateCertificate(certificate) match { - case Full(true) => // PEM certificate is ok - val generatedDigest = generateDigest(body.getOrElse("")) - val requestHeaderDigest = getHeaderValue(RequestHeader.Digest, requestHeaders) - if(generatedDigest == requestHeaderDigest) { // Verifying the Hash in the Digest Field - val signatureHeaderValue = getHeaderValue(RequestHeader.Signature, requestHeaders) - val signature = parseSignatureHeader(signatureHeaderValue).getOrElse("signature", "NONE") - val headersToSign = parseSignatureHeader(signatureHeaderValue).getOrElse("headers", "").split(" ").toList - val sn = parseSignatureHeader(signatureHeaderValue).getOrElse("keyId", "").split(" ").toList - val headers = headersToSign.map(h => - if (h.toLowerCase() == RequestHeader.Digest.toLowerCase()) { - s"$h: $generatedDigest" - } else { - s"$h: ${getHeaderValue(h, requestHeaders)}" + // checkRequestIsSigned only proves the header is present, not that it holds a usable + // certificate — an unparseable one is a 401, not a 500. + getCertificateFromTppSignatureCertificate(requestHeaders) match { + case Full(certificate) => + X509.validateCertificate(certificate) match { + case Full(true) => // PEM certificate is ok + val generatedDigest = generateDigest(body.getOrElse("")) + val requestHeaderDigest = getHeaderValue(RequestHeader.Digest, requestHeaders) + if(generatedDigest == requestHeaderDigest) { // Verifying the Hash in the Digest Field + val signatureHeaderValue = getHeaderValue(RequestHeader.Signature, requestHeaders) + val signature = parseSignatureHeader(signatureHeaderValue).getOrElse("signature", "NONE") + val headersToSign = parseSignatureHeader(signatureHeaderValue).getOrElse("headers", "").split(" ").toList + val sn = parseSignatureHeader(signatureHeaderValue).getOrElse("keyId", "").split(" ").toList + val headers = headersToSign.map(h => + if (h.toLowerCase() == RequestHeader.Digest.toLowerCase()) { + s"$h: $generatedDigest" + } else { + s"$h: ${getHeaderValue(h, requestHeaders)}" + } + ) + val signingString = headers.mkString("\n") + val isVerified = verifySignature(signingString, signature, certificate.getPublicKey) + val isValidated = CertificateVerifier.validateCertificate(certificate) + val bypassValidation = APIUtil.getPropsAsBoolValue("bypass_tpp_signature_validation", defaultValue = false) + (isVerified, isValidated) match { + case (true, true) => forwardResult + case (true, false) if bypassValidation => forwardResult + case (true, false) => apiFailure(ErrorMessages.X509PublicKeyCannotBeValidated, 401)(forwardResult) + case (false, _) => apiFailure(ErrorMessages.X509PublicKeyCannotVerify, 401)(forwardResult) + } + } else { // The two DIGEST hashes do NOT match, the integrity of the request body is NOT confirmed. + logger.debug(s"Generated digest: $generatedDigest") + logger.debug(s"Request header digest: $requestHeaderDigest") + apiFailure(ErrorMessages.X509PublicKeyCannotVerify, 401)(forwardResult) } - ) - val signingString = headers.mkString("\n") - val isVerified = verifySignature(signingString, signature, certificate.getPublicKey) - val isValidated = CertificateVerifier.validateCertificate(certificate) - val bypassValidation = APIUtil.getPropsAsBoolValue("bypass_tpp_signature_validation", defaultValue = false) - (isVerified, isValidated) match { - case (true, true) => forwardResult - case (true, false) if bypassValidation => forwardResult - case (true, false) => apiFailure(ErrorMessages.X509PublicKeyCannotBeValidated, 401)(forwardResult) - case (false, _) => apiFailure(ErrorMessages.X509PublicKeyCannotVerify, 401)(forwardResult) - } - } else { // The two DIGEST hashes do NOT match, the integrity of the request body is NOT confirmed. - logger.debug(s"Generated digest: $generatedDigest") - logger.debug(s"Request header digest: $requestHeaderDigest") - apiFailure(ErrorMessages.X509PublicKeyCannotVerify, 401)(forwardResult) + case Failure(msg, t, c) => (Failure(msg, t, c), forwardResult._2) // PEM certificate is not valid + case _ => apiFailure(ErrorMessages.X509GeneralError, 401)(forwardResult) // PEM certificate cannot be validated } - case Failure(msg, t, c) => (Failure(msg, t, c), forwardResult._2) // PEM certificate is not valid - case _ => apiFailure(ErrorMessages.X509GeneralError, 401)(forwardResult) // PEM certificate cannot be validated + case Failure(msg, t, c) => (Failure(msg, t, c), forwardResult._2) // certificate header unusable + case _ => apiFailure(ErrorMessages.X509CannotGetCertificate, 401)(forwardResult) } } } @@ -210,15 +216,40 @@ object BerlinGroupSigning extends MdcLoggable { requestHeaders.find(_.name.toLowerCase() == name.toLowerCase()).map(_.values.mkString) .getOrElse(SecureRandomUtil.csprng.nextLong().toString) } - def getCertificateFromTppSignatureCertificate(requestHeaders: List[HTTPParam]): X509Certificate = { - val certificate = getHeaderValue(RequestHeader.`TPP-Signature-Certificate`, requestHeaders) - // Decode the Base64 string - val decodedBytes = Base64.getDecoder.decode(certificate) - // Convert the bytes to a string (it could be PEM format for public key) - val decodedString = new String(decodedBytes, StandardCharsets.UTF_8) - - val certificatePemString = getCertificatePem(decodedString) - parseCertificate(certificatePemString) + /** + * The `TPP-Signature-Certificate` header parsed into an X509 certificate, or a Failure saying why + * it could not be. + * + * Returns a Box rather than throwing. Every step here consumes caller-supplied input and can fail + * on data we do not control: the header may be absent, not base64, base64 of something that is + * not a PEM, or a PEM that is not a certificate. Callers map the Failure to a 401; an exception + * would surface as a 500 instead. + * + * In particular this must NOT use `getHeaderValue`, which substitutes a random Long for an absent + * header. That is a serviceable "never matches" sentinel for the string comparisons it was + * written for (Digest, Signature), but not for a value about to be Base64-decoded: a negative + * Long renders with a leading '-', and `Base64.getDecoder` rejects it with + * "Illegal base64 character 2d" — so a merely missing header became a 500, and only for the + * ~half of calls where the random Long happened to be negative. + */ + def getCertificateFromTppSignatureCertificate(requestHeaders: List[HTTPParam]): Box[X509Certificate] = { + requestHeaders + .find(_.name.equalsIgnoreCase(RequestHeader.`TPP-Signature-Certificate`)) + .map(_.values.mkString.trim) + .filter(_.nonEmpty) match { + case None => + Failure(ErrorMessages.X509CannotGetCertificate) + case Some(headerValue) => + // Decode the Base64 string, then pull the PEM certificate out of whatever it decoded to. + Helpers.tryo(new String(Base64.getDecoder.decode(headerValue), StandardCharsets.UTF_8)) match { + case Full(decodedString) => + getCertificatePem(decodedString) match { + case "" => Failure(ErrorMessages.X509CannotGetCertificate) + case certificatePemString => Helpers.tryo(parseCertificate(certificatePemString)) ?~ ErrorMessages.X509GeneralError + } + case _ => Failure(ErrorMessages.X509GeneralError) + } + } } private def getCertificatePem(decodedString: String) = { @@ -276,8 +307,15 @@ object BerlinGroupSigning extends MdcLoggable { val tppSignatureCert: String = APIUtil.getRequestHeader(RequestHeader.`TPP-Signature-Certificate`, requestHeaders) if (tppSignatureCert.isEmpty) { Future(forwardResult) - } else { // Dynamic consumer creation/update works in case that RequestHeader.`TPP-Signature-Certificate` is present - val certificate = getCertificateFromTppSignatureCertificate(requestHeaders) + } else getCertificateFromTppSignatureCertificate(requestHeaders) match { + // Header present but unusable: leave the caller's result untouched rather than throwing. + // Consumer creation is a side effect of a signed request, not the thing being authorised — + // verifySignedRequest is what rejects a bad certificate. + case unusable if unusable.isEmpty => + logger.debug(s"getOrCreateConsumer: unusable TPP-Signature-Certificate: $unusable") + Future(forwardResult) + // Dynamic consumer creation/update, when TPP-Signature-Certificate holds a real certificate + case Full(certificate) => val extractedEmail = emailPattern.findFirstMatchIn(certificate.getSubjectDN.getName).map(_.group(1)) val extractOrganisation = organisationlPattern.findFirstMatchIn(certificate.getSubjectDN.getName).map(_.group(1)) diff --git a/obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsTest.scala b/obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsTest.scala index 781346e2de..f38ac157f5 100644 --- a/obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsTest.scala +++ b/obp-api/src/test/scala/bootstrap/http4s/Http4sMtlsTest.scala @@ -96,4 +96,27 @@ class Http4sMtlsTest extends FlatSpec with Matchers { sslContext should not be null sslContext.getProtocol shouldEqual "TLS" } + + // A TPP's certificates arrive as often in PKCS12 as in JKS, and the store type cannot be + // sniffed from the bytes cheaply — getting it wrong fails at load with an opaque parse error. + "keyStoreTypeOf" should "select PKCS12 for .p12/.pfx and JKS otherwise" in { + Http4sMtls.keyStoreTypeOf("/path/server.p12") shouldEqual "PKCS12" + Http4sMtls.keyStoreTypeOf("/path/server.PFX") shouldEqual "PKCS12" + Http4sMtls.keyStoreTypeOf("/path/server.jks") shouldEqual "JKS" + Http4sMtls.keyStoreTypeOf("/path/server") shouldEqual "JKS" + } + + it should "let buildSslContext load a PKCS12 keystore" in { + val certDir = new java.io.File(getClass.getResource("/cert/server.jks").toURI).getParent + // localhost_san_dns_ip.pfx carries SAN DNS:localhost + IP:127.0.0.1, which JKS server.jks + // lacks — clients that do not fall back to CN need this one. + val config = Http4sMtls.MtlsConfig( + keystorePath = s"$certDir/localhost_san_dns_ip.pfx", + keystorePassword = "123456", + truststorePath = s"$certDir/server.trust.jks", + truststorePassword = "123456", + needClientAuth = true + ) + Http4sMtls.buildSslContext(config) should not be null + } } diff --git a/scripts/java_env.sh b/scripts/java_env.sh new file mode 100755 index 0000000000..d99b4e04d0 --- /dev/null +++ b/scripts/java_env.sh @@ -0,0 +1,73 @@ +#!/bin/bash +################################################################################ +# OBP-API JDK selection +# +# The build compiles with `-release 17`. A JDK older than 17 fails the build with +# +# ERROR '17' is not a valid choice for '-release' +# ERROR bad option: '-release' +# +# which is easy to misread as a source error. Distributions where the default +# `java` is still 11 hit this on every build, so the run scripts source this file +# to select a >= 17 JDK before invoking Maven. +# +# An existing JAVA_HOME is honoured whenever it is new enough; it is only replaced +# when it would fail the build, and the replacement is announced. Sets both +# JAVA_HOME and PATH, because the run scripts invoke a bare `java` for the server. +################################################################################ + +# Echoes the feature version (e.g. 17) of the java binary passed in, or nothing. +java_major_of() { + local java_bin="$1" + [ -x "$java_bin" ] || return 0 + "$java_bin" -version 2>&1 \ + | head -1 \ + | sed -nE 's/.*version "([0-9]+)(\.[0-9]+)*.*/\1/p' +} + +java_env_required=17 +java_env_selected="" + +# 1. An adequate JAVA_HOME already in the environment wins — do not second-guess it. +if [ -n "$JAVA_HOME" ]; then + java_env_current="$(java_major_of "$JAVA_HOME/bin/java")" + if [ -n "$java_env_current" ] && [ "$java_env_current" -ge "$java_env_required" ] 2>/dev/null; then + java_env_selected="$JAVA_HOME" + else + echo ">>> JAVA_HOME points at Java ${java_env_current:-unknown}, but the build needs >= $java_env_required — looking for another JDK" + fi +fi + +# 2. Otherwise search the usual locations (Linux distro paths, then macOS). +if [ -z "$java_env_selected" ]; then + for java_env_candidate in \ + /usr/lib/jvm/java-17-openjdk-amd64 \ + /usr/lib/jvm/java-17-openjdk \ + /usr/lib/jvm/java-21-openjdk-amd64 \ + /usr/lib/jvm/java-21-openjdk \ + "$(/usr/libexec/java_home -v 17 2>/dev/null)" \ + "$(/usr/libexec/java_home -v 21 2>/dev/null)"; do + [ -n "$java_env_candidate" ] || continue + java_env_candidate_major="$(java_major_of "$java_env_candidate/bin/java")" + if [ -n "$java_env_candidate_major" ] && [ "$java_env_candidate_major" -ge "$java_env_required" ] 2>/dev/null; then + java_env_selected="$java_env_candidate" + break + fi + done +fi + +# 3. Fall back to whatever `java` is on PATH, warning if it cannot build. +if [ -n "$java_env_selected" ]; then + export JAVA_HOME="$java_env_selected" + export PATH="$JAVA_HOME/bin:$PATH" + echo ">>> Using JDK $(java_major_of "$JAVA_HOME/bin/java") at $JAVA_HOME" +else + java_env_path_major="$(java_major_of "$(command -v java 2>/dev/null)")" + echo ">>> WARNING: no JDK >= $java_env_required found (java on PATH is ${java_env_path_major:-missing})." + echo " The build will fail with \"'17' is not a valid choice for '-release'\"." + echo " Install a JDK $java_env_required or set JAVA_HOME to one." +fi + +# Sourced under `set -e`: end on a command that always succeeds so a false test +# in the block above can never take down the caller. +true diff --git a/scripts/mtls_env.sh b/scripts/mtls_env.sh new file mode 100755 index 0000000000..aff7ed8ef2 --- /dev/null +++ b/scripts/mtls_env.sh @@ -0,0 +1,89 @@ +#!/bin/bash +################################################################################ +# OBP-API dev-only mTLS environment +# +# Exports the OBP_* overrides that switch the http4s server into in-process mTLS +# termination (see obp-api/src/main/scala/bootstrap/http4s/Http4sMtls.scala and +# docs/MTLS_DEV_MODE.md). Sourced by the --mtls flag of the build_and_run +# scripts; can also be sourced by hand for a jar that is already built: +# +# . scripts/mtls_env.sh && java -jar obp-api/target/obp-api.jar +# +# Every OBP prop is overridable from the environment as OBP_ with dots +# replaced by underscores (APIUtil.getPropsValue reads the environment ahead of +# the props file), which is what makes this a no-edit toggle. +# +# All values below are defaults only: anything already exported wins, so +# OBP_MTLS_CLIENT_AUTH=want ./flushall_fast_build_and_run.sh --mtls +# behaves as expected. +################################################################################ + +MTLS_REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +# Mirror Lift's own props resolution, in its order. In Development, Lift's `modeName` +# is the EMPTY string, so its candidate list collapses to the user/host-specific files +# and then default.props. A file named `development.default.props` is therefore never +# loaded in Development — despite the name, it is inert, and reading the hostname from +# it silently pins local_identity_provider to a value the running app does not use. +# (`test.default.props` does work, because in Test mode modeName is "test".) +mtls_props_dir="$MTLS_REPO_ROOT/obp-api/src/main/resources/props" +mtls_user="$(id -un 2>/dev/null)" +mtls_host="$(hostname 2>/dev/null)" +mtls_props_file="" +for mtls_candidate in \ + "$mtls_props_dir/$mtls_user.$mtls_host.props" \ + "$mtls_props_dir/$mtls_user.props" \ + "$mtls_props_dir/$mtls_host.props" \ + "$mtls_props_dir/default.props"; do + if [ -f "$mtls_candidate" ]; then mtls_props_file="$mtls_candidate"; break; fi +done + +# Echoes the value of a prop from that file, or nothing when it is absent/commented out. +mtls_prop() { + [ -n "$mtls_props_file" ] || return 0 + grep -E "^[[:space:]]*$1[[:space:]]*=" "$mtls_props_file" \ + | tail -1 | cut -d= -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' +} + +# --- the toggle itself ------------------------------------------------------- +# Http4sMtls defaults the four store props to this same checked-in dev pair, so +# strictly only OBP_MTLS_ENABLED is required. They are set explicitly here as +# absolute paths so that the server also starts correctly from another working +# directory, where the repo-relative defaults would not resolve. +export OBP_MTLS_ENABLED=true +: "${OBP_MTLS_KEYSTORE_PATH:=$MTLS_REPO_ROOT/obp-api/src/test/resources/cert/server.jks}" +: "${OBP_MTLS_KEYSTORE_PASSWORD:=123456}" +: "${OBP_MTLS_TRUSTSTORE_PATH:=$MTLS_REPO_ROOT/obp-api/src/test/resources/cert/server.trust.jks}" +: "${OBP_MTLS_TRUSTSTORE_PASSWORD:=123456}" +: "${OBP_MTLS_CLIENT_AUTH:=need}" +export OBP_MTLS_KEYSTORE_PATH OBP_MTLS_KEYSTORE_PASSWORD +export OBP_MTLS_TRUSTSTORE_PATH OBP_MTLS_TRUSTSTORE_PASSWORD OBP_MTLS_CLIENT_AUTH + +# --- hostname, and why local_identity_provider has to be pinned with it ------- +# The listener now speaks TLS, so generated links must say https://. But hostname +# is also the default for local_identity_provider (code/api/constant/constant.scala), +# which is the `provider` column every AuthUser / ResourceUser row is keyed on. +# Flipping the scheme alone would give the same local users a different provider +# string on every toggle, orphaning them. So pin local_identity_provider to the +# pre-toggle hostname and let only the scheme move. Skipped when the props file +# already sets local_identity_provider explicitly — then there is nothing to pin. +mtls_props_hostname="$(mtls_prop hostname)" +if [ -n "$mtls_props_hostname" ]; then + if [ -z "$(mtls_prop local_identity_provider)" ]; then + : "${OBP_LOCAL_IDENTITY_PROVIDER:=$mtls_props_hostname}" + export OBP_LOCAL_IDENTITY_PROVIDER + fi + : "${OBP_HOSTNAME:=$(echo "$mtls_props_hostname" | sed 's|^http://|https://|')}" + export OBP_HOSTNAME +fi + +echo ">>> mTLS enabled (dev-only in-process TLS termination)" +echo " keystore : $OBP_MTLS_KEYSTORE_PATH" +echo " truststore : $OBP_MTLS_TRUSTSTORE_PATH" +echo " client_auth: $OBP_MTLS_CLIENT_AUTH" +echo " hostname : ${OBP_HOSTNAME:-}" +# Plain `if`, not `[ … ] && echo`: this is the last command of a sourced file, and the callers +# run under `set -e` — a false test would take its exit status and abort the caller. +if [ -n "$OBP_LOCAL_IDENTITY_PROVIDER" ]; then + echo " local_identity_provider pinned to: $OBP_LOCAL_IDENTITY_PROVIDER" +fi