diff --git a/docs/_docs/monitoring-metrics/new-metrics.adoc b/docs/_docs/monitoring-metrics/new-metrics.adoc index 249bbc8f120f1..d80ef2f16cac5 100644 --- a/docs/_docs/monitoring-metrics/new-metrics.adoc +++ b/docs/_docs/monitoring-metrics/new-metrics.adoc @@ -432,6 +432,20 @@ Register name: `communication.tcp.{node_consistent_id}` |sentMessagesToNode | long | Total number of messages sent by current node to the given node. |=== +== Certificate Renewal + +Register name: `ssl` + +[cols="2,1,3",opts="header"] +|=== +|Name | Type | Description +|ConsecutiveRenewalFailures | long | Sum of consecutive automatic renewal failures across the node's SSL factories. Zero when renewal is disabled. +|ManualReloadAttempts | long | Factory reload attempts started by the node for manual commands, including errors. Excludes startup loads. +|AutomaticRenewalAttempts | long | Automatic factory renewal calls, including errors and calls that find renewal is no longer due. +|=== + +See link:security/ssl-tls#automatic-certificate-renewal[Automatic Certificate Renewal] for configuration and recovery. + == Communication Connection Pool Register name: `communication.tcp.connectionPool` diff --git a/docs/_docs/security/ssl-tls.adoc b/docs/_docs/security/ssl-tls.adoc index 9ede647e3ed99..8451d4ad36beb 100644 --- a/docs/_docs/security/ssl-tls.adoc +++ b/docs/_docs/security/ssl-tls.adoc @@ -176,11 +176,18 @@ The command asks for confirmation; use `--yes` to skip it. . Check the command result and the reload logs of any failed nodes, then verify a new connection with the replacement certificates. Established connections continue using their existing TLS context. +`SslContextFactory` checks that every certificate in each private-key chain is valid whenever it loads a key store: at node startup, on reload and during automatic renewal. +A node with expired or not yet valid certificates does not start, and a reload of such certificates keeps the current ones. +Client applications and `control.sh` that use `SslContextFactory` apply the same check to their own key stores. +Replacement certificates may expire earlier than the current ones or have the same expiry date. +You can also update the trust store without replacing the node's certificate. + If a factory fails to reload, it keeps its previous context, and that node stops reloading its remaining factories. Successful reloads remain in effect; other nodes continue independently. Check the affected node's log for error details. Correct the failed store and run the command again. -To restore the previous certificates, restore the backed-up files and reload again while peers still trust those certificates. +To restore the previous certificates, restore the backed-up files and reload again. +The restored certificates must still be valid and trusted by peers. Custom factories must extend javadoc:org.apache.ignite.ssl.AbstractSslContextFactory[]. They must reread key and trust data from their sources on reload and supply the updated context for new connections. @@ -188,6 +195,94 @@ If any factory on a node has another type, the command reports it as unsupported For these factories, follow the same trust-first sequence and restart affected nodes one at a time instead of running reload. If you configured `trustManagers` directly, the command reuses them instead of reloading a trust store. +== Automatic Certificate Renewal + +Automatic renewal lets nodes load replacement certificates without a reload command. +You must supply the replacements: Ignite reads them from the configured source and does not issue certificates. +Renewal covers the same TCP transports as <>. +It does not update the HTTP REST endpoint or external client applications. + +=== Enable Renewal + +Enable renewal on each configured `SslContextFactory` before starting the node: + +[source,java] +---- +factory.setRenewalEnabled(true); +---- + +Changing this setting requires a node restart. +The key store must contain a private key, and every private key must have an X.509 certificate chain. +A chain contains the key's certificate and the certificates of its issuers. +With renewal enabled, a store that does not meet these requirements prevents node startup. + +Before replacing key stores, prepare peers' trust stores as described in <>. +Once every affected peer trusts the replacement certificates, put the replacement key stores in the configured source. +Keep the configured paths, store types and passwords unchanged. +Ignite loads the replacements automatically when renewal is due; no reload command is needed for this step. + +=== When Renewal Runs + +A chain expires when any certificate in it expires, including an issuing certificate. +Ignite selects the chain that expires first and starts renewal attempts with 15% of its lifetime remaining. +If this point has already passed when the node starts, Ignite schedules an attempt immediately. +See link:{javadoc_base_url}/org/apache/ignite/ssl/SslContextFactory.html#getRenewalTime()[getRenewalTime(), window=_blank] for the calculation. + +Automatic renewal applies the same certificate checks as <>. +If the loaded certificates are still within their renewal window, Ignite records a failed renewal attempt even though it installed the valid material. +This can happen when the certificates have not changed, only the trust store changed, or the replacements are already within their renewal window. +The loaded context remains in use while Ignite retries with the intervals below. + +After a manual reload, Ignite schedules the next automatic attempt from the loaded certificates. +If their renewal window has already started, the attempt runs immediately. + +New connections use the accepted certificates; established connections remain open. +To confirm a replacement, open a new TLS connection to each affected endpoint and compare its peer certificate with the replacement. + +=== Check Failures and Recover + +If loading or validation fails, Ignite keeps the current certificates and retries, even after they expire. +Keeping expired certificates does not extend their validity, so new connections may fail until a valid replacement is available. +Retry intervals grow from 1–1.5 minutes to 60–90 minutes. +The base delay doubles after each failure, with a random addition of up to 50%. + +Check the node log for the next attempt, affected transports and the reason for a failure. +The `ssl.ConsecutiveRenewalFailures` metric sums consecutive failed automatic attempts across the node's factories. +Each factory's count resets after a successful manual reload or an automatic attempt after which renewal is no longer due. + +`ssl.ManualReloadAttempts` and `ssl.AutomaticRenewalAttempts` count the node's calls to reload each factory. +They include errors and automatic attempts that find renewal is no longer due. +A factory shared by several transports is counted once per attempt. +Startup loading and direct application calls to the factory are not counted. + +To recover without waiting for the next retry: + +. Correct the replacement stores and run link:tools/control-script#reloading-tls-certificates[`control.sh --ssl reload`]. +. Check the command result and verify a new connection with the replacement certificates. + +*What a successful command means.* +The result `reloaded` means that the node loaded the stores and accepted their certificate dates. +The command also reports `reloaded` when the files have not changed. +Checking certificate dates does not establish whether peers trust the certificates or whether they have been revoked. +Verify a new TLS connection after a replacement. + +*What happens to automatic renewal.* +If enabled, automatic renewal continues after manual reload, as described in <>. +A failed automatic attempt does not undo the manual reload. + +*If the command cannot connect.* +First check the command's certificates and trust settings. +If the problem is on the node, put a valid replacement in its source and wait for an automatic retry. +To load corrected stores without the command or waiting for a retry, restart the affected nodes. + +=== Custom Certificate Sources + +For automatic renewal, custom sources require a subclass of javadoc:org.apache.ignite.ssl.SslContextFactory[]. +Factories extending only `AbstractSslContextFactory` support manual reload only. +The certificate checks described above belong to `SslContextFactory`. +Factories extending only the base class retain their own loading behavior. +Limit request duration in custom sources: a blocked load can delay other renewals, manual reload and node shutdown. + == SslContextFactory Properties `SslContextFactory` supports the following properties: @@ -216,6 +311,10 @@ If you configured `trustManagers` directly, the command reuses them instead of r |The protocol for secure transport. https://docs.oracle.com/en/java/javase/11/docs/specs/security/standard-names.html#sslcontext-algorithms[Supported algorithms,window=_blank]. |`TLS` +|`renewalEnabled` +|Enables automatic certificate renewal. See <>. +|`false` + |`trustStoreFilePath` |The path to the trust store file. |`N/A` diff --git a/modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java b/modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java index 15306f1d2b916..be1747d4949c3 100644 --- a/modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java +++ b/modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java @@ -137,6 +137,8 @@ private SSLContext createSslContext() throws SSLException { ctx.init(keyMgrs, trustMgrs, null); + acceptKeyManagers(keyMgrs); + return ctx; } catch (NoSuchAlgorithmException e) { @@ -147,6 +149,17 @@ private SSLContext createSslContext() throws SSLException { } } + /** + * Accepts the key managers of a new SSL context before the context replaces the current one. + * Does nothing by default. + * + * @param keyMgrs Key managers created by {@link #createKeyManagers()} for the new context. + * @throws SSLException If the new context must not replace the current one. + */ + void acceptKeyManagers(KeyManager[] keyMgrs) throws SSLException { + // No-op. + } + /** * @param param Value. * @param name Name. diff --git a/modules/commons/src/main/java/org/apache/ignite/ssl/SslContextFactory.java b/modules/commons/src/main/java/org/apache/ignite/ssl/SslContextFactory.java index 5db5c511c7b1e..ba981806d6096 100644 --- a/modules/commons/src/main/java/org/apache/ignite/ssl/SslContextFactory.java +++ b/modules/commons/src/main/java/org/apache/ignite/ssl/SslContextFactory.java @@ -21,23 +21,36 @@ import java.io.FileNotFoundException; import java.io.IOException; import java.io.InputStream; +import java.net.Socket; import java.security.GeneralSecurityException; import java.security.KeyStore; import java.security.NoSuchAlgorithmException; +import java.security.Principal; +import java.security.PrivateKey; +import java.security.cert.Certificate; import java.security.cert.CertificateException; import java.security.cert.X509Certificate; import java.util.Arrays; +import java.util.Date; +import java.util.Enumeration; +import java.util.LinkedHashMap; +import java.util.Map; import javax.net.ssl.KeyManager; import javax.net.ssl.KeyManagerFactory; +import javax.net.ssl.SSLEngine; import javax.net.ssl.SSLException; import javax.net.ssl.TrustManager; import javax.net.ssl.TrustManagerFactory; +import javax.net.ssl.X509ExtendedKeyManager; +import javax.net.ssl.X509KeyManager; import javax.net.ssl.X509TrustManager; +import org.apache.ignite.IgniteException; import org.apache.ignite.internal.systemview.SensitiveProperty; import org.apache.ignite.internal.util.typedef.internal.A; /** * SSL context factory that provides SSL context configuration with specified key and trust stores. + * Every certificate in the private-key chains of the key store must be valid when the factory loads it. *

* In some cases it is useful to disable certificate validation of client side (e.g. when connecting * to a server with self-signed certificate). This can be achieved by setting a disabled trust manager @@ -61,6 +74,9 @@ public class SslContextFactory extends AbstractSslContextFactory { /** Default key manager / trust manager algorithm. Specifying different trust manager algorithm is not supported. */ public static final String DFLT_KEY_ALGORITHM = System.getProperty("ssl.KeyManagerFactory.algorithm", "SunX509"); + /** Fraction of a certificate lifetime reserved for renewal. */ + private static final double RENEWAL_WINDOW_FRACTION = 0.15; + /** Key manager algorithm. */ protected String keyAlgorithm = DFLT_KEY_ALGORITHM; @@ -85,6 +101,64 @@ public class SslContextFactory extends AbstractSslContextFactory { /** Trust managers. */ protected TrustManager[] trustMgrs; + /** */ + private boolean renewalEnabled; + + /** */ + private transient volatile long renewalTime; + + /** + * @return Whether automatic certificate renewal is enabled. Defaults to {@code false}. + */ + public boolean isRenewalEnabled() { + return renewalEnabled; + } + + /** + * @param enabled Whether automatic certificate renewal is enabled. Changes require a node restart. + */ + public void setRenewalEnabled(boolean enabled) { + renewalEnabled = enabled; + } + + /** + * The renewal time is calculated from the certificates loaded with renewal enabled. + * The chain that expires first determines the window, which starts with 15% of its lifetime remaining. + * Its lifetime runs from the key certificate's {@code notBefore} to the earliest {@code notAfter} in its chain. + * For chains with the same expiry, the longest lifetime is used. + * + * @return Start of the renewal window in milliseconds since the epoch, or {@code 0} if the current certificates + * were loaded with renewal disabled or have not been loaded yet. + */ + public long getRenewalTime() { + return renewalTime; + } + + /** + * Replaces the current SSL context when the renewal window has started. + * Does nothing when renewal is disabled, no renewal date has been loaded, or the window has not started. + * A failed replacement leaves the current context and renewal time unchanged. + * + * @throws IgniteException If the replacement cannot be loaded or accepted. + */ + public void renew() { + if (renewalEnabled && renewalTime != 0 && renewalTime <= System.currentTimeMillis()) + init(); + } + + /** {@inheritDoc} */ + @Override void acceptKeyManagers(KeyManager[] keyMgrs) throws SSLException { + Map chains = Arrays.stream(keyMgrs) + .filter(LoadedKeyManager.class::isInstance) + .map(mgr -> ((LoadedKeyManager)mgr).chains) + .findFirst() + .orElse(Map.of()); + + checkValidity(chains); + + renewalTime = renewalEnabled ? renewalTime(chains) : 0; + } + /** * Gets key store type used for context creation. * @@ -266,7 +340,11 @@ public static TrustManager getDisabledTrustManager() { keyMgrFactory.init(keyStore, keyStorePwd); - return keyMgrFactory.getKeyManagers(); + Map chains = keyChains(keyStore); + + return Arrays.stream(keyMgrFactory.getKeyManagers()) + .map(mgr -> mgr instanceof X509KeyManager x509Mgr ? new LoadedKeyManager(x509Mgr, chains) : mgr) + .toArray(KeyManager[]::new); } catch (NoSuchAlgorithmException e) { throw new SSLException("Unsupported key algorithm: " + keyAlgorithm, e); @@ -277,6 +355,98 @@ public static TrustManager getDisabledTrustManager() { } } + /** + * @param store Loaded key store. + * @return X.509 certificate chains of the private keys by alias. + * @throws GeneralSecurityException If a key or certificate cannot be read. + * @throws SSLException If a private key has no X.509 certificate chain. + */ + private Map keyChains(KeyStore store) throws GeneralSecurityException, SSLException { + Map chains = new LinkedHashMap<>(); + + for (Enumeration aliases = store.aliases(); aliases.hasMoreElements(); ) { + String alias = aliases.nextElement(); + + if (!store.isKeyEntry(alias) || !(store.getKey(alias, keyStorePwd) instanceof PrivateKey)) + continue; + + Certificate[] chain = store.getCertificateChain(alias); + + if (chain == null || chain.length == 0) + throw new SSLException("Certificate chain is unavailable [alias=" + alias + "]. Remove the key or add its X.509 chain."); + + X509Certificate[] x509Chain = new X509Certificate[chain.length]; + + for (int i = 0; i < chain.length; i++) { + if (!(chain[i] instanceof X509Certificate cert)) + throw new SSLException("Certificate chain is not X.509 [alias=" + alias + "]. Remove the key or add its X.509 chain."); + + x509Chain[i] = cert; + } + + chains.put(alias, x509Chain); + } + + return chains; + } + + /** + * Checks that every certificate in the key chains is valid now. + * + * @param chains Key chains by alias. + * @throws SSLException If a certificate is not valid or expires before the check completes. + */ + private static void checkValidity(Map chains) throws SSLException { + Date now = new Date(); + long earliestExpiry = Long.MAX_VALUE; + + for (Map.Entry chain : chains.entrySet()) { + for (X509Certificate cert : chain.getValue()) { + try { + cert.checkValidity(now); + } + catch (CertificateException e) { + throw new SSLException("Certificate is not currently valid [alias=" + chain.getKey() + ", subject=" + + cert.getSubjectX500Principal() + ']', e); + } + + earliestExpiry = Math.min(earliestExpiry, cert.getNotAfter().getTime()); + } + } + + if (earliestExpiry < System.currentTimeMillis()) + throw new SSLException("Certificates expired while loading the SSL context [expiry=" + earliestExpiry + ']'); + } + + /** + * @param chains Key chains by alias. + * @return Start of the renewal window of the chain that expires first. + * @throws SSLException If there are no key chains. + */ + private static long renewalTime(Map chains) throws SSLException { + if (chains.isEmpty()) + throw new SSLException("Certificate renewal requires a private key with an X.509 certificate chain."); + + long earliestExpiry = Long.MAX_VALUE; + long selectedStart = Long.MAX_VALUE; + + for (X509Certificate[] chain : chains.values()) { + long chainExpiry = Long.MAX_VALUE; + + for (X509Certificate cert : chain) + chainExpiry = Math.min(chainExpiry, cert.getNotAfter().getTime()); + + long chainStart = chain[0].getNotBefore().getTime(); + + if (chainExpiry < earliestExpiry || chainExpiry == earliestExpiry && chainStart < selectedStart) { + earliestExpiry = chainExpiry; + selectedStart = chainStart; + } + } + + return earliestExpiry - (long)((earliestExpiry - selectedStart) * RENEWAL_WINDOW_FRACTION); + } + /** {@inheritDoc} */ @Override protected final TrustManager[] createTrustManagers() throws SSLException { if (trustMgrs != null) @@ -412,4 +582,63 @@ private static class DisabledX509TrustManager implements X509TrustManager { return CERTS; } } + + /** X.509 key manager that keeps the private-key chains of the key store it was created from. */ + private static class LoadedKeyManager extends X509ExtendedKeyManager { + /** */ + private final X509KeyManager delegate; + + /** */ + private final Map chains; + + /** */ + private LoadedKeyManager(X509KeyManager delegate, Map chains) { + this.delegate = delegate; + this.chains = chains; + } + + /** {@inheritDoc} */ + @Override public String[] getClientAliases(String keyType, Principal[] issuers) { + return delegate.getClientAliases(keyType, issuers); + } + + /** {@inheritDoc} */ + @Override public String chooseClientAlias(String[] keyType, Principal[] issuers, Socket sock) { + return delegate.chooseClientAlias(keyType, issuers, sock); + } + + /** {@inheritDoc} */ + @Override public String[] getServerAliases(String keyType, Principal[] issuers) { + return delegate.getServerAliases(keyType, issuers); + } + + /** {@inheritDoc} */ + @Override public String chooseServerAlias(String keyType, Principal[] issuers, Socket sock) { + return delegate.chooseServerAlias(keyType, issuers, sock); + } + + /** {@inheritDoc} */ + @Override public X509Certificate[] getCertificateChain(String alias) { + return delegate.getCertificateChain(alias); + } + + /** {@inheritDoc} */ + @Override public PrivateKey getPrivateKey(String alias) { + return delegate.getPrivateKey(alias); + } + + /** {@inheritDoc} */ + @Override public String chooseEngineClientAlias(String[] keyType, Principal[] issuers, SSLEngine engine) { + return delegate instanceof X509ExtendedKeyManager extMgr + ? extMgr.chooseEngineClientAlias(keyType, issuers, engine) + : super.chooseEngineClientAlias(keyType, issuers, engine); + } + + /** {@inheritDoc} */ + @Override public String chooseEngineServerAlias(String keyType, Principal[] issuers, SSLEngine engine) { + return delegate instanceof X509ExtendedKeyManager extMgr + ? extMgr.chooseEngineServerAlias(keyType, issuers, engine) + : super.chooseEngineServerAlias(keyType, issuers, engine); + } + } } diff --git a/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerSslReloadTest.java b/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerSslReloadTest.java index 81809a7fc91c5..3019eee97236b 100644 --- a/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerSslReloadTest.java +++ b/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerSslReloadTest.java @@ -36,6 +36,7 @@ import org.apache.ignite.internal.IgniteInternalFuture; import org.apache.ignite.internal.TestRecordingCommunicationSpi; import org.apache.ignite.internal.processors.rest.client.message.GridClientPingPacket; +import org.apache.ignite.spi.metric.LongMetric; import org.apache.ignite.ssl.SslContextFactory; import org.apache.ignite.testframework.ListeningTestLogger; import org.apache.ignite.testframework.LogListener; @@ -88,6 +89,8 @@ public void testSharedFactoryReloadedOnceOnServerAndClient() throws Exception { int[] loads = factories.stream().mapToInt(factory -> factory.loads.get()).toArray(); + factories.forEach(factory -> factory.setProtocols("TLSv1.2")); + assertEquals(EXIT_CODE_OK, executeTlsReload()); for (int i = 0; i < nodes.size(); i++) { @@ -97,6 +100,7 @@ public void testSharedFactoryReloadedOnceOnServerAndClient() throws Exception { : "communication, client connector, binary REST, discovery"; assertContains(log, testOut.toString(), nodes.get(i).localNode().id() + ": " + transports + ": reloaded"); + assertEquals(1, nodes.get(i).context().metric().registry("ssl").findMetric("ManualReloadAttempts").value()); } } diff --git a/modules/core/pom.xml b/modules/core/pom.xml index bbdf058dea191..47fd6cc45de8b 100644 --- a/modules/core/pom.xml +++ b/modules/core/pom.xml @@ -158,6 +158,13 @@ test + + org.bouncycastle + bcpkix-jdk18on + 1.86 + test + + org.eclipse.jetty.ee11 jetty-ee11-servlets diff --git a/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java b/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java index 5d11403186f4e..177c23e680e65 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java @@ -1243,6 +1243,8 @@ public void start( } } + ctx.ssl().start(ctx); + // Start plugins. for (PluginProvider provider : ctx.plugins().allProviders()) provider.onIgniteStart(); @@ -1722,6 +1724,8 @@ private void stop0(boolean cancel) { GridKernalGateway gw = this.gw.get(); if (stopGuard.compareAndSet(false, true)) { + ctx.ssl().stop(); + // Only one thread is allowed to perform stop sequence. boolean firstStop = false; diff --git a/modules/core/src/main/java/org/apache/ignite/internal/management/ssl/SslReloadTask.java b/modules/core/src/main/java/org/apache/ignite/internal/management/ssl/SslReloadTask.java index 1731563527170..2d6446ed6cf2b 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/management/ssl/SslReloadTask.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/management/ssl/SslReloadTask.java @@ -161,7 +161,7 @@ private SslReloadJob(NoArg arg, boolean debug) { .collect(joining(", ")); try { - ((AbstractSslContextFactory)factory).init(); + ignite.context().ssl().reload((AbstractSslContextFactory)factory); } catch (Exception e) { ignite.log().warning("TLS reload: " + nodeId + ": " + transports + ": failed" + initiator, e); diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistry.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistry.java index ae3db72a6640c..8a555a9058905 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistry.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistry.java @@ -17,16 +17,54 @@ package org.apache.ignite.internal.processors.ssl; +import java.util.Collections; +import java.util.Date; +import java.util.IdentityHashMap; import java.util.LinkedHashMap; import java.util.Map; +import java.util.concurrent.Future; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ThreadLocalRandom; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; import javax.cache.configuration.Factory; import javax.net.ssl.SSLContext; +import org.apache.ignite.IgniteCheckedException; +import org.apache.ignite.IgniteException; +import org.apache.ignite.IgniteLogger; +import org.apache.ignite.internal.GridKernalContext; +import org.apache.ignite.internal.IgniteInterruptedCheckedException; +import org.apache.ignite.internal.processors.metric.MetricRegistryImpl; +import org.apache.ignite.internal.thread.context.OperationContext; +import org.apache.ignite.internal.thread.context.Scope; +import org.apache.ignite.internal.util.typedef.internal.U; +import org.apache.ignite.ssl.AbstractSslContextFactory; +import org.apache.ignite.ssl.SslContextFactory; + +import static java.util.concurrent.TimeUnit.MILLISECONDS; +import static java.util.concurrent.TimeUnit.MINUTES; +import static java.util.stream.Collectors.joining; +import static org.apache.ignite.internal.thread.pool.IgniteScheduledThreadPoolExecutor.newSingleThreadScheduledExecutor; /** SSL factories used by running transports. */ public class SslFactoryRegistry { /** */ private final Map> factories = new LinkedHashMap<>(); + /** */ + private volatile Map, Renewal> renewals = Collections.emptyMap(); + + /** */ + private volatile ScheduledExecutorService executor; + + /** */ + private final AtomicLong manualReloadAttempts = new AtomicLong(); + + /** */ + private final AtomicLong automaticRenewalAttempts = new AtomicLong(); + /** */ public synchronized void register(String transport, Factory factory) { if (factories.putIfAbsent(transport, factory) != null) @@ -37,4 +75,205 @@ public synchronized void register(String transport, Factory factory) public synchronized Map> factories() { return new LinkedHashMap<>(factories); } + + /** */ + public synchronized void start(GridKernalContext ctx) { + MetricRegistryImpl metrics = ctx.metric().registry("ssl"); + + metrics.register("ConsecutiveRenewalFailures", () -> renewals.values().stream().mapToLong(r -> r.failures.get()).sum(), + "Sum of consecutive automatic certificate renewal failures across SSL factories."); + metrics.register("ManualReloadAttempts", manualReloadAttempts::get, "Number of manual SSL factory reload attempts."); + metrics.register("AutomaticRenewalAttempts", automaticRenewalAttempts::get, "Number of automatic SSL factory renewal attempts."); + + Map> factories = factories(); + Map, Renewal> renewals = new IdentityHashMap<>(); + IgniteLogger log = ctx.log(getClass()); + + for (Factory factory : factories.values()) { + if (factory instanceof SslContextFactory sslFactory && sslFactory.isRenewalEnabled() && !renewals.containsKey(factory)) { + if (sslFactory.getRenewalTime() == 0) + sslFactory.init(); + + String transports = factories.entrySet().stream() + .filter(entry -> entry.getValue() == factory) + .map(Map.Entry::getKey) + .collect(joining(", ")); + + renewals.put(factory, new Renewal(sslFactory, transports, log)); + } + } + + if (renewals.isEmpty() || ctx.isStopping()) + return; + + executor = newSingleThreadScheduledExecutor("ssl-renewal", ctx.igniteInstanceName()); + + try (Scope ignored = OperationContext.restoreSnapshot(null)) { + executor.execute(() -> renewals.values().forEach(Renewal::scheduleRenewal)); + } + + this.renewals = renewals; + } + + /** */ + public synchronized void stop() { + ScheduledExecutorService executor = this.executor; + + if (executor == null) + return; + + for (Runnable task : executor.shutdownNow()) { + if (task instanceof Future fut) + fut.cancel(false); + } + + boolean interrupted = false; + + while (!executor.isTerminated()) { + try { + executor.awaitTermination(Long.MAX_VALUE, MILLISECONDS); + } + catch (InterruptedException ignored) { + interrupted = true; + } + } + + if (interrupted) + Thread.currentThread().interrupt(); + } + + /** + * Reloads the factory and resets its automatic renewal schedule and failure count. + * + * @param factory Factory to reload. + * @throws IgniteCheckedException If renewal has not started, or an asynchronous reload fails or is interrupted. + */ + public void reload(AbstractSslContextFactory factory) throws IgniteCheckedException { + Future reload; + + synchronized (this) { + Renewal renewal = renewals.get(factory); + + if (renewal == null && factory instanceof SslContextFactory sslFactory && sslFactory.isRenewalEnabled()) + throw new IgniteCheckedException("Automatic TLS renewal has not started."); + + reload = renewal == null ? null : executor.submit(renewal::reload); + } + + if (reload == null) + reloadFactory(factory); + else { + try { + U.get(reload); + } + catch (IgniteInterruptedCheckedException e) { + reload.cancel(true); + + throw e; + } + } + } + + /** */ + private void reloadFactory(AbstractSslContextFactory factory) { + manualReloadAttempts.incrementAndGet(); + + factory.init(); + } + + /** */ + private class Renewal implements Runnable { + /** */ + private final SslContextFactory factory; + + /** */ + private final String transports; + + /** */ + private final IgniteLogger log; + + /** */ + private ScheduledFuture pending; + + /** */ + private final AtomicInteger failures = new AtomicInteger(); + + /** */ + private Renewal(SslContextFactory factory, String transports, IgniteLogger log) { + this.factory = factory; + this.transports = transports; + this.log = log; + } + + /** {@inheritDoc} */ + @Override public void run() { + if (executor.isShutdown()) + return; + + automaticRenewalAttempts.incrementAndGet(); + + try { + factory.renew(); + } + catch (Exception e) { + retry(e); + + return; + } + + long renewalTime = factory.getRenewalTime(); + + if (renewalTime != 0 && renewalTime <= System.currentTimeMillis()) + retry(new IgniteException("Loaded certificates are still due for renewal [renewalTime=" + new Date(renewalTime) + ']')); + else + scheduleRenewal(); + } + + /** */ + private void retry(Exception cause) { + if (executor.isShutdown()) + return; + + long delay = Math.min(MINUTES.toMillis(1) << Math.min(failures.incrementAndGet() - 1, 6), MINUTES.toMillis(60)); + long nextAttempt = System.currentTimeMillis() + delay + ThreadLocalRandom.current().nextLong(delay / 2 + 1); + + log.warning("TLS renewal failed [transports=" + transports + ", nextAttempt=" + new Date(nextAttempt) + ']', cause); + + schedule(nextAttempt); + } + + /** */ + private void reload() { + reloadFactory(factory); + scheduleRenewal(); + } + + /** Ends the failure streak and schedules the next attempt at the renewal time. */ + private void scheduleRenewal() { + failures.set(0); + schedule(factory.getRenewalTime()); + } + + /** */ + private void schedule(long time) { + if (pending != null) + pending.cancel(false); + + if (time == 0) + return; + + try (Scope ignored = OperationContext.restoreSnapshot(null)) { + pending = executor.schedule(this, Math.max(0, time - System.currentTimeMillis()), MILLISECONDS); + } + catch (RejectedExecutionException e) { + if (!executor.isShutdown()) + throw e; + + return; + } + + if (log.isInfoEnabled()) + log.info("TLS renewal scheduled [transports=" + transports + ", nextAttempt=" + new Date(time) + ']'); + } + } } diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistryTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistryTest.java index fe517fe19f12a..760451fc2c400 100644 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistryTest.java +++ b/modules/core/src/test/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistryTest.java @@ -17,23 +17,562 @@ package org.apache.ignite.internal.processors.ssl; +import java.util.Optional; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.CancellationException; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.Future; +import java.util.concurrent.FutureTask; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.atomic.AtomicLong; +import org.apache.ignite.IgniteCheckedException; +import org.apache.ignite.IgniteException; +import org.apache.ignite.IgniteLogger; +import org.apache.ignite.internal.GridKernalContext; +import org.apache.ignite.internal.IgniteInternalFuture; +import org.apache.ignite.internal.IgniteInterruptedCheckedException; +import org.apache.ignite.internal.processors.metric.GridMetricManager; +import org.apache.ignite.internal.processors.metric.MetricRegistryImpl; +import org.apache.ignite.internal.thread.context.OperationContext; +import org.apache.ignite.internal.thread.context.OperationContextAttribute; +import org.apache.ignite.internal.thread.context.Scope; +import org.apache.ignite.internal.thread.context.concurrent.OperationContextAwareScheduledExecutorService; +import org.apache.ignite.internal.thread.pool.IgniteScheduledThreadPoolExecutor; +import org.apache.ignite.logger.NullLogger; +import org.apache.ignite.spi.metric.LongMetric; +import org.apache.ignite.ssl.AbstractSslContextFactory; import org.apache.ignite.ssl.SslContextFactory; +import org.junit.After; +import org.junit.Before; import org.junit.Test; +import org.mockito.ArgumentCaptor; +import org.mockito.MockedStatic; +import static java.util.concurrent.TimeUnit.HOURS; +import static java.util.concurrent.TimeUnit.MILLISECONDS; +import static java.util.concurrent.TimeUnit.MINUTES; +import static java.util.concurrent.TimeUnit.SECONDS; +import static org.apache.ignite.testframework.GridTestUtils.runAsync; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertSame; import static org.junit.Assert.assertThrows; +import static org.junit.Assert.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.ArgumentMatchers.startsWith; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.mockStatic; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; /** */ public class SslFactoryRegistryTest { + /** */ + private final SslFactoryRegistry registry = new SslFactoryRegistry(); + + /** */ + private final SslContextFactory factory = mock(SslContextFactory.class); + + /** */ + private final GridKernalContext ctx = mock(GridKernalContext.class); + + /** */ + private final ScheduledExecutorService executor = mock(ScheduledExecutorService.class); + + /** */ + private final MetricRegistryImpl metrics = new MetricRegistryImpl("ssl", null, null, null, new NullLogger()); + + /** */ + private MockedStatic executorFactory; + + /** */ + private Runnable scheduled; + + /** */ + private ScheduledFuture pending; + + /** */ + private long delay; + + /** */ + @Before + public void setUp() { + GridMetricManager metricMgr = mock(GridMetricManager.class); + + when(ctx.metric()).thenReturn(metricMgr); + when(metricMgr.registry("ssl")).thenReturn(metrics); + when(ctx.log(SslFactoryRegistry.class)).thenReturn(new NullLogger()); + when(factory.isRenewalEnabled()).thenReturn(true); + when(factory.getRenewalTime()).thenReturn(System.currentTimeMillis()); + when(executor.isTerminated()).thenReturn(true); + + doAnswer(call -> { + call.getArgument(0).run(); + + return null; + }).when(executor).execute(any(Runnable.class)); + when(executor.submit(any(Runnable.class))).thenAnswer(call -> { + FutureTask task = new FutureTask<>(call.getArgument(0), null); + + task.run(); + + return task; + }); + + when(executor.schedule(any(Runnable.class), anyLong(), eq(MILLISECONDS))).thenAnswer(call -> { + scheduled = call.getArgument(0); + delay = call.getArgument(1); + + pending = mock(ScheduledFuture.class); + + return pending; + }); + + executorFactory = mockStatic(IgniteScheduledThreadPoolExecutor.class); + executorFactory.when(() -> IgniteScheduledThreadPoolExecutor.newSingleThreadScheduledExecutor("ssl-renewal", null)) + .thenReturn(executor); + } + + /** */ + @After + public void tearDown() throws Exception { + try { + runAsync(registry::stop).get(10_000); + } + finally { + executorFactory.close(); + } + } + /** */ @Test public void testDuplicateRegistrationKeepsOriginalFactory() { - SslFactoryRegistry registry = new SslFactoryRegistry(); - SslContextFactory factory = new SslContextFactory(); - registry.register("communication", factory); assertThrows(IllegalStateException.class, () -> registry.register("communication", new SslContextFactory())); assertSame(factory, registry.factories().get("communication")); } + + /** */ + @Test + public void testDisabledFactoryDoesNotStartRenewal() { + when(factory.isRenewalEnabled()).thenReturn(false); + + registry.register("communication", factory); + registry.start(ctx); + + executorFactory.verifyNoInteractions(); + assertEquals(0, failures()); + assertEquals(0, metric("ManualReloadAttempts")); + assertEquals(0, metric("AutomaticRenewalAttempts")); + } + + /** */ + @Test + public void testEnablingRenewalOnRestartInitializesAndSchedulesSharedFactoryOnce() { + when(factory.getRenewalTime()).thenReturn(0L, System.currentTimeMillis() + HOURS.toMillis(2)); + + registry.register("communication", factory); + registry.register("discovery", factory); + registry.start(ctx); + + verify(factory).init(); + verify(executor).schedule(any(Runnable.class), anyLong(), eq(MILLISECONDS)); + assertTrue(delay > HOURS.toMillis(1)); + assertEquals(0, metric("ManualReloadAttempts")); + assertEquals(0, metric("AutomaticRenewalAttempts")); + } + + /** */ + @Test + public void testEnablingRenewalRejectsInvalidMaterialAtStartup() { + when(factory.getRenewalTime()).thenReturn(0L); + doThrow(new IgniteException("Certificate chain is unavailable")).when(factory).init(); + + registry.register("communication", factory); + + assertThrows(IgniteException.class, () -> registry.start(ctx)); + executorFactory.verifyNoInteractions(); + } + + /** */ + @Test + public void testManualReloadBeforeRenewalStartDoesNotLoadFactory() { + registry.register("communication", factory); + + assertThrows(IgniteCheckedException.class, () -> registry.reload(factory)); + verify(factory, never()).init(); + + registry.start(ctx); + + assertEquals(0, metric("ManualReloadAttempts")); + } + + /** */ + @Test + public void testFailuresIncreaseRetryDelayUntilLimit() { + doThrow(new IgniteException("Source unavailable")).when(factory).renew(); + + registry.register("communication", factory); + registry.start(ctx); + + for (int failure = 1; failure <= 8; failure++) { + long before = System.currentTimeMillis(); + + scheduled.run(); + + long base = MINUTES.toMillis(Math.min(1L << (failure - 1), 60)); + + assertEquals(failure, failures()); + assertEquals(failure, metric("AutomaticRenewalAttempts")); + assertTrue(delay >= base - (System.currentTimeMillis() - before)); + assertTrue(delay <= base + base / 2); + } + } + + /** */ + @Test + public void testNormalReturnResetsFailuresAndReadsNextWindow() { + doThrow(new IgniteException("Source unavailable")).doNothing().when(factory).renew(); + + registry.register("communication", factory); + registry.start(ctx); + scheduled.run(); + + assertEquals(1, failures()); + + when(factory.getRenewalTime()).thenReturn(System.currentTimeMillis() + HOURS.toMillis(2)); + + scheduled.run(); + + assertEquals(0, failures()); + assertTrue(delay > HOURS.toMillis(1)); + assertEquals(2, metric("AutomaticRenewalAttempts")); + } + + /** */ + @Test + public void testRenewalLeavingCertificatesDueIsFailure() { + IgniteLogger log = mock(IgniteLogger.class); + + when(ctx.log(SslFactoryRegistry.class)).thenReturn(log); + + registry.register("communication", factory); + registry.start(ctx); + scheduled.run(); + + verify(factory).renew(); + assertEquals(1, failures()); + verify(log).warning(startsWith("TLS renewal failed [transports=communication, nextAttempt="), any(IgniteException.class)); + } + + /** */ + @Test + public void testManualReloadReplacesRetryWithLoadedRenewalTime() throws Exception { + doThrow(new IgniteException("Source unavailable")).when(factory).renew(); + + registry.register("communication", factory); + registry.start(ctx); + scheduled.run(); + + assertEquals(1, failures()); + + ScheduledFuture retry = pending; + + registry.reload(factory); + + verify(retry).cancel(false); + assertEquals(0, failures()); + assertTrue(delay <= 0); + + when(factory.getRenewalTime()).thenReturn(System.currentTimeMillis() + HOURS.toMillis(2)); + + registry.reload(factory); + + verify(factory, times(2)).init(); + assertTrue(delay > HOURS.toMillis(1)); + } + + /** */ + @Test + public void testManualReloadWithoutRenewalCountsAppliedAndFailedLoads() throws Exception { + AbstractSslContextFactory genericFactory = mock(AbstractSslContextFactory.class); + + when(factory.isRenewalEnabled()).thenReturn(false); + doNothing().doThrow(new IgniteException("Source unavailable")).when(factory).init(); + + registry.register("communication", factory); + registry.register("client connector", genericFactory); + registry.start(ctx); + registry.reload(factory); + registry.reload(genericFactory); + + assertThrows(IgniteException.class, () -> registry.reload(factory)); + verify(genericFactory).init(); + assertEquals(3, metric("ManualReloadAttempts")); + assertEquals(0, metric("AutomaticRenewalAttempts")); + } + + /** */ + @Test + public void testFailedManualReloadKeepsRetryAndFailures() { + doThrow(new IgniteException("Source unavailable")).when(factory).renew(); + + registry.register("communication", factory); + registry.start(ctx); + scheduled.run(); + + ScheduledFuture retry = pending; + IgniteException err = new IgniteException("Invalid replacement"); + + doThrow(err).when(factory).init(); + + assertSame(err, assertThrows(IgniteCheckedException.class, () -> registry.reload(factory)).getCause()); + assertEquals(1, failures()); + assertEquals(1, metric("ManualReloadAttempts")); + verify(retry, never()).cancel(false); + verify(executor, times(2)).schedule(any(Runnable.class), anyLong(), eq(MILLISECONDS)); + } + + /** */ + @Test + public void testStoppingAfterManualLoadDoesNotReportFailure() throws Exception { + registry.register("communication", factory); + registry.start(ctx); + + doAnswer(call -> { + when(executor.isShutdown()).thenReturn(true); + + return null; + }).when(factory).init(); + doThrow(new RejectedExecutionException()).when(executor).schedule(any(Runnable.class), anyLong(), eq(MILLISECONDS)); + + registry.reload(factory); + + verify(factory).init(); + assertEquals(1, metric("ManualReloadAttempts")); + } + + /** */ + @Test + public void testMetricKeepsOtherFactoryFailuresAfterReload() throws Exception { + SslContextFactory otherFactory = mock(SslContextFactory.class); + + when(otherFactory.isRenewalEnabled()).thenReturn(true); + when(otherFactory.getRenewalTime()).thenReturn(1L); + doThrow(new IgniteException("Source unavailable")).when(factory).renew(); + doThrow(new IgniteException("Source unavailable")).when(otherFactory).renew(); + + registry.register("communication", factory); + registry.register("client connector", otherFactory); + registry.start(ctx); + + ArgumentCaptor attempts = ArgumentCaptor.forClass(Runnable.class); + + verify(executor, times(2)).schedule(attempts.capture(), anyLong(), eq(MILLISECONDS)); + attempts.getAllValues().forEach(Runnable::run); + + assertEquals(2, failures()); + + registry.reload(factory); + + assertEquals(1, failures()); + } + + /** */ + @Test + public void testRenewalDoesNotInheritCallerContext() throws Exception { + OperationContextAttribute caller = OperationContextAttribute.newInstance(); + AtomicLong renewalTime = new AtomicLong(System.currentTimeMillis()); + BlockingQueue> attemptCallers = new LinkedBlockingQueue<>(); + + executorFactory.when(() -> IgniteScheduledThreadPoolExecutor.newSingleThreadScheduledExecutor("ssl-renewal", null)) + .thenReturn(new OperationContextAwareScheduledExecutorService(executor) { }); + when(factory.getRenewalTime()).thenAnswer(call -> renewalTime.get()); + doAnswer(call -> { + renewalTime.set(System.currentTimeMillis() + HOURS.toMillis(2)); + attemptCallers.add(Optional.ofNullable(OperationContext.get(caller))); + + return null; + }).when(factory).renew(); + doAnswer(call -> { + assertEquals("operator", OperationContext.get(caller)); + renewalTime.set(System.currentTimeMillis()); + + return null; + }).when(factory).init(); + + registry.register("communication", factory); + + try (Scope ignored = OperationContext.set(caller, "operator")) { + registry.start(ctx); + } + + scheduled.run(); + + assertEquals(Optional.empty(), attemptCallers.remove()); + + try (Scope ignored = OperationContext.set(caller, "operator")) { + registry.reload(factory); + + assertEquals("operator", OperationContext.get(caller)); + } + + scheduled.run(); + + assertEquals(Optional.empty(), attemptCallers.remove()); + } + + /** */ + @Test + public void testStoppingNodeDoesNotStartRenewal() { + when(ctx.isStopping()).thenReturn(true); + + registry.register("communication", factory); + registry.start(ctx); + + executorFactory.verifyNoInteractions(); + } + + /** */ + @Test + public void testStopInterruptsAttemptCancelsQueuedReloadAndWaits() throws Exception { + ScheduledExecutorService renewalExecutor = useRealExecutor(); + CountDownLatch attemptStarted = new CountDownLatch(1); + CountDownLatch attemptInterrupted = new CountDownLatch(1); + CountDownLatch allowCompletion = new CountDownLatch(1); + CountDownLatch terminationWaitEntered = new CountDownLatch(1); + BlockingQueue> submissions = new LinkedBlockingQueue<>(); + + doAnswer(call -> { + attemptStarted.countDown(); + + try { + assertTrue(allowCompletion.await(30, SECONDS)); + } + catch (InterruptedException e) { + attemptInterrupted.countDown(); + + assertTrue(allowCompletion.await(30, SECONDS)); + } + + return null; + }).when(factory).renew(); + doAnswer(call -> { + Future fut = (Future)call.callRealMethod(); + + submissions.add(fut); + + return fut; + }).when(renewalExecutor).submit(any(Runnable.class)); + doAnswer(call -> { + terminationWaitEntered.countDown(); + + return call.callRealMethod(); + }).when(renewalExecutor).awaitTermination(anyLong(), eq(MILLISECONDS)); + + registry.register("communication", factory); + + IgniteInternalFuture reload = null; + IgniteInternalFuture stop = null; + Future queuedReload = null; + + try { + registry.start(ctx); + + assertTrue(attemptStarted.await(5, SECONDS)); + + reload = runAsync(() -> assertThrows(IgniteCheckedException.class, () -> registry.reload(factory))); + queuedReload = submissions.poll(5, SECONDS); + + assertNotNull(queuedReload); + + stop = runAsync(registry::stop); + + assertTrue(attemptInterrupted.await(5, SECONDS)); + assertTrue(terminationWaitEntered.await(5, SECONDS)); + assertTrue(reload.get(5_000).getCause() instanceof CancellationException); + } + finally { + allowCompletion.countDown(); + + if (queuedReload != null) + queuedReload.cancel(true); + + if (stop != null) + stop.get(5_000); + + if (reload != null) + reload.get(5_000); + } + + assertTrue(renewalExecutor.isTerminated()); + verify(factory, never()).init(); + assertEquals(0, metric("ManualReloadAttempts")); + } + + /** */ + @Test + public void testInterruptedCallerCancelsQueuedReload() throws Exception { + Future queuedReload = mock(Future.class); + + when(queuedReload.get()).thenThrow(new InterruptedException()); + doReturn(queuedReload).when(executor).submit(any(Runnable.class)); + + registry.register("communication", factory); + registry.start(ctx); + + assertThrows(IgniteInterruptedCheckedException.class, () -> registry.reload(factory)); + assertTrue(Thread.interrupted()); + verify(queuedReload).cancel(true); + } + + /** */ + @Test + public void testStoppingDuringLoadDoesNotCountFailure() { + doAnswer(call -> { + when(executor.isShutdown()).thenReturn(true); + + throw new IgniteException("Load interrupted by node stop"); + }).when(factory).renew(); + + registry.register("communication", factory); + registry.start(ctx); + scheduled.run(); + + assertEquals(0, failures()); + verify(executor).schedule(any(Runnable.class), anyLong(), eq(MILLISECONDS)); + } + + /** */ + private long failures() { + return metric("ConsecutiveRenewalFailures"); + } + + /** */ + private long metric(String name) { + return metrics.findMetric(name).value(); + } + + /** */ + private ScheduledExecutorService useRealExecutor() { + ScheduledExecutorService renewalExecutor = spy(new IgniteScheduledThreadPoolExecutor("ssl-renewal", null, 1)); + + executorFactory.when(() -> IgniteScheduledThreadPoolExecutor.newSingleThreadScheduledExecutor("ssl-renewal", null)) + .thenReturn(renewalExecutor); + + return renewalExecutor; + } } diff --git a/modules/core/src/test/java/org/apache/ignite/spi/discovery/tcp/TcpDiscoverySslTrustedUntrustedTest.java b/modules/core/src/test/java/org/apache/ignite/spi/discovery/tcp/TcpDiscoverySslTrustedUntrustedTest.java index e44db54804f54..9a78ae696e6d5 100644 --- a/modules/core/src/test/java/org/apache/ignite/spi/discovery/tcp/TcpDiscoverySslTrustedUntrustedTest.java +++ b/modules/core/src/test/java/org/apache/ignite/spi/discovery/tcp/TcpDiscoverySslTrustedUntrustedTest.java @@ -121,20 +121,13 @@ public void testMismatchingCaFirst() throws Exception { checkDiscoveryFailure("node02", "trusttwo", "node01", "trustboth"); } - /** - * @throws Exception If failed. - */ + /** */ @Test - public void testExpired() throws Exception { - checkDiscoveryFailure("node02old", "trusttwo", "node03", "trusttwo"); - } + public void testExpired() { + keyStore = "node02old"; + trustStore = "trusttwo"; - /** - * @throws Exception If failed. - */ - @Test - public void testExpiredMismatchingCa() throws Exception { - checkDiscoveryFailure("node01", "trustboth", "node02old", "trusttwo"); + GridTestUtils.assertThrows(log, () -> startGrid(0), IgniteCheckedException.class, "Certificate is not currently valid"); } /** diff --git a/modules/core/src/test/java/org/apache/ignite/ssl/SslCertificateRenewalTest.java b/modules/core/src/test/java/org/apache/ignite/ssl/SslCertificateRenewalTest.java new file mode 100644 index 0000000000000..240a78b027006 --- /dev/null +++ b/modules/core/src/test/java/org/apache/ignite/ssl/SslCertificateRenewalTest.java @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.ssl; + +import java.nio.file.Files; +import java.nio.file.Path; +import javax.net.ssl.SSLContext; +import org.apache.ignite.Ignition; +import org.apache.ignite.client.ClientRetryNonePolicy; +import org.apache.ignite.client.IgniteClient; +import org.apache.ignite.client.SslMode; +import org.apache.ignite.configuration.ClientConfiguration; +import org.apache.ignite.configuration.ClientConnectorConfiguration; +import org.apache.ignite.internal.IgniteEx; +import org.apache.ignite.internal.util.typedef.internal.U; +import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest; +import org.junit.Test; + +import static java.util.concurrent.TimeUnit.DAYS; +import static java.util.concurrent.TimeUnit.MINUTES; +import static org.apache.ignite.ssl.TestCertificateAuthority.PASSWORD; +import static org.apache.ignite.ssl.TestCertificateAuthority.save; +import static org.apache.ignite.testframework.GridTestUtils.waitForCondition; + +/** */ +public class SslCertificateRenewalTest extends GridCommonAbstractTest { + /** */ + @Test + public void testNodeRenewsCertificatesForNewConnections() throws Exception { + long now = System.currentTimeMillis(); + TestCertificateAuthority ca1 = new TestCertificateAuthority("ca1", now - DAYS.toMillis(1), now + DAYS.toMillis(10)); + TestCertificateAuthority ca2 = new TestCertificateAuthority("ca2", now - DAYS.toMillis(1), now + DAYS.toMillis(10)); + Path dir = Files.createTempDirectory("ssl-renewal"); + Path keys = dir.resolve("keys.jks"); + Path trust = dir.resolve("trust.jks"); + + try { + save(ca1.keyStore("node", now - MINUTES.toMillis(110), now + MINUTES.toMillis(10)), keys); + save(ca1.trustStore(), trust); + + SslContextFactory serverFactory = configure(new SslContextFactory(), keys, trust); + + serverFactory.setRenewalEnabled(true); + + SSLContext initCtx = serverFactory.create(); + + save(ca2.keyStore("node", now - MINUTES.toMillis(1), now + DAYS.toMillis(1)), keys); + save(ca2.trustStore(), trust); + + IgniteEx node = startGrid(0, cfg -> { + cfg.setClientConnectorConfiguration(new ClientConnectorConfiguration() + .setSslEnabled(true) + .setSslClientAuth(true) + .setUseIgniteSslContextFactory(false) + .setSslContextFactory(serverFactory)); + }); + + assertTrue("The renewed context was not installed", waitForCondition(() -> serverFactory.create() != initCtx, 10_000)); + + ClientConfiguration clientCfg = new ClientConfiguration() + .setAddresses("127.0.0.1:" + node.context().clientListener().port()) + .setSslMode(SslMode.REQUIRED) + .setSslContextFactory(configure(new SslContextFactory(), keys, trust)) + .setRetryPolicy(new ClientRetryNonePolicy()) + .setPartitionAwarenessEnabled(false) + .setClusterDiscoveryEnabled(false); + + try (IgniteClient client = Ignition.startClient(clientCfg)) { + client.cacheNames(); + } + } + finally { + stopAllGrids(); + U.delete(dir); + } + } + + /** */ + private static SslContextFactory configure(SslContextFactory factory, Path keys, Path trust) { + factory.setKeyStoreFilePath(keys.toString()); + factory.setKeyStorePassword(PASSWORD); + factory.setTrustStoreFilePath(trust.toString()); + factory.setTrustStorePassword(PASSWORD); + + return factory; + } +} diff --git a/modules/core/src/test/java/org/apache/ignite/ssl/SslContextFactoryRenewalTest.java b/modules/core/src/test/java/org/apache/ignite/ssl/SslContextFactoryRenewalTest.java new file mode 100644 index 0000000000000..f5c803468797b --- /dev/null +++ b/modules/core/src/test/java/org/apache/ignite/ssl/SslContextFactoryRenewalTest.java @@ -0,0 +1,313 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.ssl; + +import java.security.KeyStore; +import java.security.cert.Certificate; +import java.security.cert.X509Certificate; +import java.util.Date; +import javax.crypto.spec.SecretKeySpec; +import javax.net.ssl.SSLContext; +import javax.net.ssl.SSLException; +import javax.net.ssl.TrustManager; +import org.apache.ignite.IgniteException; +import org.junit.Before; +import org.junit.Test; + +import static java.util.concurrent.TimeUnit.DAYS; +import static java.util.concurrent.TimeUnit.HOURS; +import static org.apache.ignite.ssl.TestCertificateAuthority.PASSWORD; +import static org.apache.ignite.ssl.TestCertificateAuthority.merge; +import static org.apache.ignite.testframework.GridTestUtils.waitForCondition; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotSame; +import static org.junit.Assert.assertSame; +import static org.junit.Assert.assertThrows; +import static org.junit.Assert.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.spy; + +/** */ +public class SslContextFactoryRenewalTest { + /** */ + private final long now = System.currentTimeMillis() / 1_000 * 1_000; + + /** */ + private TestCertificateAuthority authority; + + /** */ + @Before + public void setUp() throws Exception { + authority = new TestCertificateAuthority("issuer", now - DAYS.toMillis(365), now + DAYS.toMillis(365)); + } + + /** */ + @Test + public void testRenewalIsDisabledByDefault() { + SslContextFactory factory = new SslContextFactory(); + + assertFalse(factory.isRenewalEnabled()); + assertEquals(0, factory.getRenewalTime()); + + factory.renew(); + } + + /** */ + @Test + public void testDisabledRenewalAcceptsStoreWithoutPrivateKeys() throws Exception { + InMemorySslContextFactory factory = new InMemorySslContextFactory(authority.trustStore()); + + factory.setRenewalEnabled(false); + factory.create(); + factory.init(); + + assertEquals(0, factory.getRenewalTime()); + } + + /** */ + @Test + public void testLoadWithRenewalDisabledClearsRenewalTime() throws Exception { + InMemorySslContextFactory factory = new InMemorySslContextFactory(authority.keyStore("node", at(-85), at(15))); + + factory.create(); + factory.setRenewalEnabled(false); + factory.init(); + factory.setRenewalEnabled(true); + + assertEquals(0, factory.getRenewalTime()); + } + + /** */ + @Test + public void testRenewalWindowStartsAtEarliestChainWindow() throws Exception { + TestCertificateAuthority shortIssuer = new TestCertificateAuthority("short-issuer", now - DAYS.toMillis(365), at(15)); + KeyStore node = authority.keyStore("node", at(-85), at(15)); + KeyStore withSecretKey = KeyStore.getInstance("JCEKS"); + + withSecretKey.load(null, PASSWORD); + withSecretKey.setKeyEntry("node", node.getKey("node", PASSWORD), PASSWORD, node.getCertificateChain("node")); + withSecretKey.setKeyEntry("secret", new SecretKeySpec(new byte[16], "AES"), PASSWORD, null); + + assertRenewalWindowStartsNow(node); + assertRenewalWindowStartsNow(shortIssuer.keyStore("node", at(-85), at(100))); + assertRenewalWindowStartsNow(merge(authority.keyStore("first", at(-85), at(15)), authority.keyStore("second", at(-200), at(100)))); + assertRenewalWindowStartsNow(merge(authority.keyStore("first", at(-200), at(100)), authority.keyStore("second", at(-85), at(15)))); + assertRenewalWindowStartsNow(merge(authority.keyStore("first", at(-5), at(15)), authority.keyStore("second", at(-85), at(15)))); + assertRenewalWindowStartsNow(merge(authority.keyStore("first", at(-85), at(15)), authority.keyStore("second", at(-5), at(15)))); + assertRenewalWindowStartsNow(withSecretKey); + } + + /** */ + @Test + public void testEnabledRenewalRequiresPrivateKey() throws Exception { + InMemorySslContextFactory factory = new InMemorySslContextFactory(authority.trustStore()); + IgniteException err = assertThrows(IgniteException.class, factory::create); + + assertTrue(err.getCause().getMessage().contains("private key with an X.509 certificate chain")); + assertEquals(0, factory.getRenewalTime()); + } + + /** */ + @Test + public void testFirstLoadRejectsExpiredCertificates() throws Exception { + InMemorySslContextFactory factory = new InMemorySslContextFactory(authority.keyStore("node", at(-101), at(-1))); + + factory.setRenewalEnabled(false); + + IgniteException err = assertThrows(IgniteException.class, factory::create); + + assertTrue(err.getCause().getMessage().contains("not currently valid")); + } + + /** */ + @Test + public void testRenewBeforeWindowDoesNotLoad() throws Exception { + InMemorySslContextFactory factory = new InMemorySslContextFactory(authority.keyStore("node", at(-1), at(99))); + + factory.create(); + factory.renew(); + + assertEquals(1, factory.keyStoreLoads); + } + + /** */ + @Test + public void testRenewAcceptsAnyValidReplacement() throws Exception { + for (long expiry : new long[] {at(10), at(5)}) { + InMemorySslContextFactory factory = dueFactory(); + SSLContext ctx = factory.create(); + + factory.keyStore = authority.keyStore("node", at(-1), expiry); + factory.renew(); + + assertNotSame(ctx, factory.create()); + } + + InMemorySslContextFactory factory = new InMemorySslContextFactory(authority.keyStore("node", at(-90), at(10))); + + factory.trustStore = authority.trustStore(); + factory.setTrustManagers((TrustManager[])null); + factory.setTrustStoreFilePath("memory-trust"); + factory.setTrustStorePassword(PASSWORD); + + SSLContext ctx = factory.create(); + TestCertificateAuthority otherIssuer = new TestCertificateAuthority("other-issuer", now - DAYS.toMillis(1), now + DAYS.toMillis(1)); + + factory.trustStore.setCertificateEntry("other-issuer", otherIssuer.trustStore().getCertificate("CN=other-issuer")); + factory.renew(); + + assertNotSame(ctx, factory.create()); + } + + /** */ + @Test + public void testRenewRejectsCertificatesThatAreNotValid() throws Exception { + TestCertificateAuthority expiredIssuer = new TestCertificateAuthority("expired-issuer", at(-100), now - 1_000); + InMemorySslContextFactory factory = dueFactory(); + + factory.keyStore = expiredIssuer.keyStore("replacement", at(-1), at(99)); + assertRenewKeepsCurrentState(factory, "subject=CN=expired-issuer"); + + factory.keyStore = merge(authority.keyStore("first", at(1), at(100)), authority.keyStore("second", at(-1), at(99))); + assertRenewKeepsCurrentState(factory, "alias=first"); + + factory.keyStore = merge(authority.keyStore("first", at(-1), at(99)), authority.keyStore("second", at(1), at(100))); + assertRenewKeepsCurrentState(factory, "alias=second"); + } + + /** */ + @Test + public void testRenewRecoversAfterFailedTrustStoreLoad() throws Exception { + InMemorySslContextFactory factory = dueFactory(); + SSLContext ctx = factory.create(); + + factory.keyStore = authority.keyStore("node", at(-1), at(99)); + factory.setTrustManagers((TrustManager[])null); + factory.setTrustStoreFilePath("missing-trust-store"); + factory.setTrustStorePassword(PASSWORD); + + assertRenewKeepsCurrentState(factory, "missing-trust-store"); + + factory.setTrustManagers(SslContextFactory.getDisabledTrustManager()); + factory.renew(); + + assertNotSame(ctx, factory.create()); + assertEquals(at(84), factory.getRenewalTime()); + } + + /** */ + @Test + public void testRenewRejectsCertificatesExpiredDuringDateValidation() throws Exception { + InMemorySslContextFactory factory = dueFactory(); + long expiry = System.currentTimeMillis() / 1_000 * 1_000 + 5_000; + KeyStore replacement = authority.keyStore("replacement", at(-1), expiry); + Certificate[] chain = replacement.getCertificateChain("replacement"); + X509Certificate cert = spy((X509Certificate)chain[0]); + + doAnswer(validityCheck -> { + validityCheck.callRealMethod(); + + long deadline = System.currentTimeMillis() + 10_000; + + assertTrue(waitForCondition(() -> System.currentTimeMillis() > expiry, () -> System.currentTimeMillis() < deadline)); + + return null; + }).when(cert).checkValidity(any(Date.class)); + + chain[0] = cert; + replacement.setKeyEntry("replacement", replacement.getKey("replacement", PASSWORD), PASSWORD, chain); + factory.keyStore = replacement; + + assertRenewKeepsCurrentState(factory, "expired while loading"); + } + + /** @return Factory that has loaded certificates whose renewal window has started. */ + private InMemorySslContextFactory dueFactory() throws Exception { + InMemorySslContextFactory factory = new InMemorySslContextFactory(authority.keyStore("node", at(-90), at(10))); + + factory.create(); + + return factory; + } + + /** */ + private void assertRenewalWindowStartsNow(KeyStore keys) { + InMemorySslContextFactory factory = new InMemorySslContextFactory(keys); + + factory.create(); + + assertEquals(now, factory.getRenewalTime()); + assertEquals(1, factory.keyStoreLoads); + } + + /** */ + private static void assertRenewKeepsCurrentState(SslContextFactory factory, String reason) { + SSLContext ctx = factory.create(); + long renewalTime = factory.getRenewalTime(); + IgniteException err = assertThrows(IgniteException.class, factory::renew); + + assertTrue(err.getCause().getMessage(), err.getCause().getMessage().contains(reason)); + assertSame(ctx, factory.create()); + assertEquals(renewalTime, factory.getRenewalTime()); + } + + /** */ + private long at(long hours) { + return now + HOURS.toMillis(hours); + } + + /** */ + private static class InMemorySslContextFactory extends SslContextFactory { + /** */ + private static final long serialVersionUID = 0L; + + /** */ + private KeyStore keyStore; + + /** */ + private KeyStore trustStore; + + /** */ + private int keyStoreLoads; + + /** */ + private InMemorySslContextFactory(KeyStore keyStore) { + this.keyStore = keyStore; + + setKeyStoreFilePath("memory"); + setKeyStorePassword(PASSWORD); + setTrustManagers(getDisabledTrustManager()); + setRenewalEnabled(true); + } + + /** {@inheritDoc} */ + @Override protected KeyStore loadKeyStore(String type, String path, char[] pwd) throws SSLException { + if ("memory-trust".equals(path)) + return trustStore; + + if (!"memory".equals(path)) + return super.loadKeyStore(type, path, pwd); + + keyStoreLoads++; + + return keyStore; + } + } +} diff --git a/modules/core/src/test/java/org/apache/ignite/ssl/TestCertificateAuthority.java b/modules/core/src/test/java/org/apache/ignite/ssl/TestCertificateAuthority.java new file mode 100644 index 0000000000000..b7f67fb8d21d5 --- /dev/null +++ b/modules/core/src/test/java/org/apache/ignite/ssl/TestCertificateAuthority.java @@ -0,0 +1,135 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.ssl; + +import java.io.OutputStream; +import java.math.BigInteger; +import java.nio.file.Files; +import java.nio.file.Path; +import java.security.KeyPair; +import java.security.KeyPairGenerator; +import java.security.KeyStore; +import java.security.cert.X509Certificate; +import java.security.spec.ECGenParameterSpec; +import java.util.Collections; +import java.util.Date; +import javax.security.auth.x500.X500Principal; +import org.bouncycastle.asn1.x509.BasicConstraints; +import org.bouncycastle.asn1.x509.Extension; +import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter; +import org.bouncycastle.cert.jcajce.JcaX509v3CertificateBuilder; +import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder; + +/** Test certificates with explicit validity dates. */ +class TestCertificateAuthority { + /** */ + static final char[] PASSWORD = "123456".toCharArray(); + + /** */ + private final X500Principal name; + + /** */ + private final KeyPair keys = keyPair(); + + /** */ + private final X509Certificate cert; + + /** */ + private long serial; + + /** */ + TestCertificateAuthority(String name, long notBefore, long notAfter) throws Exception { + this.name = new X500Principal("CN=" + name); + cert = certificate(this.name, keys, notBefore, notAfter, true); + } + + /** */ + KeyStore keyStore(String alias, long notBefore, long notAfter) throws Exception { + KeyPair leafKeys = keyPair(); + X509Certificate leaf = certificate(new X500Principal("CN=" + alias), leafKeys, notBefore, notAfter, false); + KeyStore store = emptyStore(); + + store.setKeyEntry(alias, leafKeys.getPrivate(), PASSWORD, new X509Certificate[] {leaf, cert}); + + return store; + } + + /** */ + KeyStore trustStore() throws Exception { + KeyStore store = emptyStore(); + + store.setCertificateEntry(name.getName(), cert); + + return store; + } + + /** */ + static KeyStore merge(KeyStore... stores) throws Exception { + KeyStore merged = emptyStore(); + + for (KeyStore store : stores) { + for (String alias : Collections.list(store.aliases())) + merged.setKeyEntry(alias, store.getKey(alias, PASSWORD), PASSWORD, store.getCertificateChain(alias)); + } + + return merged; + } + + /** */ + static void save(KeyStore store, Path path) throws Exception { + try (OutputStream out = Files.newOutputStream(path)) { + store.store(out, PASSWORD); + } + } + + /** */ + private X509Certificate certificate(X500Principal subject, KeyPair subjectKeys, long notBefore, long notAfter, boolean ca) + throws Exception { + JcaX509v3CertificateBuilder builder = new JcaX509v3CertificateBuilder( + name, + BigInteger.valueOf(++serial), + new Date(notBefore), + new Date(notAfter), + subject, + subjectKeys.getPublic()); + + if (ca) + builder.addExtension(Extension.basicConstraints, true, new BasicConstraints(true)); + + return new JcaX509CertificateConverter().getCertificate( + builder.build(new JcaContentSignerBuilder("SHA256withECDSA").build(keys.getPrivate()))); + } + + /** */ + private static KeyPair keyPair() throws Exception { + KeyPairGenerator generator = KeyPairGenerator.getInstance("EC"); + + generator.initialize(new ECGenParameterSpec("secp256r1")); + + return generator.generateKeyPair(); + } + + /** */ + private static KeyStore emptyStore() throws Exception { + KeyStore store = KeyStore.getInstance("JKS"); + + store.load(null, PASSWORD); + + return store; + } +} diff --git a/modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java b/modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java index 7b207d1226a0e..4a1aad4784a21 100644 --- a/modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java +++ b/modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java @@ -83,7 +83,9 @@ import org.apache.ignite.plugin.security.SecurityBasicPermissionSetSerializationTest; import org.apache.ignite.ssl.MultipleSSLContextsTest; import org.apache.ignite.ssl.SSLContextWrapperTest; +import org.apache.ignite.ssl.SslCertificateRenewalTest; import org.apache.ignite.ssl.SslContextFactoryInitTest; +import org.apache.ignite.ssl.SslContextFactoryRenewalTest; import org.apache.ignite.ssl.SslContextReloadTest; import org.apache.ignite.tools.junit.JUnitTeamcityReporter; import org.junit.BeforeClass; @@ -149,6 +151,8 @@ MultipleSSLContextsTest.class, SSLContextWrapperTest.class, SslContextFactoryInitTest.class, + SslContextFactoryRenewalTest.class, + SslCertificateRenewalTest.class, SslFactoryRegistryTest.class, SslReloadTaskTest.class, SslContextReloadTest.class,