Skip to content

IGNITE-29211 Renew TLS certificates automatically before expiry - #13786

Open
anton-vinogradov wants to merge 356 commits into
apache:masterfrom
anton-vinogradov:ignite-29211
Open

anton-vinogradov wants to merge 356 commits into
apache:masterfrom
anton-vinogradov:ignite-29211

Conversation

@anton-vinogradov

@anton-vinogradov anton-vinogradov commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

IGNITE-29211 adds optional automatic TLS certificate renewal. Nodes read replacements from the configured source before expiry. Ignite does not issue certificates.

Behavior

  • Renewal is disabled by default. SslContextFactory calculates renewal dates from the key store actually loaded, including custom sources. Attempts start with 15% of the selected chain's lifetime remaining.
  • Manual and automatic replacements use the same init() path and validate every certificate in the private-key chains. Currently valid certificates are accepted with equal, earlier or later expiry, including identical certificates and trust-store-only changes. The same validity check applies at startup, including in client applications and the control script.
  • Loading or validation failures preserve the current context and renewal date. A successful load installs the context for new connections; established connections remain open. The manual command reports reloaded, including when the material has not changed.
  • Each node runs automatic attempts and manual reloads of enabled factories on one executor. Before the renewal registry starts, manual reloads of enabled factories fail without loading material. If an automatic load leaves renewal due, the attempt counts as a failure and retries with exponential backoff and jitter. The installed context remains in use. A successful manual reload clears that factory's failures and schedules its next attempt, immediately if still due.
  • ssl.ConsecutiveRenewalFailures sums consecutive automatic failures on the node. ssl.ManualReloadAttempts and ssl.AutomaticRenewalAttempts count per-factory calls, including errors and automatic skips. They exclude startup, direct application calls and queued calls cancelled before execution.
  • Shutdown cancels queued operations, interrupts the running executor task and waits for it to finish. A task already waiting for a shared factory may still load and install a valid replacement after shutdown starts.

The existing final void init() contract is preserved. renew() checks whether loading is due; the registry evaluates the resulting renewal date. Startup initializes missing renewal dates. Nodes sharing a factory keep separate schedules and may load it sequentially. A startup load already waiting for the factory may proceed after shutdown starts.

Renewal covers TCP transports. HTTPS remains in IGNITE-29212. Factories extending only AbstractSslContextFactory retain their own manual loading behavior.

Tests cover certificate dates, equal or earlier expiry, trust-only changes, failure recovery, scheduling, shutdown and metrics. They also cover the reload command and replacement on a live TLS connection.

Depends on IGNITE-29210, merged into Apache master.

anton-vinogradov and others added 30 commits October 27, 2026 19:19
# Conflicts:
#	modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java
#	modules/core/src/main/java/org/apache/ignite/spi/communication/tcp/internal/ClusterStateProvider.java
#	modules/core/src/main/java/org/apache/ignite/spi/communication/tcp/internal/GridNioServerWrapper.java
#	modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java
# Conflicts:
#	docs/_docs/security/ssl-tls.adoc
#	modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java
#	modules/core/src/main/java/org/apache/ignite/internal/ssl/SslContextProvider.java
#	modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java
# Conflicts:
#	docs/_docs/security/ssl-tls.adoc
#	modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java
#	modules/core/src/main/java/org/apache/ignite/internal/ssl/SslContextProvider.java
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	modules/core/src/test/java/org/apache/ignite/ssl/SslTestUtils.java
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	modules/core/src/main/java/org/apache/ignite/internal/ssl/SslContextRegistry.java
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Renewal reads the same clock as the provider: System.currentTimeMillis()
instead of the coarse U.currentTimeMillis().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Also drop an unused dated key store helper and check that the report of
testFailingFactoryDoesNotSkipOthers lists the client connector after the
failed factory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the logged reason

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anton-vinogradov and others added 30 commits November 11, 2026 19:56
SslContextFactory no longer declares synchronized methods. It publishes
renewal dates only after the whole load succeeds, so getRenewalTime()
reads a volatile field. renew() builds and installs the context under the
factory monitor and passes the replacement mode as a parameter.

The registry reload no longer takes the registry monitor. The registry
test bounds stop() in tearDown, so a broken shutdown fails the test
instead of hanging it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
renew() no longer takes a cancellation callback. It checks the interrupt
flag after it gets the factory monitor and before it reads the source.
stop() already interrupts the renewal thread with shutdownNow(), so an
attempt that waited for another load of the same factory exits without
reading the source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An automatic attempt fails if the loaded certificates are still due for
renewal. The rule covers unchanged material and replacements that are
themselves in the renewal window. The failure count stays until the
node's certificates leave the window, and retries use the regular backoff.

The factory loads the same way for manual reload and renewal and does
not compare material. The registry and the reload command call init(),
and the command reports "reloaded" after each successful load.

Manual reload and the initial scheduling both use the factory's renewal
date, so the registry does not take its monitor in reload().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
start() writes ctx and log under the registry monitor, and the renewal
thread reads them without it, so both fields are volatile.

Pattern variables replace casts after instanceof checks in the registry
and the factory. This also removes the cast to SslContextFactory that
IDEA reports as overly strong.

testInitialSchedulingRunsOnRenewalThread is removed:
testManualReloadFollowsInitialScheduling also fails when start()
schedules attempts on its own thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the renewal thread reads the node context and the logger, so each
Renewal holds them in final fields. The registry no longer writes
fields under its monitor that the renewal thread reads without it, and
IDEA stops reporting ctx and log as accessed in both contexts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each Renewal counts its consecutive failures, and
ssl.ConsecutiveRenewalFailures sums these counters. The registry no
longer mirrors them in a second counter that a reset had to subtract
from.

An attempt handles its outcomes in order: a failed load and a load that
leaves the certificates due go to retry(), and any other result ends the
failure streak and schedules the next renewal date.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scheduleRenewal() resets the failure count and schedules the attempt at
the renewal time. The initial scheduling, a successful attempt and a
manual reload all return to the regular schedule through it, so none of
them resets the count separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A manual reload loads the factory in the calling thread. The factory
monitor keeps its loads one at a time. For a factory with automatic
renewal, the renewal thread then only moves the schedule to the loaded
certificates, so the reload no longer needs a future, cancellation or
the registry monitor.

start() publishes the renewals after it queues the initial scheduling.
A manual reload that arrives earlier only loads the factory, and the
initial scheduling reads the loaded renewal time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
createSslContext() applies two independent rules. A replacement of an
existing context must have every certificate in its key chains valid
now, whether renewal is enabled or not. With renewal enabled, the
factory calculates the renewal time from the same chains.

keyChains() reads the X.509 chains of the private keys, checkValidity()
checks a replacement, and renewalTime() selects the chain that expires
first. The factory no longer writes a zero renewal time when renewal is
disabled: getRenewalTime() already returns zero in that case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every load stores the renewal time of the loaded certificates, or zero
when renewal is disabled, so getRenewalTime() returns the field as is.
The registry plans only factories with renewal enabled, and renew()
checks the setting itself.

A load with renewal disabled now clears the time. Before this change,
enabling renewal again for the same factory object in one JVM left the
time of the certificates loaded before, and the node scheduled renewal
from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SslContextFactory checks that every certificate in the private-key
chains is valid whenever it loads a key store: at startup, on manual
reload and during renewal. A node with expired or not yet valid
certificates does not start, and client applications that use the
factory, including control.sh, reject such key stores before they
connect. The fix is to supply valid certificates and start again.

TcpDiscoverySslTrustedUntrustedTest.testExpired now expects the node
with the expired certificate to fail at startup.
testExpiredMismatchingCa is removed: that node fails at startup before
it reaches a peer, so the test would check the same thing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
renew() calls init() when renewal is enabled and due. init() already
loads under the factory monitor, so loads stay one at a time.

Without the extra lock, an attempt that waited for another load of the
same factory may read the source again, and node shutdown may wait for
that load. Both happen only when loads race, and the registry still
judges the result by the loaded renewal time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Renewal attempts check executor.isShutdown() for node stop. IgniteKernal
stops the registry right after it sets the node stop flag, so attempts
still do not start or count failures after shutdown begins, and Renewal
no longer needs the kernal context. schedule() relies on the rejected
task after shutdown instead of checking first, and start() checks for
stop once after it prepares the renewals.

Tests that repeated renew() checks through a manual reload are removed:
renew() is init() once renewal is due. The metric reset test is removed
because the failure metric is calculated from the counters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each rule is checked once, at the level that decides it. Renewal window
cases are rows of one factory test, and rejected replacements share one
check that the current context and renewal time stay. One registry test
checks that stop interrupts the attempt, cancels the queued reload and
waits for the executor.

The startup order test is removed: attempts read the factory date when
they are scheduled, so the order does not change the schedule. The node
test checks the renewed context through a new connection;
SslContextReloadTest already covers established connections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants