feat(spec.tls.additionalTrustedCAs)

[yoav-katz]

Problem:
When a PostgresCluster is configured with a TLS issuer that returns no CA certificate of its own - most commonly an ACME issuer such as Let's Encrypt, which writes only tls.crt and tls.key into the issued Secret - the operator has nothing to put in the files that verify peers. Every internal TLS path that needs a trust anchor breaks:

• ssl_ca_file (/pgconf/tls/ca.crt), which Postgres uses to verify the _crunchyrepl client certificate
• sslrootcert (/pgconf/tls/replication/ca.crt), used by Patroni for replication and pg_rewind
• pgbackrest.ca-roots, the single file pgBackRest uses both as tls-server-ca-file and as pgN-/repoN-host-ca-file
• pgBouncer's backend authority, used to verify the Postgres server it proxies to

Reconciliation previously failed outright with external issuer did not return a CA certificate for the instance - an accurate message pointing at a field the user cannot change, since an ACME issuer never returns one.

Cause:
The operator only ever trusted CA material it could obtain itself: the internal PKI root, or the ca.crt returned by the configured cert-manager Issuer/ClusterIssuer. There was no way to supply a supplementary anchor, so a valid, correctly-configured ACME setup produced an incomplete (or entirely empty) trust bundle.

Solution:
A new optional field, spec.tls.additionalTrustedCAs, takes a list of Secret references. Each Secret carries a PEM-encoded CA bundle under key ca.crt. The operator merges them into every trust file listed above.

Merging, not replacing. The additional CAs are always concatenated with whatever the issuer already provided - the issuer CA, the internal PKI root, or nothing at all. Anything that verified before the field was set still verifies after. Concatenation goes through pki.TrustBundle, which:

• accepts any number of sources and preserves their order,
• de-duplicates by certificate DER, so the same anchor listed twice appears once,
• re-encodes each block canonically, so bundles differing only in whitespace or PEM headers produce identical bytes (no spurious Secret churn / pod rolls),
• drops non-CERTIFICATE blocks and CERTIFICATE blocks whose DER isn't a parseable X.509, rather than writing unexamined bytes into a trust file.

Where the merged bundle is written. For the internal-PKI path the operator owns the certificate Secrets, so the merge happens in place (<cluster>-tls ca.crt, and the replication Secret's ca.crt). When cert-manager owns the certificate Secrets the operator must not write into them - cert-manager rebuilds them on every issuance and the operator's apply takes field ownership by force, so the two controllers would overwrite each other indefinitely. In that case the merged bundle goes to a Secret the operator does own, <cluster>-ca-bundle, projected into the certificate volume at both ca.crt paths. The certificate Secrets' own ca.crt items are stripped from their projections (withoutCA), because a projected volume rejects two sources writing the same path.

Unlike the certificate Secrets, which K8SPG-330 deliberately retains after cluster deletion, <cluster>-ca-bundle holds nothing that can't be rebuilt and is owned by the cluster, so it's garbage-collected with it.

pgBouncer. The backend authority projection switches to the merged bundle when one exists. The cluster-wide anchors are also appended to pgBouncer's frontend bundle, alongside the pre-existing spec.proxy.pgBouncer.additionalTrustedCAs (K8SPG-952) - without them an ACME-issued frontend certificate has nothing to chain to.

Rotation and late creation. The referenced Secrets are indexed (IndexFieldAdditionalTrustedCASecrets) and watched, so creating a previously-missing CA Secret or rotating an existing one's ca.crt enqueues the cluster immediately, instead of leaving the bundle absent or stale until some unrelated reconcile happens to fire.

Error handling.

• A referenced Secret that doesn't exist yet is logged and skipped; the watch picks it up when it's created.
• A Secret that exists but has no ca.crt, or whose ca.crt contains no PEM certificate, is a configuration mistake and fails reconciliation with the Secret name and the key.
• An empty resulting bundle distinguishes the two causes: if some source had content, "failed to parse CA certificate"; if all sources were empty, "issuer did not return a CA certificate; set spec.tls.additionalTrustedCAs to supply one" - pointing the reader at the field that actually fixes it.

No-op when unset. With the field absent, the certificate volume and every trust file are byte-identical to before. The merge is also skipped entirely under certManagementPolicy: userProvidedOnly and when spec.customTLSSecret is set - in both cases the operator owns no Secret to merge into.

Note that setting the field adds a source to the certificate volume, which rolls the instances.

Coverage. e2e-tests/tests/cert-manager-tls/29-verify-additional-trusted-cas.yaml signs a standalone CA that no issuer in the test chain uses, adds it to a running cluster, and asserts both anchors (the issuer's and the added one) are present in /pgconf/tls/ca.crt, the replication ca.crt, and the pgBackRest roots.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PG version? - not releated
• Does the change support oldest and newest supported Kubernetes version? - not releated

[JNKPercona]

Test Name	Result	Time
backup-enable-disable	passed	00:14:16
builtin-extensions	passed	00:06:22
custom-extensions	passed	00:17:57
cert-manager-tls	passed	00:13:14
cert-management-policy	passed	00:06:09
custom-envs	passed	00:20:48
custom-tls	passed	00:06:38
database-init-sql	passed	00:03:27
demand-backup	failure	00:31:17
demand-backup-offline-snapshot	failure	00:13:24
dynamic-configuration	passed	00:03:55
extra-volumes	passed	00:03:58
finalizers	passed	00:04:54
init-deploy	passed	00:03:41
huge-pages	passed	00:03:13
logical-replica	passed	00:10:19
major-upgrade-14-to-15	passed	00:11:58
major-upgrade-15-to-16	passed	00:11:11
major-upgrade-16-to-17	passed	00:11:51
major-upgrade-17-to-18	passed	00:18:21
major-upgrade-18-to-19	passed	00:10:44
ldap	passed	00:03:45
ldap-tls	passed	00:05:11
logcollection	passed	00:08:39
monitoring	passed	00:09:31
operator-upgrade-cert-manager	failure	00:04:29
operator-upgrade-extensions	failure	00:08:53
one-pod	passed	00:06:26
repo-host-autogrow	passed	00:02:01
operator-self-healing	passed	00:11:12
pgbouncer	passed	00:08:13
pg-tde	passed	00:11:52
pg-tde-wal-encrypt	failure	00:18:16
pitr	passed	00:13:47
scaling	passed	00:05:55
scheduled-backup	passed	00:25:25
self-healing	passed	00:10:00
sidecars	passed	00:03:01
standby-pgbackrest	passed	00:21:22
standby-streaming	passed	00:17:29
start-from-backup	passed	00:12:27
start-from-volumes	passed	00:03:52
tablespaces	passed	00:07:25
telemetry-transfer	passed	00:06:37
upgrade-consistency	passed	00:07:04
upgrade-minor	passed	00:13:52
users	passed	00:05:38
migration-from-crunchy-standby	passed	00:18:58
migration-from-crunchy-pv	passed	00:10:08
migration-from-crunchy-backup-restore	passed	00:16:51

Summary	Value
Tests Run	50/50
Job Duration	01:56:47
Total Test Time	08:46:20

commit: c16ea2a
image: perconalab/percona-postgresql-operator:PR-1784-c16ea2aad