fix(auth,gax): align Java mTLS and agent identity certificate handling with Python parity

[macastelaz]

Why

An audit comparing the Java (agentic-identities-bound-token) and Python (googleapis/google-cloud-python#18559) agent identity and mTLS implementations surfaced several edge-case behavioral differences:

1. GAX silent fallback on mTLS initialization error: When CertificateBasedAccess.useMtlsClientCertificate() is true and DefaultMtlsProviderFactory.create() throws an IOException (for example, CertificateSourceUnavailableException when a certificate file disappears mid-read or a configured certificate path is invalid/corrupt), EndpointContext, InstantiatingGrpcChannelProvider, and InstantiatingHttpJsonChannelProvider caught the IOException and silently degraded to mtlsProvider = null (sending requests over an unauthenticated channel or regular endpoint without client certificates).
2. Startup polling when GOOGLE_API_CERTIFICATE_CONFIG is unset: In AgentIdentityUtils.getWellKnownCertificatePathWithRetry(), when GOOGLE_API_CERTIFICATE_CONFIG was unset and GOOGLE_API_USE_CLIENT_CERTIFICATE=true was set on a non-agent GCE VM where no certificates exist, Java polled the well-known Cloud Run paths for ~3 seconds and threw an IOException on timeout instead of falling back to unbound tokens in 0ms.
3. Symlinked parent directories and empty (0-byte) files during startup polling:
   • AgentIdentityUtils.isPathInWellKnownDir(String) used Files.exists(path) before calling toRealPath(), which skipped symlink resolution when the leaf certificate file did not yet exist on disk (e.g., on systems where /var/run is a symlink to /run and GOOGLE_API_CERTIFICATE_CONFIG uses /run/secrets/workload-spiffe-credentials/...).
   • AgentIdentityCacheUtils.checkExistsOrAccessDenied(Path) only checked file existence via BasicFileAttributes without verifying attrs.isRegularFile() && attrs.size() > 0, which could cause startup polling to return immediately on a newly created 0-byte file before certificate contents were written.
4. GOOGLE_API_USE_CLIENT_CERTIFICATE non-boolean values: MtlsUtils.useMtlsClientCertificate() and MtlsUtils.getWorkloadCertPath() only checked for "true" and "false", allowing arbitrary non-boolean values (e.g., "invalid" or "1") to fall through to auto-discovery instead of treating any non-"true" value as disabled.

What changes

• GAX (CertificateBasedAccess, EndpointContext, InstantiatingGrpcChannelProvider, InstantiatingHttpJsonChannelProvider):
  • Propagate IOException from DefaultMtlsProviderFactory.create() (routed via package-private CertificateBasedAccess.createDefaultMtlsProvider()) when certificateBasedAccess.useMtlsClientCertificate() is true, failing closed instead of degrading to mtlsProvider = null.
  • When useMtlsClientCertificate() is false, IOException during default MtlsProvider creation continues to be logged at FINE and ignored.
• AgentIdentityUtils & AgentIdentityCacheUtils:
  • Remove the retry loop and IOException throw from getWellKnownCertificatePathWithRetry() when GOOGLE_API_CERTIFICATE_CONFIG is unset; check well-known paths once (GKE_CREDENTIAL_BUNDLE_PATH, WELL_KNOWN_CREDENTIAL_BUNDLE_PATH, WELL_KNOWN_CERT_PATH + WELL_KNOWN_KEY_PATH) and immediately return certsPresent=false if none are present.
  • Startup polling remains exclusively gated on GOOGLE_API_CERTIFICATE_CONFIG pointing inside /var/run/secrets/workload-spiffe-credentials/.
  • Add resolveRealPathAllowingMissing(Path) to resolve symlinks on the nearest existing ancestor directory and re-append remaining relative path components in isPathInWellKnownDir(String).
  • Require attrs.isRegularFile() && attrs.size() > 0 in AgentIdentityCacheUtils.checkExistsOrAccessDenied(Path) so polling waits until certificate files are non-empty regular files (> 0 bytes).
• MtlsUtils:
  • Treat any non-empty trimmed value of GOOGLE_API_USE_CLIENT_CERTIFICATE other than "true" (case-insensitive) as disabled (false / null) in both useMtlsClientCertificate() and getWorkloadCertPath().

Tests

• EndpointContextTest, InstantiatingGrpcChannelProviderTest, InstantiatingHttpJsonChannelProviderTest:
  • Added tests verifying that IOException from createDefaultMtlsProvider() is propagated when useMtlsClientCertificate() is true and ignored when useMtlsClientCertificate() is false.
• AgentIdentityUtilsTest:
  • Updated unset GOOGLE_API_CERTIFICATE_CONFIG tests to verify 0-sleep non-polling behavior and immediate null return when no well-known certificates exist, even when GOOGLE_API_USE_CLIENT_CERTIFICATE=true.
  • Added tests for 0-byte -> populated file polling in getConfigFilePathWithRetry and getWorkloadCertPathFromConfigWithRetry.
  • Added test for isPathInWellKnownDir with symlinked parent directory and non-existent leaf file.
• MtlsUtilsTest:
  • Added tests verifying non-boolean values of GOOGLE_API_USE_CLIENT_CERTIFICATE ("invalid") return false in useMtlsClientCertificate() and null in getWorkloadCertPath() even when workload certificates exist on disk.

[gemini-code-assist]

Code Review

This pull request refactors mTLS configuration and certificate resolution. Key changes include optimizing the handling of the GOOGLE_API_USE_CLIENT_CERTIFICATE environment variable, ensuring that file checks verify that paths are non-empty regular files, resolving symlinks while preserving trailing missing elements, and removing startup polling when the certificate configuration is unset. Additionally, the default MtlsProvider creation is refactored to propagate IOExceptions rather than silently falling back to non-mTLS channels. Review feedback suggests optimizing useMtlsClientCertificate by avoiding redundant environment variable fetches and adding logging for unexpected exceptions caught during certificate checks in AgentIdentityUtils.

[nbayati]

Just a few nits

[nbayati]

This test still passes if the attrs.size() > 0 check in checkExistsOrAccessDenied is reverted. Without the check, the 0-byte files count as ready and polling returns right away, but loadAndVerifyCredentials then makes 3 read attempts with a timeService.sleep between them on the same fake clock. The callback fills in the cert and key on those two retry sleeps, so the test still ends with 2 sleeps and a valid CertInfo.

Does it make sense to fill the files in after the load retries would have run out (e.g. the cert on sleep 4 and the key on sleep 5, and assert 5 sleeps)?

[macastelaz]

Good catch. Updated the test in a14152d5ca4 to populate the certificate on sleep 4 and the private key on sleep 5, and assert 5 total sleeps so loadAndVerifyCredentials's 3 attempts (2 sleeps) cannot mask a missing 0-byte check.

[nbayati]

nit: Could we also add a test where createDefaultMtlsProvider() throws CertificateSourceUnavailableException (here and in the two channel provider tests)? That covers the GOOGLE_API_USE_CLIENT_CERTIFICATE=true fallback when no certs exist on disk, where build() should still succeed without mTLS on the regular endpoint.

[macastelaz]

Done in a14152d5ca4. Added unit tests covering CertificateSourceUnavailableException from createDefaultMtlsProvider() in EndpointContextTest, InstantiatingGrpcChannelProviderTest, and InstantiatingHttpJsonChannelProviderTest.

[nbayati]

nit: this doesn't follow a symlink whose target doesn't exist yet (e.g. a config path outside the well-known dir that links to a file Cloud Run hasn't written), so it isn't quite os.path.realpath. That's fine for the /var/run -> /run case this targets, but could we reword the comment so it doesn't claim full parity?

[macastelaz]

Done in a14152d5ca4. Reworded the Javadoc on resolveRealPathAllowingMissing to note that it resolves symlinks on existing ancestor directories (such as /var/run to /run before startup files are created) without claiming full os.path.realpath parity.