Repository navigation
feat(oauth2): implement IAM impersonation mTLS transport pinning and 401 recovery - #14212
macastelaz merged 21 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces support for actor tokens in IdentityPoolCredentials (specifically for file-based JSON credential sources over mTLS endpoints) and adds certificate pinning with automatic certificate reloading on 401 Unauthorized errors. It also updates other credential classes to support passing a custom HttpTransportFactory during token refresh, masks actor tokens in logs, and adds corresponding unit tests. The review feedback suggests persisting the reloaded transport factory back to this.transportFactory after a successful retry to prevent subsequent refreshes from failing first, and preserving programmatically set actorTokenSupplier instances in the Builder copy constructor.
73a0301 to
4be79cb
Compare
4be79cb to
ce341c7
Compare
…401 recovery - Pin mTLS HttpTransportFactory across multi-step STS and IAM token exchanges so both requests use the exact same certificate snapshot within a single refresh cycle. - Add 401 Unauthorized recovery with automatic certificate reload from X509Provider and single-retry coordination in IdentityPoolCredentials and ImpersonatedCredentials. - Preserve custom non-default HttpTransportFactory instances when X509Provider is configured. - Add comprehensive unit tests across IdentityPoolCredentialsTest, ImpersonatedCredentialsTest, and OAuth2UtilsTest.
e09d76e to
dadeda6
Compare
…dentials and address review findings - Explicitly scope inner sourceCredentials to CLOUD_PLATFORM_SCOPE in ExternalAccountCredentials.buildImpersonatedCredentials and ImpersonatedCredentials.refreshAccessToken so STS issues tokens authorized to call IAM generateAccessToken even when downstream target scopes are configured via createScoped. - Ensure public no-arg ImpersonatedCredentials.refreshAccessToken delegates without overriding source credential transport settings. - Preserve custom actorTokenSupplier in IdentityPoolCredentials.Builder copy constructor when credentialSource is present. - Ensure HTTP response is closed in a finally block in ImpersonatedCredentials.refreshAccessToken. - Attach initial 401 exception as suppressed when the 401 retry attempt fails in IdentityPoolCredentials.refreshWithRetry. - Add unit tests in IdentityPoolCredentialsTest and ImpersonatedCredentialsTest covering scoped impersonation, custom actorTokenSupplier preservation, and retry exception chaining.
| } | ||
| if (this.impersonatedCredentials != null) { | ||
| return this.impersonatedCredentials.refreshAccessToken(); | ||
| return this.impersonatedCredentials.refreshAccessToken(cycleTransportFactory); |
There was a problem hiding this comment.
When serviceAccountImpersonationUrl is set, IdentityPoolCredentials.refreshWithRetry fetches the subject and actor tokens to build stsTokenExchangeRequest, calls exchangeExternalCredentialForAccessToken, and then immediately delegates to impersonatedCredentials.refreshAccessToken here while discarding stsTokenExchangeRequest. The inner credential then fetches the subject and actor tokens a second time in the same refresh cycle, or four times on a 401 retry. Can we short-circuit before fetching tokens when serviceAccountImpersonationUrl is present?
There was a problem hiding this comment.
Done! Added ImpersonatedCredentials impersonated = getImpersonatedCredentials();
if (impersonated != null) {
return impersonated.refreshAccessToken(cycleTransportFactory);
}
at the start of: IdentityPoolCredentials.refreshWithRetry, AwsCredentials.refreshAccessToken, and
PluggableAuthCredentials.refreshAccessToken before retrieveSubjectToken() / getActorToken() are called.
Subject and actor tokens (and AWS metadata requests) are now fetched only once per refresh attempt (or twice on a 401 retry). Kept the check inside exchangeExternalCredentialForAccessToken as a defensive fallback if called directly.
| * @return the refreshed access token | ||
| * @throws IOException if the token refresh fails | ||
| */ | ||
| public AccessToken refreshAccessToken(HttpTransportFactory transportFactory) throws IOException { |
There was a problem hiding this comment.
The default refreshAccessToken(HttpTransportFactory) delegates to refreshAccessToken() and silently drops the transport factory. All internal subclasses override it, but a third-party subclass would lose transport pinning without warning. Should we annotate this method or class with @InternalExtensionOnly per the repository versioning guidelines?
There was a problem hiding this comment.
Annotated public AccessToken refreshAccessToken(HttpTransportFactory cycleTransportFactory) with @InternalExtensionOnly across ExternalAccountCredentials, IdentityPoolCredentials, AwsCredentials, and PluggableAuthCredentials (and kept ImpersonatedCredentials.refreshAccessToken(HttpTransportFactory) package-private).
| } | ||
|
|
||
| /** | ||
| * Refreshes the access token using the specified transport factory. Default implementation |
There was a problem hiding this comment.
Nit: The Javadoc says the default implementation delegates to refreshAccessToken(), whereas all three subclasses do the reverse by delegating the no-arg method into this overload. Also, naming the parameter transportFactory in AwsCredentials.java:127 and ImpersonatedCredentials.java:601 shadows the class field, whereas IdentityPoolCredentials names it cycleTransportFactory.
There was a problem hiding this comment.
Updated the Javadoc on ExternalAccountCredentials.refreshAccessToken(HttpTransportFactory) to clarify that internal subclasses delegate refreshAccessToken() into this method while the default implementation delegates back to refreshAccessToken(), and renamed the parameter to cycleTransportFactory across all classes to avoid shadowing this.transportFactory.
| .refreshAccessToken(effectiveTransportFactory); | ||
| Credentials authCredentials = | ||
| intermediateAccessToken != null | ||
| ? OAuth2Credentials.create(intermediateAccessToken) |
There was a problem hiding this comment.
Wrapping intermediateAccessToken in OAuth2Credentials.create creates a bare credential with a null quotaProjectId, which drops the x-goog-user-project header from the IAM generateAccessToken request. Can we use OAuth2Credentials.newBuilder().setAccessToken(intermediateAccessToken).setQuotaProjectId(this.sourceCredentials.getQuotaProjectId()).build() so quota attribution is preserved?
There was a problem hiding this comment.
Replaced OAuth2Credentials.create(intermediateAccessToken) with a GoogleCredentials wrapper initialized via GoogleCredentials.newBuilder().setQuotaProjectId(this.sourceCredentials.getQuotaProjectId()).setUniverseDomain(this.sourceCredentials.getUniverseDomain()) so both quotaProjectId (x-goog-user-project) and universeDomain are preserved on the IAM request.
| } | ||
| AccessToken intermediateAccessToken = | ||
| (transportFactory == null) | ||
| ? ((ExternalAccountCredentials) this.sourceCredentials).refreshAccessToken() |
There was a problem hiding this comment.
In the null transport branch, calling refreshAccessToken directly instead of refreshIfExpired bypasses the token cache, lock, and change listeners on sourceCredentials for non-mTLS WIF impersonation. Should line 620 call refreshIfExpired and read getAccessToken, while keeping the uncached refreshAccessToken(effectiveTransportFactory) call on line 621 for pinned mTLS transports?
There was a problem hiding this comment.
Done! When cycleTransportFactory == null, we now call this.sourceCredentials.refreshIfExpired() and pass this.sourceCredentials directly to new HttpCredentialsAdapter(this.sourceCredentials) (preserving token caching, locking, and change listeners), while keeping the uncached refreshAccessToken(effectiveTransportFactory) call when cycleTransportFactory != null.
| "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/test@project.iam.gserviceaccount.com:generateAccessToken")) { | ||
| @Override | ||
| HttpTransportFactory createMtlsTransportFactory(KeyStore keyStore) { | ||
| usedKeyStores.add(keyStore); |
There was a problem hiding this comment.
In the five new retry and impersonation tests, createMtlsTransportFactory records usedKeyStores.add(keyStore) and returns () -> mockTransport, which is the exact same HttpTransport instance regardless of which keystore was passed. The tests would still pass even if refreshWithRetry passed the stale transport on retry. Can we return a distinct transport per keystore so the mock verifies that the retry request ran on the transport created from freshKeyStore, and add tests for a persistent 401 failure and a non-401 IAM error?
There was a problem hiding this comment.
Updated createMtlsTransportFactory(KeyStore) in the impersonation mTLS tests to return a distinct HttpTransport wrapper per KeyStore that records which KeyStore executed each request (requestKeyStores), and added "refreshAccessToken_impersonation_persistent401OnIam_throwsWithSuppressed" and "refreshAccessToken_impersonation_non401OnIam_doesNotRetry".
| } | ||
|
|
||
| @Test | ||
| void builder_actorToken_plainPublicTokenUrl_throws() throws Exception { |
There was a problem hiding this comment.
Nit: The five actor-token endpoint validation tests added at lines 1821 through 1928 duplicate the existing five tests at lines 2873 through 2981 in the same file. Can we remove the duplicate block?
There was a problem hiding this comment.
Removed the 5 duplicate actor-token endpoint validation tests.
| errorResponse.put("error", "invalid_token"); | ||
| errorResponse.put("error_description", "Invalid or expired client certificate."); | ||
| return new MockLowLevelHttpResponse() | ||
| .setStatusCode(statusCode) |
There was a problem hiding this comment.
The early return for non-200 status codes exits before reaching assertEquals(EXPECTED_GRANT_TYPE, grantType), so the failed first attempt never has its request form parameters verified. Can we move the grant_type assertion above the status code check?
There was a problem hiding this comment.
Moved the grant_type and subject_token assertions above if (statusCode != 200) so request parameters are verified on simulated error responses as well.
|
|
||
| @Test | ||
| void isUnauthorizedException_null_returnsFalse() { | ||
| org.junit.jupiter.api.Assertions.assertFalse(OAuth2Utils.isUnauthorizedException(null)); |
There was a problem hiding this comment.
Nit: This file already static-imports assertEquals and assertThrows at lines 35 and 36, but the new tests write out org.junit.jupiter.api.Assertions.assertFalse, org.junit.jupiter.api.Assertions.assertTrue, java.io.IOException, com.google.api.client.http.HttpResponseException, and com.google.api.client.http.HttpHeaders inline. Can we use static imports and top-level imports here?
There was a problem hiding this comment.
Replaced all inline fully qualified names with static and top-level imports in OAuth2UtilsTest.
| .getTransport() | ||
| .addStatusCodeAndMessage(HttpStatusCodes.STATUS_CODE_OK, ""); | ||
|
|
||
| java.util.concurrent.atomic.AtomicReference<HttpTransportFactory> capturedSourceTransport = |
There was a problem hiding this comment.
Nit: The line-wrapping changes to the existing private key string and universe-domain assertion message are unrelated formatting churn, and java.util.concurrent.atomic.AtomicReference is written out inline. Can we revert the unrelated rewrapping to keep the diff clean and add a top-level import for AtomicReference?
There was a problem hiding this comment.
Reverted the line-wrapping changes on SA_PRIVATE_KEY_PKCS8 and the universe-domain assertion message, and replaced inline fully qualified names with top-level imports.
…in IdentityPoolCredentials
…ss and FILE cert pinning
…InternalExtensionOnly
| try { | ||
| KeyStore mtlsKeyStore = this.x509Provider.getKeyStore(); | ||
| this.transportFactory = new MtlsHttpTransportFactory(mtlsKeyStore); | ||
| if (shouldUseMtlsTransportFactory()) { |
There was a problem hiding this comment.
After deserialization this only rebuilds the factory when shouldUseMtlsTransportFactory() is true, so an MtlsHttpTransportFactory subclass or a ServiceLoader-provided factory comes back keyless without a client certificate, whereas the old readObject always restored the keystore-backed factory. Should this also replace the factory when transportFactory instanceof MtlsHttpTransportFactory && !((MtlsHttpTransportFactory) transportFactory).hasKeyStore()?
There was a problem hiding this comment.
Done - updated readObject to also rebuild this.transportFactory via createMtlsTransportFactory(this.x509Provider.getKeyStore()) when this.transportFactory instanceof MtlsHttpTransportFactory && !((MtlsHttpTransportFactory) this.transportFactory).hasKeyStore().
| * @return the refreshed access token | ||
| * @throws IOException if the token refresh fails | ||
| */ | ||
| @InternalExtensionOnly |
There was a problem hiding this comment.
Does this overload need to be public? Commit f2d752e compiled cleanly with it package-private, and making it public @InternalExtensionOnly advertises a call target whose default implementation ignores transportFactory and whose IdentityPoolCredentials override skips 401 cert recovery.
There was a problem hiding this comment.
Oops - that was over expansive, thanks for catching. Reverted back to package-private
| iamTransportFactory.getTransport().setExpireTime(getDefaultExpireTime()); | ||
| iamTransportFactory.getTransport().addStatusCodeAndMessage(HttpStatusCodes.STATUS_CODE_OK, ""); | ||
|
|
||
| IdentityPoolCredentials sourceCredentials = |
There was a problem hiding this comment.
The source credential here defaults to cloud-platform, so the new !currentScopes.contains(CLOUD_PLATFORM_SCOPE) branch at ImpersonatedCredentials.java:613 never runs. Can we pass .setScopes(Collections.singletonList("https://www.googleapis.com/auth/devstorage.read_only")) on the source builder so the test covers that branch?
There was a problem hiding this comment.
Configured devstorage.read_only on sourceCredentials so the !currentScopes.contains(CLOUD_PLATFORM_SCOPE) branch is exercised.
| .setTokenUrl("https://sts.mtls.googleapis.com/v1/token") | ||
| .build(); | ||
| IdentityPoolCredentials deserialized = serializeAndDeserialize(regularCredentials); | ||
| assertTrue( |
There was a problem hiding this comment.
Nit: Can this assert ((MtlsHttpTransportFactory) deserialized.getTransportFactory()).hasKeyStore() too? Right now instanceof MtlsHttpTransportFactory passes even when readObject leaves the deserialized subclass with a null KeyStore.
There was a problem hiding this comment.
Added assertTrue(((MtlsHttpTransportFactory) deserialized.getTransportFactory()).hasKeyStore()).
| @Override | ||
| HttpTransportFactory createMtlsTransportFactory(KeyStore keyStore) { | ||
| usedKeyStores.add(keyStore); | ||
| return () -> transport; |
There was a problem hiding this comment.
Nit: createMtlsTransportFactory returns the same () -> transport instance for both ksA and ksB, so this test passes even if the 401 retry reuses ksA's transport. Could it return a per-keystore recording transport like the impersonation retry tests do?
There was a problem hiding this comment.
Good catch - wired separate MockExternalAccountCredentialsTransport instances (transportA returning 401 for ksA, transportB returning 200 for ksB) and asserted each receives 1 request.
| throw new IllegalArgumentException( | ||
| "Actor tokens are only supported for mTLS token exchanges. Please configure a certificate" | ||
| + " source or MtlsHttpTransportFactory."); | ||
| + " configuration in the credential source or provide an mTLS-enabled transport."); |
There was a problem hiding this comment.
Nit: The exception message says "or provide an mTLS-enabled transport", but isMtlsConfigured() at line 230 only accepts x509Provider != null or an MtlsHttpTransportFactory with hasKeyStore(). Should the message say "or provide an MtlsHttpTransportFactory constructed with a KeyStore"?
There was a problem hiding this comment.
Updated the exception message (and matching test assertion) to "or provide an MtlsHttpTransportFactory constructed with a KeyStore.".
| return credential.refreshAccessToken(); | ||
| }); | ||
| Future<AccessToken> futureA = executor.submit(() -> credential.refreshAccessToken()); | ||
| Future<AccessToken> futureB = executor.submit(() -> credential.refreshAccessToken()); |
There was a problem hiding this comment.
Nit: With the new CyclicBarrier, getKeyStoreCount is deterministically 3, so line 2442 can assert assertEquals(3, ...) instead of >= 3 and verify that credential.getTransportFactory() is still mtlsTransport after the retry.
There was a problem hiding this comment.
Tightened to assertEquals(3, getKeyStoreCount.get()) and added assertSame(mtlsTransport, credential.getTransportFactory()).
| } | ||
|
|
||
| @Test | ||
| void hasCertificateChanged_nullOrSameReference_returnsFalse() throws Exception { |
There was a problem hiding this comment.
Nit: hasCertificateChanged_nullOrSameReference_returnsFalse asserts assertTrue at lines 162-163 and only checks identical references. Could we also compare two distinct KeyStore instances loaded with the same cert so oldCerts.equals(newCerts) is tested directly?
There was a problem hiding this comment.
Added hasCertificateChanged_distinctKeyStoreInstances_comparesCertificates() comparing two distinct KeyStore instances loaded with the same certificate (assertFalse) and with a rotated certificate (assertTrue).
| return new AccessToken("intermediate-sts-token-" + count, null); | ||
| } | ||
|
|
||
| @Override |
There was a problem hiding this comment.
Nit: The createScoped override here is never called because line 1554 already sets cloud-platform scope, and the 401 retry comes from HttpCredentialsAdapter.handleResponse on the null-transport path. Can we drop the unused createScoped override and rename the test to reflect that?
There was a problem hiding this comment.
Removed the unused createScoped override and renamed the test to refreshAccessToken_withoutCycleTransportFactory_externalAccountSourceRetriesOn401FromIam.
| try { | ||
| ImpersonatedCredentials impersonated = getImpersonatedCredentials(); | ||
| if (impersonated != null) { | ||
| return impersonated.refreshAccessToken(cycleTransportFactory); |
There was a problem hiding this comment.
cycleTransportFactory is never null here because this.transportFactory is always initialized in ExternalAccountCredentials. As a result, non-mTLS pools also take the uncached branch in ImpersonatedCredentials, which skips sourceCredentials.refreshIfExpired() on every refresh and clears HttpCredentialsAdapter's 401 retry handler on the IAM request. Can we pass pinnedKeyStore != null ? cycleTransportFactory : null so only pinned mTLS cycles opt out?
There was a problem hiding this comment.
Updated refreshWithRetry() to pass pinnedKeyStore != null ? cycleTransportFactory : null into impersonated.refreshAccessToken(...) so only pinned mTLS cycles opt out of sourceCredentials.refreshIfExpired() caching and HttpCredentialsAdapter's 401 retry handler.
| AccessToken refreshAccessToken(HttpTransportFactory cycleTransportFactory) throws IOException { | ||
| ImpersonatedCredentials impersonated = getImpersonatedCredentials(); | ||
| if (impersonated != null) { | ||
| return impersonated.refreshAccessToken(cycleTransportFactory); |
There was a problem hiding this comment.
Since this.transportFactory is never null, passing cycleTransportFactory here always forces ImpersonatedCredentials onto the uncached path and disables the 401 retry on IAM. Can this pass null since AWS credentials do not pin a per-cycle mTLS transport?
There was a problem hiding this comment.
Updated AwsCredentials to pass cycleTransportFactory == this.transportFactory ? null : cycleTransportFactory into impersonated.refreshAccessToken(...) so standard refreshes retain sourceCredentials.refreshIfExpired() caching and IAM 401 retry handling.
| AccessToken refreshAccessToken(HttpTransportFactory cycleTransportFactory) throws IOException { | ||
| ImpersonatedCredentials impersonated = getImpersonatedCredentials(); | ||
| if (impersonated != null) { | ||
| return impersonated.refreshAccessToken(cycleTransportFactory); |
There was a problem hiding this comment.
Same as AwsCredentials, cycleTransportFactory is always non-null here even though no per-cycle mTLS transport is pinned. Can we pass null so ImpersonatedCredentials keeps sourceCredentials.refreshIfExpired() caching and the IAM 401 retry?
There was a problem hiding this comment.
Updated PluggableAuthCredentials to pass cycleTransportFactory == this.transportFactory ? null : cycleTransportFactory into impersonated.refreshAccessToken(...) so ImpersonatedCredentials retains sourceCredentials.refreshIfExpired() caching and IAM 401 retry handling.
|
|
||
| private final @Nullable String tokenInfoUrl; | ||
| private final @Nullable String serviceAccountImpersonationUrl; | ||
| private transient @Nullable String targetServiceAccountEmail; |
There was a problem hiding this comment.
Because targetServiceAccountEmail is marked transient and mutated after construction at line 316 on a sourceCredentials copy whose serviceAccountImpersonationUrl is null, deserializing ImpersonatedCredentials or its sourceCredentials resets targetServiceAccountEmail to null. After deserialization, getServiceAccountEmail() returns null and drops GOOGLE_EXTERNAL_ACCOUNT_IMPERSONATED_EMAIL in PluggableAuthCredentials. Can we drop transient, make this field final, and set it on the Builder before calling .build()?
There was a problem hiding this comment.
Dropped transient, made targetServiceAccountEmail final, set it on ExternalAccountCredentials.Builder before calling .build() in buildImpersonatedCredentials(), and added an assertion in AwsCredentialsTest.serialize() verifying getServiceAccountEmail() is preserved across serialization.
| && ((MtlsHttpTransportFactory) this.transportFactory).hasKeyStore()); | ||
| } | ||
|
|
||
| private boolean shouldUseMtlsTransportFactory() { |
There was a problem hiding this comment.
In readObject() at lines 465 to 472, if x509Provider.getKeyStore() fails transiently during deserialization, the catch comment notes that cert loading will be handled on refreshAccessToken(). However, if this.transportFactory is a deserialized MtlsHttpTransportFactory subclass with hasKeyStore() == false, shouldUseMtlsTransportFactory() returns false because it omits the this.transportFactory instanceof MtlsHttpTransportFactory && !((MtlsHttpTransportFactory) this.transportFactory).hasKeyStore() check from line 466. Should shouldUseMtlsTransportFactory() include that check too?
There was a problem hiding this comment.
Added (this.transportFactory instanceof MtlsHttpTransportFactory && !((MtlsHttpTransportFactory) this.transportFactory).hasKeyStore()) directly into shouldUseMtlsTransportFactory() and simplified readObject() to call if (shouldUseMtlsTransportFactory()).
| * @return the refreshed access token | ||
| * @throws IOException if the token refresh fails | ||
| */ | ||
| @InternalExtensionOnly |
There was a problem hiding this comment.
Nit: Now that refreshAccessToken(HttpTransportFactory) is package-private here and in IdentityPoolCredentials, AwsCredentials, and PluggableAuthCredentials, @InternalExtensionOnly is no longer needed since that annotation is only for public APIs. Can we drop @InternalExtensionOnly and its import from these four classes?
There was a problem hiding this comment.
Removed @InternalExtensionOnly from refreshAccessToken(HttpTransportFactory) (and its unused imports) across ExternalAccountCredentials, IdentityPoolCredentials, AwsCredentials, and PluggableAuthCredentials.
| // Properly deserialize the transient transportFactory. | ||
| input.defaultReadObject(); | ||
| transportFactory = newInstance(transportFactoryClassName); | ||
| impersonatedCredentials = null; |
There was a problem hiding this comment.
Nit: impersonatedCredentials is declared transient at line 100, so it is already null after input.defaultReadObject() and this assignment can be dropped.
There was a problem hiding this comment.
Removed the redundant this.impersonatedCredentials = null; assignment in readObject().
| */ | ||
| AccessToken refreshAccessToken(@Nullable HttpTransportFactory cycleTransportFactory) | ||
| throws IOException { | ||
| HttpTransportFactory effectiveTransportFactory = |
There was a problem hiding this comment.
Nit: this.transportFactory is already guaranteed non-null by firstNonNull in the constructor and readObject(), so firstNonNull(cycleTransportFactory, this.transportFactory) can replace this nested ternary.
There was a problem hiding this comment.
Replaced the ternary with firstNonNull(cycleTransportFactory, this.transportFactory).
| } catch (IOException e) { | ||
| throw new IOException("Unable to refresh sourceCredentials", e); | ||
| } | ||
| Credentials authCredentials = |
There was a problem hiding this comment.
Nit: Declaring this local as GoogleCredentials authCredentials avoids adding the com.google.auth.Credentials import at line 49.
There was a problem hiding this comment.
Declared authCredentials as GoogleCredentials and removed the com.google.auth.Credentials import.
| } | ||
| // Note: when credentialSource is present, subjectTokenSupplier and actorTokenSupplier | ||
| // Note: when credentialSource is present, subjectTokenSupplier and file-based | ||
| // actorTokenSupplier |
There was a problem hiding this comment.
Nit: // actorTokenSupplier is left orphaned on its own line here. Can we rewrap this comment block cleanly?
There was a problem hiding this comment.
Re-wrapped the comment block in Builder(IdentityPoolCredentials).
|
Found a few issues during cert rotation and impersonation:
|
| return this.impersonatedCredentials.refreshAccessToken(); | ||
| ImpersonatedCredentials impersonated = getImpersonatedCredentials(); | ||
| if (impersonated != null) { | ||
| return impersonated.refreshAccessToken(cycleTransportFactory); |
There was a problem hiding this comment.
Passing cycleTransportFactory here forces ImpersonatedCredentials onto the uncached mTLS path and disables 401 retry on IAM. Can we pass null instead, matching AwsCredentials and PluggableAuthCredentials?
There was a problem hiding this comment.
Updated ExternalAccountCredentials.exchangeExternalCredentialForAccessToken to pass cycleTransportFactory == this.transportFactory ? null : cycleTransportFactory (which evaluates to null when no per-cycle transport override is active, matching AwsCredentials and PluggableAuthCredentials).
| "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/test@project.iam.gserviceaccount.com:generateAccessToken")) { | ||
| @Override | ||
| HttpTransportFactory createMtlsTransportFactory(KeyStore keyStore) { | ||
| return () -> mockTransport; |
There was a problem hiding this comment.
Because createMtlsTransportFactory returns () -> mockTransport without capturing the KeyStore, this test still passes if allowRetry is flipped back to true at line 247. Can we record the KeyStore passed into createMtlsTransportFactory and assert the expected keystore sequence?
There was a problem hiding this comment.
createMtlsTransportFactory(KeyStore keyStore) in refreshAccessToken_impersonation_stsReturns401_retriesOnceViaOuterCycle now records capturedKeyStores.add(keyStore), returns 401 when keyStore == ks1 and 200 only when keyStore == ks2, and asserts assertEquals(Arrays.asList(ks1, ks2, ks2), capturedKeyStores).
| try { | ||
| KeyStore mtlsKeyStore = this.x509Provider.getKeyStore(); | ||
| this.transportFactory = new MtlsHttpTransportFactory(mtlsKeyStore); | ||
| if (shouldUseMtlsTransportFactory()) { |
There was a problem hiding this comment.
For ServiceLoader-registered transport factories, transportFactoryClassName captures the SPI class before the mTLS upgrade. After deserialization, shouldUseMtlsTransportFactory() returns false, so mTLS is not restored. Can we persist a flag so shouldUseMtlsTransportFactory() rebuilds the mTLS transport on deserialization?
There was a problem hiding this comment.
Persisted private final boolean useMtlsTransportFactory on IdentityPoolCredentials (and Builder.useMtlsTransportFactory, preserved across toBuilder()), and added deserialization_respectsUseMtlsTransportFactoryFlag in IdentityPoolCredentialsTest to verify that shouldUseMtlsTransportFactory() rebuilds the mTLS transport on deserialization.
| assertEquals(testCredentials.hashCode(), deserializedCredentials.hashCode()); | ||
| assertEquals(testCredentials.toString(), deserializedCredentials.toString()); | ||
| assertSame(Clock.SYSTEM, deserializedCredentials.clock); | ||
| assertNotNull(deserializedCredentials.getServiceAccountEmail()); |
There was a problem hiding this comment.
Because serviceAccountImpersonationUrl is set on the fixture, getServiceAccountEmail() reads the email from the URL rather than targetServiceAccountEmail. Can we assert getServiceAccountEmail() on the deserialized inner source copy where the URL was cleared?
There was a problem hiding this comment.
Added assertions on deserializedInnerSource ((AwsCredentials) deserializedCredentials.getImpersonatedCredentials().getSourceCredentials()) verifying both assertNull(deserializedInnerSource.getServiceAccountImpersonationUrl()) and assertEquals(testCredentials.getServiceAccountEmail(), deserializedInnerSource.getServiceAccountEmail()).
The new constructor and getHttpStatusCode() change the computed serialVersionUID, so instances serialized by released versions would fail to load. Same change as in PR googleapis#14212.
| @CanIgnoreReturnValue | ||
| public Builder setHttpTransportFactory(HttpTransportFactory transportFactory) { | ||
| super.setHttpTransportFactory(transportFactory); | ||
| this.useMtlsTransportFactory = isDefaultOrMtlsTransportFactory(transportFactory); |
There was a problem hiding this comment.
can we just simply this to set this only in the constructor?
There was a problem hiding this comment.
Builder(IdentityPoolCredentials) needs to preserve this.useMtlsTransportFactory = credentials.useMtlsTransportFactory across clones (toBuilder(), createScoped(), buildImpersonatedCredentials()) because super(credentials) copies credentials.transportFactory into builder.transportFactory — which on an already-constructed credential is no longer null and may have been resolved via ServiceLoader (per #14212 (comment)) or an overridden createMtlsTransportFactory rather than DefaultHttpTransportFactory. If the constructor re-evaluated isDefaultOrMtlsTransportFactory(builder.transportFactory) on the cloned builder without preserving the flag, useMtlsTransportFactory would flip from true to false on the clone. And because Builder(IdentityPoolCredentials) copies that flag, setHttpTransportFactory updates it in case a caller calls credentials.toBuilder().setHttpTransportFactory(customFactory).build(). Added a comment on the builder copy constructor to clarify this!
| @VisibleForTesting | ||
| boolean shouldUseMtlsTransportFactory() { | ||
| return this.useMtlsTransportFactory | ||
| || (this.transportFactory instanceof MtlsHttpTransportFactory | ||
| && !((MtlsHttpTransportFactory) this.transportFactory).hasKeyStore()); | ||
| } |
There was a problem hiding this comment.
qq, this isn't immediately clear to me why we can't just use the value of useMtlsTransportFactory? useMtlsTransportFactory seems to be the reuslt of isDefaultOrMtlsTransportFactory which looks like it already checks for this
There was a problem hiding this comment.
At construction time, if a caller supplies a custom MtlsHttpTransportFactory subclass that already has a KeyStore (e.g. new CustomMtlsHttpTransportFactory(ks)), isDefaultOrMtlsTransportFactory returns false so useMtlsTransportFactory is false (preserving the custom subclass during construction and normal refreshes).
However, upon deserialization, ExternalAccountCredentials.readObject() instantiates transportFactoryClassName via its no-arg constructor, leaving this.transportFactory as a keyless MtlsHttpTransportFactory (hasKeyStore() == false) while the deserialized useMtlsTransportFactory field is still false. This second check ensures readObject() (and refreshAccessToken() if the cert load in readObject() fails transiently) still rebuilds a KeyStore-backed MtlsHttpTransportFactory in that case (see #14212 (comment)). Added an inline comment here to explain this!
There was a problem hiding this comment.
Hmm, would it be possible to use the above isDefaultOrMtlsTransportFactory to simply the checks?
There was a problem hiding this comment.
Done — simplified shouldUseMtlsTransportFactory() to return this.useMtlsTransportFactory || isDefaultOrMtlsTransportFactory(this.transportFactory);.
| /* allowRetry= */ false); | ||
| } | ||
|
|
||
| private AccessToken refreshWithRetry( |
There was a problem hiding this comment.
a few quick notes: This method is quite large and it's a bit hard to understand what is going on here (also admittedly I'm not as familiar with IdentityPoolCredentials or 3p creds in general).
Can we split this into a series of helpers. On a high level I think I see a few things
- Resolving the transport
- Retrieving the subject and actor tokens
- Making the request and setting the scopes
I think we can keep the logic regarding retries and all that inside this method
There was a problem hiding this comment.
Done — extracted resolveCycleTransportFactory, retrieveSubjectAndActorTokens, and buildAndExchangeStsRequest as private helpers while keeping the pinnedKeyStore snapshot and retry handling inside refreshWithRetry.
There was a problem hiding this comment.
Thanks. I still find this a bit hard to follow. I think the cor elogic is in helpers, but the main challenge is with the retry logic.
Is it possible to not use recursion for this? IMO it's easier to follow just seeing the 1 or 2 requests flowing in the code.
I think it may be easier to separate the refresh call from the retry logic.
Perhaps we can create a helper for making the refresh call and invoke it once or twice inside this method depending on whether we are allowed to retry or not.
There was a problem hiding this comment.
Done — separated the single-attempt refresh execution into executeRefresh(...) and removed the recursion from refreshWithRetry(...). Non-retryable callers now invoke executeRefresh(...) directly, and refreshWithRetry(...) executes Attempt 1 in the try block and Attempt 2 directly in the catch block.
| } | ||
| } | ||
|
|
||
| private static @Nullable String extractAndEncodeLeafCertificate(@Nullable KeyStore keyStore) |
There was a problem hiding this comment.
note for us in a future PR: looks like parsing through the keystore is a repeated operation in multiple spots (I think I recall seeing this in mtls PR before). Maybe we can extract some common functionality
There was a problem hiding this comment.
Agreed! Once both PRs land, we can extract the KeyStore leaf-certificate lookup into a shared utility method in a follow-up PR.
| ImpersonatedCredentials.extractTargetPrincipal(serviceAccountImpersonationUrl); | ||
| sourceBuilder | ||
| .setServiceAccountImpersonationUrl(null) | ||
| .setScopes(Collections.singletonList(OAuth2Utils.CLOUD_PLATFORM_SCOPE)); |
There was a problem hiding this comment.
can you add a note to explain why we add this scope?
There was a problem hiding this comment.
Done — added a comment explaining that sourceCredentials needs CLOUD_PLATFORM_SCOPE to authenticate to the IAM generateAccessToken endpoint, while this.scopes are passed to ImpersonatedCredentials for the final impersonated token.
| ExternalAccountCredentials.Builder sourceBuilder; | ||
| if (this instanceof AwsCredentials) { | ||
| sourceCredentials = | ||
| AwsCredentials.newBuilder((AwsCredentials) this) | ||
| .setServiceAccountImpersonationUrl(null) | ||
| .build(); | ||
| sourceBuilder = AwsCredentials.newBuilder((AwsCredentials) this); | ||
| } else if (this instanceof PluggableAuthCredentials) { | ||
| sourceCredentials = | ||
| PluggableAuthCredentials.newBuilder((PluggableAuthCredentials) this) | ||
| .setServiceAccountImpersonationUrl(null) | ||
| .build(); | ||
| sourceBuilder = PluggableAuthCredentials.newBuilder((PluggableAuthCredentials) this); | ||
| } else { | ||
| sourceCredentials = | ||
| IdentityPoolCredentials.newBuilder((IdentityPoolCredentials) this) | ||
| .setServiceAccountImpersonationUrl(null) | ||
| .build(); | ||
| sourceBuilder = IdentityPoolCredentials.newBuilder((IdentityPoolCredentials) this); | ||
| } |
There was a problem hiding this comment.
can this just be simplified to sourceBuilder = toBuilder()?
There was a problem hiding this comment.
ExternalAccountCredentials inherits the concrete public GoogleCredentials.Builder toBuilder() from GoogleCredentials and doesn't override toBuilder() (since ExternalAccountCredentials.Builder is abstract), so toBuilder() here has return type GoogleCredentials.Builder rather than ExternalAccountCredentials.Builder. Re-declaring @Override public abstract Builder toBuilder() on ExternalAccountCredentials would turn an inherited concrete method into an abstract method on a public class (which is a breaking binary/source compatibility change in Clirr), and casting (ExternalAccountCredentials.Builder) toBuilder() would throw ClassCastException at runtime if a custom subclass didn't override toBuilder(), so we kept the explicit instanceof branches here.
| AccessToken refreshAccessToken(HttpTransportFactory cycleTransportFactory) throws IOException { | ||
| return refreshAccessToken(); | ||
| } | ||
|
|
||
| AccessToken refreshAccessToken( | ||
| HttpTransportFactory cycleTransportFactory, @Nullable KeyStore pinnedKeyStore) | ||
| throws IOException { | ||
| return refreshAccessToken(cycleTransportFactory); |
There was a problem hiding this comment.
qq, it's not super clear to me why we need this override. Is this due to the funky behavior with 3p creds and impersonation. IIRC, it seems to call each other in some weird way that ends up being somewhat of a cycle?
There was a problem hiding this comment.
One more note: can these methods just be abstract methods that the subclasses override?
There was a problem hiding this comment.
Yes, spot on — when ExternalAccountCredentials is configured with service_account_impersonation_url, it builds an internal ImpersonatedCredentials whose sourceCredentials is a cloned copy of the ExternalAccountCredentials (with serviceAccountImpersonationUrl = null). During a refresh:
- The outer
IdentityPoolCredentials.refreshAccessToken()snapshotspinnedKeyStoreandcycleTransportFactoryand delegates toimpersonatedCredentials.refreshAccessToken(cycleTransportFactory, pinnedKeyStore). ImpersonatedCredentialsthen calls back into itssourceCredentials(typed asExternalAccountCredentials) viaexternalSource.refreshAccessToken(effectiveTransportFactory, pinnedKeyStore)to exchange the 3P subject token at STS using that same pinned mTLS transport andKeyStore, before calling IAMgenerateAccessToken.
On making them abstract: because ExternalAccountCredentials is a public abstract class with protected constructors and these overloads are intentionally package-private (per #14212 (comment)), making a package-private method abstract in Java would prevent any class outside com.google.auth.oauth2 from subclassing ExternalAccountCredentials (and making them protected abstract would be a breaking public API change). Expanded the Javadoc here to explain both the impersonation call flow and why the default fallback is provided.
There was a problem hiding this comment.
This looks primarily really only used/ needed for IdentityPoolCredentials. Can this just be put into that class?
There was a problem hiding this comment.
Done — removed refreshAccessToken(HttpTransportFactory) and refreshAccessToken(HttpTransportFactory, KeyStore) from ExternalAccountCredentials (and AwsCredentials / PluggableAuthCredentials) so they live solely on IdentityPoolCredentials.
| return refreshAccessToken(this.transportFactory); | ||
| } | ||
|
|
||
| @Override | ||
| AccessToken refreshAccessToken(HttpTransportFactory cycleTransportFactory) throws IOException { | ||
| ImpersonatedCredentials impersonated = getImpersonatedCredentials(); | ||
| if (impersonated != null) { | ||
| return impersonated.refreshAccessToken( | ||
| cycleTransportFactory == this.transportFactory ? null : cycleTransportFactory); | ||
| } |
There was a problem hiding this comment.
I'm not entirely sure I follow what is going on here. This looks to set refreshAccessToken with the existing transportFactory? There would not be any "cycle" here. This also seems to be the case for PluggableAuth and Aws, but IdentityPool is different.
Does it need this overload then for refreshAccessToken(HttpTransportFactory cycleTransportFactory)?
There was a problem hiding this comment.
Good point — neither AwsCredentials nor PluggableAuthCredentials uses per-cycle transport pinning, so they don't need the refreshAccessToken(HttpTransportFactory) overload. Removed the overload from ExternalAccountCredentials, AwsCredentials, and PluggableAuthCredentials (keeping it only on IdentityPoolCredentials), while retaining the early getImpersonatedCredentials() delegation inside refreshAccessToken() so the subject token isn't fetched twice when service account impersonation is configured.
| && this.x509Provider != null | ||
| && shouldUseMtlsTransportFactory() |
There was a problem hiding this comment.
in this method would these conditions every be false?
There was a problem hiding this comment.
Previously they could be false because non-mTLS refreshes, refreshAccessToken(cycleTransportFactory), and the recursive retry call all went through refreshWithRetry. With executeRefresh(...) separated out and recursion removed, refreshWithRetry is now only called when this.x509Provider != null && shouldUseMtlsTransportFactory(), so those checks are no longer needed and have been removed.
| && (OAuth2Utils.isUnauthorizedException(e) | ||
| || OAuth2Utils.isInvalidGrantException(e) | ||
| || isRetryableTransportException(e))) { |
There was a problem hiding this comment.
can we extract this to a helper to help explain this. isMtlsRetryableExceptions?
There was a problem hiding this comment.
Done — extracted isMtlsRetryableException(Throwable) to encapsulate the retryable exception check.
…cope cycle overloads
No region tags are edited in this PR.This comment is generated by snippet-bot.
|
133b9cc to
fe93d90
Compare
80b8228
into
googleapis:oauth2-bound-tokens
🥞 Stacked PRs
Summary
Propagates the cycle-pinned mTLS transport factory into
ImpersonatedCredentialsduring Workload Identity Federation with Service Account Impersonation, ensuring the IAMgenerateAccessTokencall uses the exact same pinned client certificate as the STS token exchange.Also extends 401 recovery in
IdentityPoolCredentialsto catch unauthorized responses across both STS and IAM calls, reloading fresh certificates and re-executing the full refresh cycle.Test Coverage
oauth2_http(including transport pinning lifecycle, multi-cycle rotation, 401 retry on IAM, and suppressed error handling).com.spotify.fmt:fmt-maven-plugin:2.25).Manual Testing
Sample test "app": https://paste.googleplex.com/5495919190081536
Test results: https://paste.googleplex.com/5757145073713152
See b/542238030 for tracking.