Skip to content

feat(gax): support cancelling API calls via CallOptions.signal - #9462

Open
feywind wants to merge 2 commits into
mainfrom
pubsub/hedging-gax-abort
Open

feywind wants to merge 2 commits into
mainfrom
pubsub/hedging-gax-abort

Conversation

@feywind

@feywind feywind commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Overview

This PR is Layer 1 of 7 in Stack #9467, which implements Publish Hedging for @google-cloud/pubsub to mitigate high-percentile (p99/p99.9) publish tail latencies by sending backup (hedged) publish RPCs when an in-flight batch takes longer than a configurable hedgeDelay threshold.

Stack Structure

  1. feat(gax): support cancelling API calls via CallOptions.signal #9462 - feat(gax): support AbortSignal cancellation in CallOptions and createApiCall (this PR)
  2. feat(pubsub): add HedgingOptions validation and token bucket #9463 - feat(pubsub): add HedgingOptions validation and HedgingTokenBucket
  3. feat(pubsub): add CancellationSharer and HedgingScheduler #9464 - feat(pubsub): add CancellationSharer and HedgingScheduler
  4. feat(pubsub): add OpenTelemetry events and logging for hedged publish #9465 - feat(pubsub): add OpenTelemetry events and logging for hedged publish
  5. feat(pubsub): integrate publish hedging into MessageQueue #9466 - feat(pubsub): integrate publish hedging into MessageQueue
  6. test(pubsub): add unit tests for publish hedging queue integration #9635 - test(pubsub): add unit tests for publish hedging queue integration
  7. test(pubsub): add publish hedging system tests #9636 - test(pubsub): add publish hedging system tests

Summary of Changes in This PR

To cleanly cancel losing publish attempts when either the original or a hedged publish RPC wins - even when the gapic client is invoked via callback or util.promisify - this PR adds standard ECMA AbortSignal support to google-gax:

  • CallOptions & CallSettings (core/packages/gax/src/gax.ts):
    • Added optional signal?: AbortSignal to CallOptions and CallSettings, propagating options.signal in CallSettings.prototype.merge.
  • createApiCall (core/packages/gax/src/createApiCall.ts):
    • When thisSettings.signal is provided on an API call:
      • If signal.aborted is already true when the call starts, ongoingCall.cancel() is invoked immediately on the next tick.
      • Otherwise, registers a {once: true} 'abort' listener on thisSettings.signal that invokes ongoingCall.cancel(), and removes the listener once ongoingCall.promise settles to prevent memory leaks.
  • Unit Tests (core/packages/gax/test/unit/apiCallable.ts):
    • Added unit tests verifying AbortSignal cancellation for both in-flight calls and pre-aborted signals.

@feywind
feywind added this pull request to stack #9467 September 29, 2026 02:09

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for cancelling ongoing API calls using an AbortSignal passed via CallOptions, and includes comprehensive unit tests. The review feedback highlights a critical memory leak and MaxListenersExceededWarning risk in Node.js because the 'abort' event listener is not cleaned up when the API call completes. A code suggestion is provided to ensure the listener is removed upon call completion.

Comment thread core/packages/gax/src/createApiCall.ts
@feywind

feywind commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

I've hand-reviewed this PR.

Comment thread core/packages/gax/src/createApiCall.ts Outdated
Comment on lines +146 to +159
signal.addEventListener('abort', abortListener, {once: true});
const removeAbortListener = () => {
signal.removeEventListener('abort', abortListener);
};
if (ongoingCall instanceof StreamProxy) {
ongoingCall.once('close', removeAbortListener);
ongoingCall.once('end', removeAbortListener);
ongoingCall.once('error', removeAbortListener);
} else {
const originalCallback = ongoingCall.callback;
ongoingCall.callback = (...args: Parameters<APICallback>) => {
removeAbortListener();
originalCallback?.(...args);
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If signal.aborted is true, do we need to still go through this code? I think it can be moved into the previous else block

Comment thread core/packages/gax/src/createApiCall.ts Outdated
Comment on lines +140 to +142
if (signal.aborted) {
ongoingCall.cancel();
} else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was curious about when .cancel() would actually be invoked, and I think there's a bug here. ongoingCall.cancel() is going to be called synchronously, so I think it's going to cause the error prematurely for both streams and promises.

I think something like

if (signal) {
  const abortListener = () => {
    ongoingCall.cancel();
  };
}

if (signal.aborted) {
  process.nextTick(() => {
    ongoingCall.cancel();
  });
}

might be the solution so the callback execution is deferred until the stack is finished executing.

Comment thread core/packages/gax/src/createApiCall.ts Outdated
Comment on lines +151 to +153
ongoingCall.once('close', removeAbortListener);
ongoingCall.once('end', removeAbortListener);
ongoingCall.once('error', removeAbortListener);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using .once() will only register the first one that fires, and the other 2 listeners won't get unattached. I'd suggest a cleanup callback:

const cleanup = () => {
  removeAbortListener();
  ongoingCall.removeListener('close', cleanup);
  ongoingCall.removeListener('end', cleanup);
  ongoingCall.removeListener('error', cleanup);
};

ongoingCall.once('close', cleanup);
ongoingCall.once('end', cleanup);
ongoingCall.once('error', cleanup);

@feywind
feywind force-pushed the pubsub/hedging-gax-abort branch from 2e30d7d to 98dd2ff Compare October 19, 2026 21:57

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants