test(pubsub): add publish hedging system tests

[feywind]

Overview

This PR is Layer 7 of 7 in Stack #9467, which implements Publish Hedging for @google-cloud/pubsub to mitigate high-percentile (p99/p99.9) publish tail latencies by sending backup (hedged) publish RPCs when an in-flight batch takes longer than a configurable hedgeDelay threshold.

Stack Structure

1. feat(gax): support cancelling API calls via CallOptions.signal #9462 - feat(gax): support AbortSignal cancellation in CallOptions and createApiCall
2. feat(pubsub): add HedgingOptions validation and token bucket #9463 - feat(pubsub): add HedgingOptions validation and HedgingTokenBucket
3. feat(pubsub): add CancellationSharer and HedgingScheduler #9464 - feat(pubsub): add CancellationSharer and HedgingScheduler
4. feat(pubsub): add OpenTelemetry events and logging for hedged publish #9465 - feat(pubsub): add OpenTelemetry events and logging for hedged publish
5. feat(pubsub): integrate publish hedging into MessageQueue #9466 - feat(pubsub): integrate publish hedging into MessageQueue
6. test(pubsub): add unit tests for publish hedging queue integration #9635 - test(pubsub): add unit tests for publish hedging queue integration
7. test(pubsub): add publish hedging system tests #9636 - test(pubsub): add publish hedging system tests (this PR)

---

Summary of Changes in This PR

This layer adds live end-to-end system tests against the Pub/Sub service in handwritten/pubsub/system-test/hedging.test.ts:

• Live End-to-End Delivery:
  • Publishes and receives messages with default hedging: {} enabled (hedgeDelay: 1000ms, maxTokens: 50, refillRatio: 0.1).
  • Publishes and receives a burst of messages across multiple batches with custom HedgingOptions, asserting token bucket replenishment and maxTokens capping.
  • Dynamically enables and updates hedging options via setPublishOptions.
• Live Hedged Backup RPC Trigger & Race Verification:
  • Uses a gRPC client interceptor on the live PublisherClient that holds attempt 0 while allowing the backup (attempt 1) RPC through to the live Pub/Sub service:
    • Hedged attempt wins: Asserts attempt 1 resolves the publish with a valid live messageId, cancels attempt 0 via AbortSignal, delivers the message to the live subscription, and records PubsubEvents.publishStartHedged and PubsubEvents.publishEndHedged on the OpenTelemetry publish_rpc span.
    • Original attempt wins after hedge: Asserts attempt 0 resolves the publish when it finishes just after hedgeDelay, cancels attempt 1, and records PubsubEvents.publishStartHedged + PubsubEvents.publishEnd on the publish_rpc span.
    • Rate limiting when token bucket is at 0: Asserts no backup RPC is dispatched when tokenBucket.tokens < 1 and logs 'Hedging rate limited due to lack of tokens.' on the pubsub:publish-hedged sublogger.
• Validation & Failure Propagation:
  • Verifies mutual exclusivity with enableMessageOrdering: true.
  • Verifies permanent live server error (NOT_FOUND on a non-existent topic) propagates cleanly and cancels any scheduled/in-flight hedged attempts.

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive suite of system tests for publish hedging in handwritten/pubsub/system-test/hedging.test.ts, covering default and custom hedging configurations, dynamic option updates, rate limiting, OpenTelemetry tracing, and message ordering interactions. The review feedback highlights potential race conditions in several tests where mock request interceptors set up both a setTimeout and an AbortSignal listener. If the timeout fires before an abort occurs, a subsequent abort could trigger the callback a second time. The reviewer suggests removing the abort event listeners once the timers fire to prevent duplicate callback invocations.