Disallow benchmarks on commit tag-triggered pipelines

[sarahchen6]

What Does This Do

Disallow benchmarks from running on commit tag-triggered pipelines

Motivation

Commit tag-triggered pipelines run for releases only off of the latest master pipeline commit. This commit already has a master GitLab pipeline run with benchmarks, so there's no need to run the same benchmarks for the commit tag-triggered pipelines.

Furthermore, benchmark jobs pull artifacts from a merge-base pipeline to compare benchmarking results against. There is currently no support for finding the merge-base pipeline corresponding to a commit tag-triggered pipeline (see example failed jobs). We could add support; however, we don't need to run the benchmarks on commit tag-triggered pipelines anyway - hence the changes in this PR.

Additional Notes

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f6e730100

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-official]

More details

Configuration change correctly prevents benchmark execution on version tag-triggered pipelines by changing from manual to never trigger rules. No behavioral regressions detected in the core intent. The addition of explicit rules to the linux-java-spring-petclinic-parallel job changes its execution model (previously no explicit rules, now requires manual trigger for PR branches), but this aligns with the stated goal to reduce unnecessary benchmark runs.

Was this helpful? React 👍 or 👎

_{📊 Validated against 6 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 8f6e730 · What is Autotest? · Any feedback? Reach out in #autotest}

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.03 s	13.99 s	[-0.6%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent	12.86 s	13.05 s	[-2.1%; -0.8%] (maybe better)
startup:petclinic:appsec:Agent	16.96 s	16.71 s	[+0.5%; +2.5%] (maybe worse)
startup:petclinic:iast:Agent	16.91 s	16.95 s	[-0.9%; +0.5%] (no difference)
startup:petclinic:profiling:Agent	16.75 s	16.86 s	[-1.6%; +0.4%] (no difference)
startup:petclinic:sca:Agent	17.02 s	16.89 s	[-0.1%; +1.7%] (no difference)
startup:petclinic:tracing:Agent	15.99 s	16.06 s	[-1.3%; +0.5%] (no difference)

Commit: 9cd0b67e · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[dougqh]

Review from Claude

Overview

Disallows JMH benchmark jobs from running on commit-tag-triggered (release) pipelines in .gitlab/java-benchmark-configs.yml, since tag pipelines re-run benchmarks already run on master and currently fail anyway (no merge-base pipeline to diff against).

Correctness

• The anchor changes (.parallel_benchmark_rules, .parallel_startup_benchmark_rules: tag rule manual/allow_failure → never) are correct and minimal.
• The new rules: block added to linux-java-spring-petclinic-parallel (which previously had no rules at all, so ran unconditionally including on tags) correctly closes the same gap — good catch.

Issues / Risks

1. Behavioral inconsistency for linux-java-spring-petclinic-parallel. Instead of reusing *parallel_benchmark_rules like every sibling job, this job gets a hand-rolled rule set that:

   • Is missing the mq-working-branch- / gh-readonly-queue/ conditions from the shared anchor, so unlike every other benchmark job it will not run on merge-queue pipelines. Worth confirming this is intentional.
   • Adds a $CI_PIPELINE_SOURCE == "schedule" → when: always condition not present on any other job in this file (it's an established pattern elsewhere, e.g. .gitlab/benchmarks.yml, .gitlab/exploration-tests.yml, but its introduction here for only one job isn't mentioned in the PR description).
   • Net effect: ~90% duplication of *parallel_benchmark_rules with two undocumented behavioral deltas. Reusing the anchor (or extending it) would fix the tag issue while staying consistent with the rest of the file.

2. Lost documentation comment. # Ensure the tracer artifact publish finishes before the benchmark jobs start. above linux-java-spring-petclinic-parallel was deleted incidentally while adding the new rules: block. Still relevant, worth keeping.

3. Scope creep vs. description. The PR body only discusses tag-pipeline rules; the schedule-pipeline behavior change for this one job isn't mentioned — worth calling out explicitly if intentional.

Test coverage

Pure CI config change; hard to unit test. Worth confirming this was validated against an actual tag pipeline (or GitLab CI lint), since rule-matching bugs are easy to introduce subtly, as the linux-java-spring-petclinic-parallel gap itself shows.

Summary

Core fix is sound. Recommend either reusing the shared anchor for linux-java-spring-petclinic-parallel or explicitly documenting/justifying why it diverges (missing merge-queue support, added schedule support) before merging.

[mhdatie]

The rationale behind the change makes sense to me 👍 Great catch

[sarahchen6]

Behavioral inconsistency for linux-java-spring-petclinic-parallel. Instead of reusing *parallel_benchmark_rules...

These benchmarks never inherited the local parallel_benchmark_rules and instead inherited CI configurations from https://github.com/DataDog/apm-sdks-benchmarks/blob/main/.gitlab/ci-parallel-template.yml#L27 where benchmarks run on schedule and main branches only - what we maintain here.

Lost documentation comment.

This is obvious IMO when reading the code.

Scope creep vs. description.

There are no schedule behavior changes except that the benchmarks are not manually runnable on commit tag-triggered pipelines any more - prior to this PR, they did not run automatically here anyway.

[sarahchen6]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-07-17 20:26:21 UTC ℹ️ Start processing command /merge

---

2026-07-17 20:26:26 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

---

2026-07-17 21:31:12 UTC ℹ️ MergeQueue: This merge request was merged