Skip to content

FINERACT-2810: Add automatic retry for flaky tests - #6418

Open
SamaSVM wants to merge 1 commit into
apache:developfrom
SamaSVM:FINERACT-2810/flaky-test-retry
Open

FINERACT-2810: Add automatic retry for flaky tests#6418
SamaSVM wants to merge 1 commit into
apache:developfrom
SamaSVM:FINERACT-2810/flaky-test-retry

Conversation

@SamaSVM

@SamaSVM SamaSVM commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Description

We've been seeing intermittent test failures in CI that aren't caused by actual code defects, but by external or non-deterministic factors such as timing, network issues, resource contention, or test ordering. These "flaky" failures currently block PRs and require manual re-runs, wasting CI time and reviewer attention.

This PR introduces the official Gradle Test Retry Plugin, which automatically re-runs failed tests up to two times within the same Gradle test task before the build fails.

What's configured

The configuration is added to the shared test { } block in build.gradle:

  • CI-only by default — Retry is enabled only when the CI environment variable is set, which GitHub Actions provides automatically. It remains disabled locally by default, so flaky tests aren't silently masked during development. This can be overridden with -PtestRetry=true or -PtestRetry=false.
  • maxRetries = 2 — Each failed test gets up to two additional attempts.
  • failOnPassedAfterRetry = false — If a test passes on any retry, the build succeeds.
  • maxFailures = 20 — A safety cap prevents retries when 20 or more tests fail in the same round. A large number of simultaneous failures is much more likely to indicate a real regression than isolated flakiness.
  • Scope — Applies to all standard Test-typed tasks, including unit tests across all Java modules, as well as integration-tests, oauth2-tests, and twofactor-tests. It does not apply to the separate Cucumber tasks (fineract-provider and fineract-e2e-tests-runner), since those tasks are not Test-typed.

Visibility

Retried tests remain fully visible and traceable. They can be identified through:

  • Develocity Build Scan — retried tests are explicitly tagged in the Tests tab.
  • Standard HTML test reports — available under build/reports/tests/test/index.html.

Therefore, this mechanism does not hide test instability. It only prevents transient failures from unnecessarily blocking CI.

The behavior and configuration are also documented in CONTRIBUTING.md and fineract-doc/.../development/gradle.adoc.

Why this is safe

The retry mechanism operates entirely within a single Gradle test task, in the same CI job and on the same runner, using the same permissions and token as the rest of the build. This is fundamentally different from re-running an entire GitHub Actions job.

  • No additional executor, runner, or job is created.
  • No new GITHUB_TOKEN or permission set is involved. The build's permission scope remains unchanged.
  • The added dependency is the official Gradle Test Retry Plugin (org.gradle.test-retry), licensed under Apache 2.0 and used as a standard build-time plugin dependency.
  • Retries do not replace fixing flaky tests. Failed tests remain visible through Develocity and HTML reports, allowing chronically flaky tests to be identified and addressed.

Checklist

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Write the commit message as per our guidelines
  • Acknowledge that we will not review PRs that are not passing the build ("green") - it is your responsibility to get a proposed PR to pass the build, not primarily the project's maintainers.
  • Create/update unit or integration tests for verifying the changes made.
  • Follow our coding conventions.
  • Add required Swagger annotation and update API documentation at fineract-provider/src/main/resources/static/legacy-docs/apiLive.htm with details of any API changes
  • This PR must not be a "code dump". Large changes can be made in a branch, with assistance. Ask for help on the developer mailing list.
  • If merging this PR resolves a JIRA issue, I will mark that issue as resolved and set "Fix Version/s" appropriately.

Your assigned reviewer(s) will follow our guidelines for code reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant