FINERACT-2810: Add automatic retry for flaky tests - #6418
Open
SamaSVM wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
We've been seeing intermittent test failures in CI that aren't caused by actual code defects, but by external or non-deterministic factors such as timing, network issues, resource contention, or test ordering. These "flaky" failures currently block PRs and require manual re-runs, wasting CI time and reviewer attention.
This PR introduces the official Gradle Test Retry Plugin, which automatically re-runs failed tests up to two times within the same Gradle test task before the build fails.
What's configured
The configuration is added to the shared
test { }block inbuild.gradle:-PtestRetry=trueor-PtestRetry=false.maxRetries = 2— Each failed test gets up to two additional attempts.failOnPassedAfterRetry = false— If a test passes on any retry, the build succeeds.maxFailures = 20— A safety cap prevents retries when 20 or more tests fail in the same round. A large number of simultaneous failures is much more likely to indicate a real regression than isolated flakiness.Test-typed tasks, including unit tests across all Java modules, as well asintegration-tests,oauth2-tests, andtwofactor-tests. It does not apply to the separate Cucumber tasks (fineract-providerandfineract-e2e-tests-runner), since those tasks are notTest-typed.Visibility
Retried tests remain fully visible and traceable. They can be identified through:
build/reports/tests/test/index.html.Therefore, this mechanism does not hide test instability. It only prevents transient failures from unnecessarily blocking CI.
The behavior and configuration are also documented in
CONTRIBUTING.mdandfineract-doc/.../development/gradle.adoc.Why this is safe
The retry mechanism operates entirely within a single Gradle test task, in the same CI job and on the same runner, using the same permissions and token as the rest of the build. This is fundamentally different from re-running an entire GitHub Actions job.
GITHUB_TOKENor permission set is involved. The build's permission scope remains unchanged.org.gradle.test-retry), licensed under Apache 2.0 and used as a standard build-time plugin dependency.Checklist
Please make sure these boxes are checked before submitting your pull request - thanks!
Your assigned reviewer(s) will follow our guidelines for code reviews.