Skip to content

codecov: widen project threshold to 0.5% for measurement noise - #212

Merged
blehnen merged 1 commit into
masterfrom
codecov-threshold-tuning
Jul 21, 2026
Merged

codecov: widen project threshold to 0.5% for measurement noise#212
blehnen merged 1 commit into
masterfrom
codecov-threshold-tuning

Conversation

@blehnen

@blehnen blehnen commented Jul 21, 2026

Copy link
Copy Markdown
Owner

Follow-up to #211, now that codecov/project is a required check on master.

Problem

project was set to auto + 0% threshold — any decrease fails. But the measurement is not deterministic: commit 78bf5839 was observed reporting 29,373 hits, then 29,512 (+0.41 points) with no code change. Repeat uploads get unioned, and the integration suite covers slightly different paths run to run.

With project required, that noise blocks PRs for no reason.

Change

threshold: 0%0.5%.

Still far tighter than the original 2% (which is what allowed coverage to drift downward unnoticed, since auto rebaselines every merge), and comfortably below the size of a real regression — the Dashboard.Ui shortfall this gate exists to prevent is ~2.5 points.

Docs updated with the reasoning so the number isn't mistaken for arbitrary slack.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Documentation

    • Updated code coverage policy guidance to require patch coverage checks.
    • Clarified project coverage thresholds and their relationship to measurement variability.
  • Chores

    • Tightened the project coverage ratchet to prevent coverage declines beyond a 0.5% tolerance.

The project status is now a required check, so a 0% threshold is unsafe:
the measurement is not deterministic. Commit 78bf583 reported 29,373
hits and later 29,512 (+0.41 points) with no code change, because repeat
uploads are unioned and the integration suite covers slightly different
paths run to run. At 0% that noise fails PRs for no reason.

0.5% still catches real erosion by a wide margin -- the Dashboard.Ui
shortfall this gate exists to prevent is ~2.5 points -- while staying far
tighter than the original 2%, which is what let coverage drift unnoticed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2ce244c5-39a8-49b8-b536-bff2d1b9f2f4

📥 Commits

Reviewing files that changed from the base of the PR and between 262c807 and 5dcff05.

📒 Files selected for processing (2)
  • codecov.yml
  • docs/code-coverage.md

📝 Walkthrough

Walkthrough

The coverage policy raises the automatic project ratchet threshold from 0% to 0.5%, marks patch coverage as required, and updates comments and documentation to explain the measurement noise floor.

Changes

Coverage gate policy

Layer / File(s) Summary
Coverage threshold and documentation
codecov.yml, docs/code-coverage.md
Codecov uses a 0.5% automatic project threshold, while the coverage documentation marks patch checks as required and explains the updated anti-erosion policy.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Poem

A rabbit checks the coverage line,
“Half a percent” now keeps it fine.
Patch gates stand required and bright,
Noise no longer starts a fight.
Hop, hop—clean builds take flight!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: increasing the Codecov project threshold to 0.5% to account for measurement noise.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.09%. Comparing base (1c5d11a) to head (5dcff05).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #212      +/-   ##
==========================================
+ Coverage   88.06%   88.09%   +0.02%     
==========================================
  Files        1022     1022              
  Lines       33476    33476              
  Branches     2790     2790              
==========================================
+ Hits        29480    29490      +10     
+ Misses       3133     3126       -7     
+ Partials      863      860       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blehnen
blehnen merged commit d678edb into master Jul 21, 2026
8 checks passed
@blehnen
blehnen deleted the codecov-threshold-tuning branch July 21, 2026 20:07
@coderabbitai coderabbitai Bot mentioned this pull request Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant