Repository navigation
Conversation
System UI can briefly cover the app during capture, for example the Apple Intelligence notification iOS shows shortly after boot. A covered screenshot published as the baseline from `main` fails every PR based on that commit, and three baselines on `main` were spoiled this way or by flows racing the UI. After capture, `capture.js` now finds the nearest baseline over the public bucket URL and reruns, once and after a short delay, only the flows whose screenshots differ from it or have no baseline: - A changed screenshot is replaced by its retake only if the retake matches the baseline, so a real change still fails. - A new screenshot is unstable unless both captures are identical. The manifest marks it, the report says so, and compare doesn't publish it as the baseline. Screenshots that already matched are never retaken. The step is skipped without a baseline or `VISUAL_PUBLIC_BASE_URL`. Changelog: [Internal]
meta-cla
Bot
added
the
CLA Signed
facebook-github-tools
Bot
added
the
Shared with Meta
cortinico
approved these changes
Oct 26, 2026
| if (baseline == null) { | ||
| return { | ||
| unstable: [], | ||
| lines: [`No baseline for \`${environmentKey}\`, so nothing was retaken.`], |
Contributor
There was a problem hiding this comment.
Shouldn't we retake pictures even if the baseline is empty? That would make sure we don't accidentally pollute the baseline with flaky screenshots
Collaborator
Author
There was a problem hiding this comment.
Yeah agreed - tbh I thought this was covered already. Done now.
Contributor
|
@robhogan CI is red |
Collaborator
Author
I think that's unrelated. I've incorporated a fix for open-example.yml into this branch to test CI but it probably ought to be fixed separately. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
System UI can briefly cover the app during visual capture. On iOS 26 that's the Apple Intelligence notification shown shortly after boot, which got published as a baseline on
mainand failed #58945 with a diff it had nothing to do with. Two more baselines were spoiled by flows racing the UI. I've deleted all three by hand.This retakes, once, any screenshot that differs from the nearest baseline or has none, before upload:
Screenshots that already match aren't retaken, so a clean run costs nothing extra.
Changelog: [Internal]
Test plan:
New unit tests. Ran the retake step against the real bucket with Maestro stubbed, with one spoiled and two new screenshots, and it replaced, kept and flagged them as above. First CI run is this PR's own Test All.