Repository navigation
Github CI is not useful for external contributors #16212
Description
As I am not an expert with Github CI (I am more familiar with Gitlab), parts of these might be Github's fault, but nonetheless - the CI is not doing any favor to external contributors. IMHO, it should support them better, to motivate more outside contributions
-
Email flood
- After a lot of quiet on the PR, I am suddenly bombarded with ~13 emails about failing jobs. Especially if I have multiple PRs open, it is quite hard to keep track of what is affected
- Any chance of having one email per PR instead, reporting the CI failure once, per PR?
-
-
It is unclear when the CI runs and where
- We should explain in the CONTRIBUTING.md that most jobs (but not all, I think?) run only when a maintainer starts them
- There is also a Drone CI running somewhere?
- It is unclear where I see my newly added tests actually running
- I tried to get to the bottom of this here, but didn't get far: Minor improvements for automated testing #15424 (comment)
-
Some jobs simply fail by design, when it is on a fork
- Example: https://github.com/nextcloud/android/actions/runs/20276837329/job/59030849217?pr=16145
- this leads to an unnecessary and wrongful impression that there is something wrong. Rather, any CI failure should directly map to an action item - but in these cases, there is nothing you can do.
-
It is unclear what checks should be run locally, to ensure a smooth CI experience
- E.g.
gradlew lintis failing the CI, but it is not explained anywhere that you should run it. The pre-commit hook also doesn't execute it (probably because it is too expensive to run every time) - Example: https://github.com/nextcloud/android/actions/runs/20276837368/job/59030849289?pr=16145
- E.g.
-
Details of a CI failure are hard to understand
- For example no full lint report downloadable on https://github.com/nextcloud/android/actions/runs/20276837368/job/59030849289?pr=16145
- For example no access to details for screenshot test failures: https://github.com/nextcloud/android/actions/runs/20276837409/job/59030849251?pr=16145
-
Said linter job
just alwaysfails, unrelated to the changes in the PR- Example: https://github.com/nextcloud/android/actions/runs/20276837368/job/59030849289?pr=16145 fails due to issues in files the PR didn't even touch. So it is unclear, if and what I am supposed to do.
-
The
detectWrongSettingsjob is destined to fail- Because the maintainer triggering of the CI job is happening weeks later than the original PR, at time of triggering the PR is almost always out-of-date, so it doesn't use one of the latest 10 android-library commits.
- This issue is fixable by the contributor, by rebasing, but it does cause additional friction as well (yet another failure, yet another alert, ...)
Activity
[-]Github CI is not useful for contributors[/-][+]Github CI is not useful for external contributors[/+]AndyScherzinger commentedon Dec 28, 2025
CC @tobiasKaminsky @alperozturk96
AndyScherzinger commentedon Dec 28, 2025
Added the devs and teamlead.
As a quick fix for you, since you already contributed, sent you an invite for the Nextcloud org on Github, so you can don't need to work on a fork.
I agree that docs can be improve. Several CI checks terminate early when executing on a fork for security reasons to protect the original repo.
PhilLab commentedon Jun 6, 2026
@tobiasKaminsky @alperozturk96 next to the initially stated problems, the test pipeline is always red for many many months now. So the CI is not useful for both external and internal contributors:
Is there any plan on addressing those now useless tests?
cc @AndyScherzinger
hannesa2 commentedon Jun 6, 2026
I agree to @PhilLab that always failing jobs makes no sense at all and just makes every pull request faulty.
If nobody takes care about it, then it makes even no sense at all ! --> just pointless overhead
What's about to get rid of them complete and (maybe at the same time) make a new pull request to re-introduce that again.
There the CI can be fixed separately.
AndyScherzinger commentedon Jun 6, 2026
As mentioned in #16212 (comment)
@tobiasKaminsky @alperozturk96 would need to comment / look into it. I personally don't have the time at the moment.
PhilLab commentedon Jun 26, 2026
@tobiasKaminsky can you take a look at this? This repository effectively has no kind of instrumented test anymore. Neither screenshot tests nor drone tests do what they are supposed to do.
tobiasKaminsky commentedon Jun 29, 2026
We are in transition to github action runners (#17038), that is why I did not spent that much love into Drone recently.
I will make this working again within this week.
Screenshot tests are a bit harder.
For me it seems that shot (https://github.com/pedrovgs/Shot) is a bit abandonded.
I tested it at some point in past any only got it working again with downgrading junit:
#16578
tobiasKaminsky commentedon Jul 1, 2026
GARM is now merged and works "reliable", as in: we have 14 failing tests.
But I already merged it, so that we can work on fixing the tests, while knowing that we do not add more regressions.
tobiasKaminsky commentedon Jul 1, 2026
PhilLab commentedon Jul 5, 2026
@tobiasKaminsky oh, this is great news! Good to know that Drone is phased out and the pipeline is now more unified!
tobiasKaminsky commentedon Jul 6, 2026
We have now also this PR #17268, where we try to get a green CI.
Then we need to check/discuss screenshots.
PhilLab commentedon Sep 23, 2026
With the tests now being in a stable state, the remaining issues here are actually rooted in Github's UX itself. Closing the ticket, as the leftovers are not actionable for the team.