[multiple] Enable Zuul auto-retry for infrastructure failures - #4107
[multiple] Enable Zuul auto-retry for infrastructure failures#4107imatza-rh wants to merge 1 commit into
Conversation
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 4h 59m 18s |
2574724 to
208e4ce
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 7h 50m 35s |
4331da7 to
2ef9cdc
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 5h 01m 47s |
|
@coderabbitai review |
|
|
Warning Review limit reached
Next review available in: 31 minutes Limit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited) Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
recheck |
| parent: base-crc-cloud | ||
| timeout: 10800 | ||
| attempts: 1 | ||
| pre-timeout: 3600 |
There was a problem hiding this comment.
Hey @imatza-rh, I like the pre-timeout: 3600 on the base-crc multinode job, that's a sensible way to fail fast on a stuck CRC startup. I'm wondering whether the ci-bootstrap multinode jobs need the same treatment, since they also stop pinning attempts: 1. Do you know if their pre-run phase can hit the same hang pattern?
There was a problem hiding this comment.
Good catch - you're right, and it goes deeper than just pre-timeout. The ci-bootstrap jobs' parent chain goes through base-extracted-crc-wo-networks which pins attempts: 1 and runs playbooks/crc/prepare-crc.yaml in pre-run. So removing attempts: 1 from the ci-framework level alone was a no-op - they'd still inherit attempts: 1 from the grandparent.
Fixed in the latest push: both ci-bootstrap jobs now have explicit attempts: 3 (to override the parent chain) and pre-timeout: 3600 (same CRC hang protection as the multinode base job). Updated the PR description and evidence table accordingly.
|
LGTM from me. The retry/timeout tuning looks well reasoned, and the PR description made the parent inheritance and CRC timing easy to follow. I dropped a small nit on whether the ci-bootstrap multinode jobs should get the same pre-timeout treatment, happy to merge it when ready :) |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 03m 59s |
Remove explicit attempts: 1 overrides from Zuul base job definitions, restoring auto-retry for pre-run playbook failures and unreachable-node errors. Add timeout and pre-timeout to molecule CRC jobs and multinode base jobs to address CRC startup timeouts. Fix content-provider-bootc timeout and mark broken whitebox test non-voting. Changes: - zuul.d/base.yaml: remove attempts: 1 from 3 multinode base jobs (inherit attempts: 3 from parent base-crc-cloud). Add attempts: 3 to cifmw-base-crc (override base-simple-crc parent). Add pre-timeout: 3600 to cifmw-podified-multinode-edpm-base-crc (cap pre-run at 60 min so CRC hangs retry instead of consuming the full 3h timeout). - zuul.d/adoption.yaml: change attempts: 1 to attempts: 2 on 4 adoption base jobs (limit retries for long-running jobs). - zuul.d/molecule-base.yaml: add timeout: 3600 (60 min), pre-timeout: 2100 (35 min cap on CRC startup), and attempts: 3 to cifmw-molecule-base-crc. - zuul.d/content_provider.yaml: add timeout: 7200 and attempts: 2 to content-provider-bootc (parent timeout of 2700 is insufficient for bootc image builds, timed out at 47 min in buildset e2b7708f). - zuul.d/project-templates.yaml: set whitebox-neutron-tempest job non-voting on ci-framework (71% failure rate from broken test_igmp_snooping_after_openvswitch_restart, independent of PR changes, uses override-checkout: main). Assisted-By: Claude Code Signed-off-by: Itay Matza <imatza@redhat.com>
2ef9cdc to
a5c8c1d
Compare
|
@coderabbitai review |
PR Summary by QodoEnable Zuul auto-retry and fix CRC/bootc job timeouts
AI Description
Diagram
High-Level Assessment
Files changed (5)
|
Code Review by Qodo🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)
Great, no issues found!Qodo reviewed your code and found no material issues that require reviewTip of the day💡 Did you know, you can commit Qodo's fix in one click with committable suggestions (GitHub & GitLab) |
3 participants
Summary
Remove explicit
attempts: 1overrides from Zuul base job definitions, restoring auto-retry for pre-run playbook failures and unreachable-node errors. Addtimeoutandpre-timeoutto molecule CRC jobs and multinode base jobs to fix the root cause of CRC startup timeouts. Fix content-provider-bootc timeout and mark broken whitebox test non-voting.What changed
zuul.d/base.yamlattempts: 1fromcifmw-podified-multinode-edpm-base-crcand replace withpre-timeout: 3600attempts: 3from parentbase-crc-cloud; cap pre-run at 60 min so CRC hang retries fastzuul.d/base.yamlattempts: 1from 2 ci-bootstrap multinode jobs, addattempts: 3+pre-timeout: 3600base-extracted-crc-wo-networksparent (attempts: 1); cap pre-run for CRC hang protectionzuul.d/base.yamlattempts: 3tocifmw-base-crcbase-simple-crcparent (attempts: 1in RDO config)zuul.d/adoption.yamlattempts: 1toattempts: 2on 4 adoption base jobstimeout: 14400, 1 inherits from parent)zuul.d/molecule-base.yamltimeout: 3600,pre-timeout: 2100,attempts: 3tocifmw-molecule-base-crcbase-simple-crcparentzuul.d/content_provider.yamltimeout: 7200,attempts: 2tocontent-provider-bootce2b7708f)zuul.d/project-templates.yamlvoting: falseon whitebox-neutron-tempest jobtest_igmp_snooping_after_openvswitch_restart; independent of PR changesWhy this is needed
Root cause: CRC startup consumes the default timeout
4 CRC molecule jobs (
ci_multus,ci_nmstate,os_must_gather,set_openstack_containers) inherit a ~30-minute default timeout frombase-simple-crc. CRC startup takes ~24 minutes, leaving only ~6 minutes for tests - causing timeouts on first attempt.Fix:
timeout: 3600(60 min) gives ~36 min for tests after CRC startup.pre-timeout: 2100(35 min) caps pre-run so a CRC hang fails fast and retries instead of consuming the full 60 min. The 1 CRC molecule job with its own override (install_openstack_ca: timeout: 5400) keeps its value.Content-provider-bootc timeout
The bootc content provider inherits
timeout: 2700(45 min) from its parent. Zuultimeoutis per-playbook (each pre-run, run, and post-run playbook getstimeoutseconds individually). Bootc image builds can exceed this per-playbook limit - timed out at 47 min in buildsete2b7708f. Fix:timeout: 7200(120 min per playbook) for headroom +attempts: 2.Multinode base and ci-bootstrap pre-timeout
cifmw-podified-multinode-edpm-base-crchastimeout: 10800(3h) and 5 child jobs. CRC startup in the pre-run phase can hang indefinitely, consuming the full 3h timeout before a retry.pre-timeout: 3600(60 min) caps the pre-run phase - CRC startup (~24 min) fits comfortably, but a hang fails fast and retries instead of wasting hours.The 2 ci-bootstrap multinode jobs (
cifmw-podified-multinode-edpm-ci-bootstrapand-staging) have the same CRC hang risk: their parent chain includesbase-extracted-crc-wo-networkswhich runsplaybooks/crc/prepare-crc.yamlin pre-run. That parent also pinsattempts: 1, so removing the ci-framework-levelattempts: 1alone would be a no-op. Explicitattempts: 3+pre-timeout: 3600overrides the parent and adds CRC hang protection.Whitebox non-voting
whitebox-neutron-tempest-plugin-podified-multinode-edpm-deployment-crc-2comphas a 71% failure rate on ci-framework PRs (5/7 recent runs) but passes 100% on its native project (neutron-tempest-plugin). The failing test (test_igmp_snooping_after_openvswitch_restart) is deterministically broken. The job usesoverride-checkout: main- it does not test PR changes. Marked non-voting until the test is fixed upstream.Retry behavior
Zuul
attemptsretries these conditions, allocating a fresh Nodepool node each time:NOT retried: run-phase failures (test exits non-zero), post-run failures, Nodepool request failures (
NODE_FAILUREbefore any playbook runs).Reference: Zuul job.attempts documentation
Evidence
Verified via live RDO Zuul API and upstream config repo (
components-integration-config):attemptsbase-crc-cloudcomponents-integration-config/zuul.d/_jobs-crc.yaml:183base-extracted-crc-wo-networkscomponents-integration-config/zuul.d/_jobs-crc.yaml:79(grandparent of ci-bootstrap jobs)base-simple-crccomponents-integration-config/zuul.d/_jobs-crc.yaml:30CRC startup timing (from build logs, buildset
d9467e70)cifmw-molecule-ci_multus(1st attempt)cifmw-molecule-ci_nmstate(2nd attempt)cifmw-molecule-install_openstack_catimeout: 5400→ enough roomMotivation
PR #3922 demonstrated the problem: 20+ buildsets over 6 weeks, different jobs failing each time due to infrastructure flakes. Each failure required a full manual
recheckre-running all jobs (2-3 hours). With this change, transient failures auto-retry and molecule CRC jobs get enough timeout for CRC startup + tests.