Skip to content

Add ssh-bastion to interop job and exclude interop tests from FAR standalone jobs - #83179

Open
ugreener wants to merge 1 commit into
openshift:mainfrom
ugreener:feat/interop-ssh-bastion
Open

Add ssh-bastion to interop job and exclude interop tests from FAR standalone jobs#83179
ugreener wants to merge 1 commit into
openshift:mainfrom
ugreener:feat/interop-ssh-bastion

Conversation

@ugreener

@ugreener ugreener commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Problem

The NHC+FAR interop test (RHWA-1035) needs to stop kubelet on a worker node to simulate an unhealthy node for NHC to detect. Without ssh-bastion, the test relies on oc debug which consistently fails: the debug pod dies when kubelet stops, causing oc debug to exit with error. Kubelet then auto-restarts before NHC's 30s unhealthy duration threshold is met, so NHC never triggers remediation.

Additionally, the FAR standalone jobs (e2e-far-aws, e2e-far-weekly-aws) filter with !tier:upgrade but would still match interop tests. Since those jobs don't install NHC or deploy ssh-bastion, interop tests would fail.

Summary

  • Adds ssh-bastion step reference to the e2e-far-nhc-interop-aws presubmit job, matching the existing e2e-nhc-aws job pattern
  • Excludes interop tests from FAR standalone jobs by changing the label filter from !tier:upgrade to !tier:upgrade && !tier:interop

Changes

  • medik8s-system-tests-main__4.22-konflux.yaml: added - ref: ssh-bastion to the interop job's test steps
  • medik8s-system-tests-main__4.22-konflux.yaml: changed ECO_TEST_LABELS from !tier:upgrade to !tier:upgrade && !tier:interop in both e2e-far-aws (presubmit) and e2e-far-weekly-aws (periodic)

Summary by CodeRabbit

  • Updates the Medik8s OpenShift CI configuration for FAR AWS workflows.
  • Excludes interop-tier tests from FAR AWS and weekly FAR AWS ECO_TEST_LABELS.
  • Adds the ssh-bastion step to the FAR/NHC interop AWS workflow after AWS credential setup.
  • This allows CI to stop kubelet directly on worker nodes so NHC can reach its 30-second unhealthy threshold and remediate the node.

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

The Medik8s FAR AWS workflows exclude upgrade and interop tiers from test selection. The FAR/NHC interop AWS workflow runs ssh-bastion after AWS credential setup.

Changes

Medik8s AWS workflow updates

Layer / File(s) Summary
FAR AWS test-label selection
ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml
The FAR AWS and weekly FAR AWS workflows exclude upgrade and interop tiers from ECO_TEST_LABELS.
FAR/NHC interop AWS bastion step
ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml
The workflow runs ssh-bastion after AWS credential setup.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

Suggested reviewers: razo7, beekhof, mshitrit


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name Status Explanation Resolution
Container-Privileges ❌ Error The new ssh-bastion ref deploys a Deployment whose container sets securityContext.privileged: true and uses the privileged SCC. Use a non-privileged bastion, or document and approve the required privileged container for this test.
✅ Passed checks (14 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The patch changes only YAML labels and an ssh-bastion step reference; it adds no Ginkgo test titles or dynamic test-name values.
Test Structure And Quality ✅ Passed PASS: The commit changes only YAML job configuration; it adds no Ginkgo It blocks, assertions, waits, or resource setup. The ssh-bastion order matches e2e-nhc-aws.
Microshift Test Compatibility ✅ Passed The PR changes only one Konflux YAML file; the diff adds no Ginkgo It, Describe, Context, or When tests, so the MicroShift test compatibility check is not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The only changed file is CI YAML; the patch adds no Ginkgo declarations or test source, so SNO compatibility checks are not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed The only changed path is CI YAML; the edits change test labels and add an existing ssh-bastion step, with no deployment, operator, controller, or scheduling constraints.
Ote Binary Stdout Contract ✅ Passed The PR changes only one CI YAML file; it adds workflow steps and label filters, with no OTE binary or process-level suite code changed.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The diff changes only CI YAML workflow labels and step references; it adds no Ginkgo e2e tests or IPv4 or external-connectivity assumptions.
No-Weak-Crypto ✅ Passed The patch only changes ECO_TEST_LABELS and adds the ssh-bastion step; it introduces no weak crypto, custom crypto, or secret-comparison code.
No-Sensitive-Data-In-Logs ✅ Passed The patch only adds an existing ssh-bastion step reference and changes test labels; it adds no logging commands or sensitive data literals.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes both main changes: adding ssh-bastion to the interop job and excluding interop tests from standalone FAR jobs.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ugreener

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 10, 2026
@openshift-ci
openshift-ci Bot requested review from beekhof and razo7 August 10, 2026 13:49

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml`:
- Line 78: Update the ssh-bastion step, identified by the removed ref:
ssh-bastion, so its remote deploy script is fetched from a reviewed immutable
commit or otherwise ensure AWS credentials are unset before invoking it.
Preserve the step’s deployment behavior while preventing mutable master-branch
code from running with credentials.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 60a15060-e89e-4330-b789-6ae9e114ec07

📥 Commits

Reviewing files that changed from the base of the PR and between 9be95ee and d74a111.

📒 Files selected for processing (1)
  • ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml

test:
- chain: medik8s-setup
- ref: medik8s-aws-credentials
- ref: ssh-bastion

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate target file =="
git ls-files | rg 'ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml|ci-operator/step-registry/ssh-bastion|zz_generated_metadata' || true

echo
echo "== target file around referenced step =="
sed -n '1,140p' ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml 2>/dev/null || true

echo
echo "== ssh-bastion command registry files =="
find ci-operator/step-registry -path '*ssh-bastion*' -type f -print | sort
for f in $(find ci-operator/step-registry -path '*ssh-bastion*' -type f | sort); do
  echo "--- $f"
  cat -n "$f"
done

echo
echo "== AWS cred setup references in target config =="
rg -n "AWS|aws|credentials|credential|env|step|ref:|curl|ssh-bastion" ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml || true

Repository: openshift/release

Length of output: 10882


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== all references to ssh-bastion in CI configs =="
rg -n "ssh-bastion|e2e-far-nhc-interop-aws" ci-operator/config || true

echo
echo "== generated metadata references =="
rg -n "medik8s-system-tests-main__4.22-konflux|ssh-bastion|e2e-far-nhc-interop-aws" ci-operator zzz_generated_metadata zz_generated_metadata 2>/dev/null || true

Repository: openshift/release

Length of output: 16773


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== diff stat/status =="
git diff --stat || true
git diff --name-only || true

echo
echo "== deterministic grep for pipe-to-bash pattern in ssh-bastion =="
python3 - <<'PY'
from pathlib import Path
root=Path('ci-operator/step-registry/ssh-bastion')
for p in sorted(root.rglob('*')):
    if p.is_file():
        text=p.read_text(errors='replace')
        for i,line in enumerate(text.splitlines(),1):
            if 'curl' in line or 'bash' in line or 'AWS_ACCESS_KEY_ID' in line or 'AWS_SECRET_ACCESS_KEY' in line:
                print(f"{p}:{i}:{line}")
PY

Repository: openshift/release

Length of output: 698


Pin the remote bastion deployment script before running it with AWS credentials.

medik8s-aws-credentials runs immediately before ssh-bastion, and ssh-bastion-commands.sh executes curl https://raw.githubusercontent.com/eparis/ssh-bastion/master/deploy/deploy.sh | bash -x. The later tested step has no credentials-unset guard, so a changed master branch can execute code in that env. Pin the script to a reviewed commit or remove credentials before invoking it.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml`
at line 78, Update the ssh-bastion step, identified by the removed ref:
ssh-bastion, so its remote deploy script is fetched from a reviewed immutable
commit or otherwise ensure AWS credentials are unset before invoking it.
Preserve the step’s deployment behavior while preventing mutable master-branch
code from running with credentials.

@ugreener
ugreener force-pushed the feat/interop-ssh-bastion branch from d74a111 to b948f5b Compare August 10, 2026 14:11
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@ugreener ugreener changed the title Add ssh-bastion to FAR+NHC interop job for reliable kubelet stop Add ssh-bastion to interop job and exclude interop tests from FAR standalone jobs Aug 10, 2026
@ugreener

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-nhc-interop-aws

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@ugreener: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-aws medik8s/system-tests presubmit Ci-operator config changed
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-nhc-interop-aws medik8s/system-tests presubmit Ci-operator config changed
periodic-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-weekly-aws N/A periodic Ci-operator config changed
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci

openshift-ci Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

@ugreener: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant