Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .ci/manual/aro-disconnected/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# ARO disconnected deployment variables.
# Copy this file to .env and edit it. The .env file is ignored by git.
# Used by install-aro-disconnected.sh and create-azure-firewall-rules.sh on the local machine.

BASE_NAME=aro-disconnected
SUFFIX=6
LOCATION=eastus
RESOURCEGROUP=${BASE_NAME}_${SUFFIX}
CLUSTER=${BASE_NAME}_${SUFFIX}
VNET_NAME=${BASE_NAME}-vnet_${SUFFIX}
FIREWALL_NAME=${BASE_NAME}-firewall_${SUFFIX}
FIREWALL_COLLECTION_NAME=azure_ms
FIREWALL_ALLOWED_LIST_BASE="management.azure.com mirror.openshift.com login.microsoftonline.com gcs.prod.monitoring.core.windows.net *.blob.core.windows.net *.servicebus.windows.net *.table.core.windows.net"
FIREWALL_ALLOWED_LIST_INSTALL="*.quay.io sso.redhat.com registry.redhat.io management.azure.com mirror.openshift.com api.openshift.com registry.access.redhat.com"
BASTION_IMAGE=RedHat:RHEL:10_1:latest
SSH_KEY_NAME=azure-disconnected-key_${SUFFIX}
WORKER_COUNT=4
PULL_SECRET_FILE=pull-secret.txt
OPENSHIFT_VERSION=4.19.20
BASTION_NAME=bastion-${SUFFIX}
BASTION_SOURCE_ADDRESS_PREFIX=
8 changes: 8 additions & 0 deletions .ci/manual/aro-disconnected/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
.env
.local/
pull-secret.txt
*_access-information.txt
azure-disconnected-key_*
*.key
*.pem
*.pub
94 changes: 94 additions & 0 deletions .ci/manual/aro-disconnected/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Manual ARO Disconnected Cluster

These scripts provision an Azure Red Hat OpenShift (ARO) cluster in a disconnected Azure network for
manual RHDH testing. They are not connected to Prow.

The workflow is based on the
[ARO disconnected installation guide](https://github.com/redhat-cop/ocp-disconnected-docs/blob/main/AROInstall.md).
Restricting outbound traffic can violate the ARO support policy; use this workflow only where that
trade-off is understood.

## Prerequisites

- Azure CLI with access to the target subscription
- `ssh-keygen` on the local workstation
- An OpenShift pull secret from <https://console.redhat.com/openshift/install/pull-secret>
- A bastion VM image available in the selected Azure region

## Configuration

Run these commands in this directory:

```bash
cp .env.example .env
```

Edit at least `SUFFIX`, `LOCATION`, `OPENSHIFT_VERSION`, and `BASTION_SOURCE_ADDRESS_PREFIX`. Set
the latter to the public IP address or CIDR of the workstation that will SSH to the bastion, for
example `198.51.100.10/32`. Place `pull-secret.txt` in this directory, or set `PULL_SECRET_FILE` to
an absolute path. The configuration contains names and network allow-lists only; it must not contain
registry passwords, tokens, or other secret values.

## Workflow

### 1. Create the ARO cluster and bastion

Run on the local workstation:

```bash
./install-aro-disconnected.sh
```

This creates the resource group, VNet, master and worker subnets, Azure Firewall, routes, private
ARO cluster, and bastion VM. The bastion allows SSH only from `BASTION_SOURCE_ADDRESS_PREFIX`. The
script creates an SSH key pair and writes the SSH command, API server, console URL, and kubeadmin
credentials to a `*_access-information.txt` file with mode `0600`. The file is ignored by git.

The installer firewall rule is removed after ARO creation. Use the next step to add the outbound
rules needed by the workloads you run.

### 2. Add workload firewall rules

Run on the local workstation:

```bash
./create-azure-firewall-rules.sh
```

The script creates the configured rule collection and adds each rule from its indexed rule list in
order. If the collection already exists, it asks before replacing it.

### 3. Set up the bastion

Copy the setup script to the bastion, then SSH to it using the protected access information file.
Replace the placeholders with the values from that file:

```bash
scp -i <SSH_KEY_PATH> setup-bastion.sh azureuser@<BASTION_PUBLIC_IP>:~/setup-bastion.sh
ssh -i <SSH_KEY_PATH> azureuser@<BASTION_PUBLIC_IP>
```

Export the cluster values in the bastion shell without committing or logging them:

```bash
export API_SERVER='https://...'
export KUBEADMIN_PASSWORD='...'
export OPENSHIFT_VERSION='4.19.20'
chmod 700 ~/setup-bastion.sh
~/setup-bastion.sh
```

The script installs version-pinned `oc`, Helm, and `opm` binaries, verifies their published SHA-256
manifests, installs the pinned `umoci` release after checksum verification, enables the OpenShift
internal registry, and resizes the bastion partitions. The default Helm and umoci versions can be
overridden with `HELM_VERSION` and `UMOCI_VERSION`.

## Cleanup

When testing is finished, delete the resource group from the local workstation:

```bash
az group delete --name "$RESOURCEGROUP" --yes --no-wait
```

The command reads `RESOURCEGROUP` from the `.env` file if it is sourced in the current shell.
162 changes: 162 additions & 0 deletions .ci/manual/aro-disconnected/create-azure-firewall-rules.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
#!/usr/bin/env bash
#
# Run on: LOCAL MACHINE (workstation with Azure CLI installed).
# Purpose: Create the application rules required by an ARO disconnected cluster.
# Based on: https://github.com/redhat-cop/ocp-disconnected-docs/blob/main/AROInstall.md
# Requires: An ARO cluster and Azure Firewall created by install-aro-disconnected.sh.
#

set -euo pipefail

SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)

print_status() {
printf '[INFO] %s\n' "$1"
}

print_warning() {
printf '[WARNING] %s\n' "$1"
}

print_error() {
printf '[ERROR] %s\n' "$1" >&2
}

if [[ ! -f "${SCRIPT_DIR}/.env" ]]; then
print_error 'Missing .env. Copy .env.example to .env and edit it first.'
exit 1
fi

# shellcheck disable=SC1091
source "${SCRIPT_DIR}/.env"

if [[ -z "${RESOURCEGROUP:-}" || -z "${FIREWALL_NAME:-}" || -z "${FIREWALL_COLLECTION_NAME:-}" ]]; then
print_error 'RESOURCEGROUP, FIREWALL_NAME, and FIREWALL_COLLECTION_NAME are required.'
exit 1
fi

RESOURCE_GROUP="$RESOURCEGROUP"
RULE_COLLECTION_NAME="$FIREWALL_COLLECTION_NAME"
SOURCE_ADDRESSES=(10.0.0.0/24 10.0.1.0/24)
PROTOCOLS=(Http=80 Https=443)
RULE_NAMES=(
azure
redhat
ms-graph
github
gitlab
pagerduty
quay
okta
auth0
atlassian
atlassian-third-party
)
RULE_TARGETS=(
'management.azure.com mirror.openshift.com login.microsoftonline.com gcs.prod.monitoring.core.windows.net *.blob.core.windows.net *.servicebus.windows.net *.table.core.windows.net'
'*.redhat.com redhat.com redhat.io *.redhat.io'
'graph.microsoft.com'
'*.github.com github.com *.githubusercontent.com'
'gitlab.com *.gitlab.com *.gitlab.io'
'*.pagerduty.com pagerduty.com'
'quay.io *.quay.io'
'*.okta.com *.mtls.okta.com *.oktapreview.com *.mtls.oktapreview.com *.oktacdn.com *.okta-emea.com *.mtls.okta-emea.com *.kerberos.okta.com *.kerberos.okta-emea.com *.kerberos.oktapreview.com *.okta-gov.com *.mtls.okta-gov.com *.okta.mil *.mtls.okta.mil *.awsglobalaccelerator.com'
'auth0.com *.auth0.com'
'*.atlassian.com atlassian.com'
'*.pndsn.com *.cloudfront.net *.wp.com *.gravatar.com *.googleapis.com'
)

check_prerequisites() {
print_status 'Checking prerequisites'
if ! command -v az > /dev/null 2>&1; then
print_error 'Azure CLI is not installed.'
exit 1
fi
if ! az account show > /dev/null 2>&1; then
print_error "Not logged in to Azure. Run 'az login' first."
exit 1
fi
}

check_firewall() {
print_status "Checking firewall: ${FIREWALL_NAME}"
if ! az network firewall show \
--resource-group "$RESOURCE_GROUP" \
--name "$FIREWALL_NAME" > /dev/null; then
print_error "Firewall '${FIREWALL_NAME}' not found in resource group '${RESOURCE_GROUP}'."
exit 1
fi
}

check_existing_rule_collection() {
print_status "Checking rule collection: ${RULE_COLLECTION_NAME}"
if ! az network firewall application-rule collection show \
--resource-group "$RESOURCE_GROUP" \
--firewall-name "$FIREWALL_NAME" \
--collection-name "$RULE_COLLECTION_NAME" > /dev/null 2>&1; then
print_status "Rule collection '${RULE_COLLECTION_NAME}' does not exist."
return
fi

print_warning "Rule collection '${RULE_COLLECTION_NAME}' already exists."
read -r -p 'Overwrite it? (y/N): ' -n 1 REPLY || REPLY=''
printf '\n'
if [[ ! "$REPLY" =~ ^[Yy]$ ]]; then
print_status 'Operation cancelled.'
exit 0
fi

print_status "Removing existing rule collection: ${RULE_COLLECTION_NAME}"
az network firewall application-rule collection delete \
--resource-group "$RESOURCE_GROUP" \
--firewall-name "$FIREWALL_NAME" \
--collection-name "$RULE_COLLECTION_NAME"
}

create_application_rule() {
local rule_name="$1"
local target_definition="$2"
local rule_index="$3"
local -a target_fqdns
local -a command

read -r -a target_fqdns <<< "$target_definition"
command=(
az network firewall application-rule create
--resource-group "$RESOURCE_GROUP"
--firewall-name "$FIREWALL_NAME"
--collection-name "$RULE_COLLECTION_NAME"
--name "$rule_name"
--target-fqdns "${target_fqdns[@]}"
--source-addresses "${SOURCE_ADDRESSES[@]}"
--protocols "${PROTOCOLS[@]}"
)

if [[ "$rule_index" -eq 0 ]]; then
command+=(--action Allow --priority 200)
fi

print_status "Adding firewall rule: ${rule_name}"
"${command[@]}"
}

create_rule_collection() {
print_status "Creating firewall rule collection: ${RULE_COLLECTION_NAME}"
if [[ "${#RULE_NAMES[@]}" -ne "${#RULE_TARGETS[@]}" ]]; then
print_error 'Firewall rule names and target definitions are out of sync.'
exit 1
fi

for rule_index in "${!RULE_NAMES[@]}"; do
create_application_rule \
"${RULE_NAMES[$rule_index]}" \
"${RULE_TARGETS[$rule_index]}" \
"$rule_index"
done
}

check_prerequisites
check_firewall
check_existing_rule_collection
create_rule_collection
print_status "Rule collection '${RULE_COLLECTION_NAME}' created successfully."
Loading
Loading