diff --git a/docs/architecture.md b/docs/architecture.md index 003818b6..bfbe6480 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -186,7 +186,7 @@ hole from the monitoring VLAN into the monitored one. | --- | --- | --- | --- | | `prometheus` (10.0.99.20) | 🔴 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, blackbox-exporter, speedtest-tracker, docker-socket-proxy, Alloy | | `Saruman` (10.0.30.20) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, hosting twenty-one guests: `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)); `phoenix`, built 2026-09-20 ([#436](https://github.com/Gerrrt/HomeLab/issues/436)); `odin`, built 2026-09-27 for the security tooling (ADR-0030); `fenrir`, built 2026-09-30 as the Zeek sensor ([#437](https://github.com/Gerrrt/HomeLab/issues/437)); `golem`, built 2026-10-03 as the backup server ([#485](https://github.com/Gerrrt/HomeLab/issues/485)); `eden`, built 2026-10-07 as the BloodHound guest ([#451](https://github.com/Gerrrt/HomeLab/issues/451)), off between sessions; `garuda`, built 2026-10-10 as the analyst workstation ([#921](https://github.com/Gerrrt/HomeLab/issues/921)); and ADR-0029's six-machine domain, built by hand 2026-09-24 to 2026-09-25 ([#414](https://github.com/Gerrrt/HomeLab/issues/414)), of which `carbuncle` and `siren` run per session; and all eight of [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)'s on-demand dotfiles test VMs, `dot-debian`, `dot-fedora` and `dot-windows`, built 2026-10-08, and `dot-opensuse`, `dot-arch`, `dot-alpine`, `dot-gentoo` and `dot-nixos`, built 2026-10-09 ([#920](https://github.com/Gerrrt/HomeLab/issues/920)). Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest | -| `gandalf` (10.0.30.30) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, the range host, installed 2026-11-07 ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). No guests yet. Guest disks go on the ZFS pool `range`, targets go on the portless `vmbr1`, and it is off between sessions ([ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md), [ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)) | +| `gandalf` (10.0.30.30) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, the range host, installed 2026-11-07 ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). One guest, the attack VM `ragnarok` (VMID 130), made 2026-11-09 by [`scripts/make-attack-vm.sh`](../scripts/make-attack-vm.sh) from template 902 and dual-homed: a DHCP lease on VLAN 30, and `172.30.30.10` on `vmbr1`. No targets yet. Guest disks go on the ZFS pool `range`, targets go on the portless `vmbr1`, and it is off between sessions ([ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md), [ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)) | | `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy and its Docker socket proxy, and Caddy as the ingest proxy in front of the first two ([#834](https://github.com/Gerrrt/HomeLab/issues/834)) — the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) | | `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. Built 2026-09-27 by [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), the stack authored and CI-validated ahead of the guest the way `stacks/lab` was ahead of `alexander` ([#266](https://github.com/Gerrrt/HomeLab/issues/266), [#267](https://github.com/Gerrrt/HomeLab/issues/267)); the six domain machines enrol as agents by GPO, which is the day those two issues close. Its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) | | `fenrir` (10.0.30.90) | 🟢 30 | [`stacks/sensor`](../stacks/sensor) | Zeek on a `tc` mirror of `Saruman`'s lab bridge, so that the domain's east-west traffic, which crosses no router and so never reaches Suricata on `morpheus`, is observed at all ([#437](https://github.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)). A guest on `Saruman`, VMID 190, with a second NIC alone on the port-less `vmbr1` that receives only the mirror's copies. Its logs go to `alexander`, never to VLAN 99; whether the mirror is delivering is answered on the hypervisor by `homelab_zeek_mirror_active`. Built 2026-09-30 by [`build-the-sensor-guest.md`](runbooks/build-the-sensor-guest.md), and proved by rebooting `Saruman` with the mirror's timer disabled: the gauge fell to 0 and `ZeekMirrorInactive` fired. Alloy agent (Docker) | diff --git a/docs/changelog.md b/docs/changelog.md index 0eccd1fc..aa435dbc 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -19,6 +19,39 @@ docstring gives: it is a record, not a claim about now. ## 2026-11-09 +- **[#421](https://github.com/Gerrrt/HomeLab/issues/421), + [#790](https://github.com/Gerrrt/HomeLab/issues/790): `tpl-kali` (902) is + built on `gandalf`, and the attack VM `ragnarok` is up, + `build-the-playground.md` §5.** + - **Access.** `gandalf` is not clustered, so it got its own `phoenix@pve` + user, `PhoenixBuilder` and `PhoenixIsoCleanup` roles and + `phoenix@pve!builder` token. The role is `build-the-jumpbox.md` §4's plus + `VM.GuestAgent.Audit`. Its ACLs cover `range`, `local`, and both + `vmbr0` and `vmbr1`. The two builds, the two smoke tests and the attack + VM asked for nothing more. `phoenix` trusts `gandalf`'s own cluster CA + and keeps the token in `gandalf.env`, apart from `phoenix.env`. + - **The builds.** The ISO is the 2026.2 installer already pinned for 910, + fetched onto `gandalf`'s `local` and checked against that hash. The + first build stopped at validation, because the variables file refuses + an empty `PKR_VAR_build_password` even for Kali. Sourcing `phoenix.env` + under `gandalf.env` fixed it, and §5 says so now. The build took + 10 min 33 s and the `-force` rebuild 10 min 41 s. `packer-smoke.sh 902` + passed after each, which is #790's acceptance. 910's boot-order and 20 s + fixes were not needed here. `ufw` is inactive on `phoenix`, so no port + was opened. + - **The attack VM.** `ragnarok`, VMID 130, is a full clone on `range` with + 4 cores and 8 GiB, made by the new `scripts/make-attack-vm.sh`. `net0` is + on `vmbr0` by DHCP and took `10.0.30.111`. `net1` is on `vmbr1` at + `172.30.30.10/24` with no gateway. Both have the firewall flag off. The + script wrote `99-no-forward.conf` and ran §8's third check: both + sysctls `0`, no NAT in nft or iptables, and one default route, via + `10.0.30.1`. + - **#1196's check.** On the fresh clone it read `running`, 0 failed units + and 0 accounts expiring at the epoch, the check + [#1196](https://github.com/Gerrrt/HomeLab/issues/1196) owed 902. + - **What is not done.** Sliver is not installed, there are no targets + (§6), and §8's first, second and fourth checks and §9 have not run. + - **[#421](https://github.com/Gerrrt/HomeLab/issues/421): `gandalf`'s management plane is closed, `build-the-playground.md` §4.** The firewall had been off since the install, so ADR-0014's rules did not exist yet and diff --git a/docs/diagrams/current/network.png b/docs/diagrams/current/network.png index 4b77ac61..15b06ea2 100644 Binary files a/docs/diagrams/current/network.png and b/docs/diagrams/current/network.png differ diff --git a/docs/diagrams/current/network.svg b/docs/diagrams/current/network.svg index 8fcb3c30..db99d622 100644 --- a/docs/diagrams/current/network.svg +++ b/docs/diagrams/current/network.svg @@ -658,7 +658,7 @@ gandalf -Proxmox VE · range host, no guests yet +Proxmox VE · range host · attack VM ragnarok .30 diff --git a/docs/network.md b/docs/network.md index 5c9ef7f8..ad5c9288 100644 --- a/docs/network.md +++ b/docs/network.md @@ -713,7 +713,8 @@ Where things get broken on purpose. away before the attack VM starts taking leases from the same pool. - A second hypervisor, `gandalf`, carries the attack tooling and the deliberately-vulnerable targets. It is installed (Proxmox VE 9, - 2026-11-07) and has no guests yet. It sits on this segment single-homed, on + 2026-11-07) and has one guest, the attack VM `ragnarok` (VMID 130, made + 2026-11-09). It sits on this segment single-homed, on `neo`'s untagged `ether20`, at `10.0.30.30`: a static below `.100`, with a reservation. Its Intel AMT, which would be the name `ifrit`, is disabled ([ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)). @@ -726,7 +727,7 @@ Where things get broken on purpose. is the third private block in the house and the only one that is not routed anywhere, which is what makes a `172.30.30.x` source in a firewall block a leak reporting itself. The attack VM takes a lease from the pool like any - other guest. The build is + other guest, with no reservation, and holds `172.30.30.10` on the bridge. The build is [`build-the-playground.md`](runbooks/build-the-playground.md). - **The WireGuard peers live on `172.31.0.0/24`, and it is routed rather than diff --git a/docs/roadmap.md b/docs/roadmap.md index 90f1a242..604b19bc 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -240,8 +240,8 @@ Closes on BloodHound running where nothing attacks it. [ADR-0074](adr/0074-build-the-lab-templates-with-packer-from-phoenix.md): 901 Ubuntu, 911 Windows 11 and 912 Server 2025 built from `smaug-iso`, each clone passing `scripts/packer-smoke.sh`, SSH included, - → [runbook](runbooks/build-the-lab-templates.md); Kali's - template waits for its build on `gandalf`, [#790](https://github.com/Gerrrt/HomeLab/issues/790)) → + → [runbook](runbooks/build-the-lab-templates.md); Kali's 902 + built on `gandalf` 2026-11-09, [#790](https://github.com/Gerrrt/HomeLab/issues/790)) → [#445](https://github.com/Gerrrt/HomeLab/issues/445) OpenTofu (done 2026-10-03, [ADR-0076](adr/0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md): @@ -286,9 +286,11 @@ Gated on the domain, on a household observation, or on something to publish. - **[#421](https://github.com/Gerrrt/HomeLab/issues/421) Build the range on `gandalf`.** Gated on #414 and the SOC: an attack VM pointed at an - uninstrumented estate teaches nothing. Both are met. The runbook's §1–§4 - are behind it: the host, its bridges, `Saruman` at `.20` and the closed - management plane. Next is §5, the attack VM, then §6–§9, in order. + uninstrumented estate teaches nothing. Both are met. The runbook's §1–§5 + are behind it: the host, its bridges, `Saruman` at `.20`, the closed + management plane, and the attack VM `ragnarok` on template 902. Next is §6, + the targets, then §7–§9, in order. Sliver goes on `ragnarok` before the + first exercise. The name is [ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)'s. → [runbook](runbooks/build-the-playground.md) diff --git a/docs/runbooks/build-the-analyst-workstation.md b/docs/runbooks/build-the-analyst-workstation.md index 7eab9b8a..b8bb7de2 100644 --- a/docs/runbooks/build-the-analyst-workstation.md +++ b/docs/runbooks/build-the-analyst-workstation.md @@ -553,6 +553,9 @@ the console password. - **Still owed:** the same check on 902 (gandalf's Kali, same preseed) when it is next built — [#1196](https://github.com/Gerrrt/HomeLab/issues/1196), tied to [#790](https://github.com/Gerrrt/HomeLab/issues/790). + - **Paid 2026-11-09:** 902 was built twice on `gandalf`, and its clone + `ragnarok` read `running`, 0 failed units and 0 accounts at the epoch + ([`build-the-lab-templates.md`](build-the-lab-templates.md) §5). - **§12, phase 2, 2026-10-10** ([#1163](https://github.com/Gerrrt/HomeLab/pull/1163), [ADR-0092](../adr/0092-enrol-garuda-in-odins-soc-with-debian-packages-staged-on-odin.md)). diff --git a/docs/runbooks/build-the-lab-templates.md b/docs/runbooks/build-the-lab-templates.md index 241906e4..31dc0df1 100644 --- a/docs/runbooks/build-the-lab-templates.md +++ b/docs/runbooks/build-the-lab-templates.md @@ -1,8 +1,8 @@ # Runbook: Build the lab's VM templates with Packer, from `phoenix` **Target:** templates 901, 903–912 on `Saruman`, built from -`phoenix` through the Proxmox API. 902, Kali, is written here and built on -`gandalf`, the range host, installed 2026-11-07 and not yet built on. 910 is the same Kali build on `Saruman`, for +`phoenix` through the Proxmox API. 902, Kali, is built on `gandalf`, the +range host, through `gandalf`'s own API (§5, first built 2026-11-09). 910 is the same Kali build on `Saruman`, for `garuda` (§5b, [ADR-0091](../adr/0091-put-a-kali-purple-analyst-workstation-on-saruman.md)). 903 (Debian), 904 (Fedora), 905 (openSUSE Tumbleweed), 906 (Arch), 907 (Alpine), 908 (Gentoo) and 909 (NixOS) are the dotfiles OS layers' templates @@ -488,34 +488,93 @@ node. The preseed is served over Packer's HTTP server on port 8800 of `phoenix`, because Debian's installer reads one from a URL or its own medium, not from a second disc. -1. Admit the guest to that port on `phoenix` for the length of the build. If - `ufw` is active, run `sudo ufw allow from 10.0.30.0/24 to any port 8800 proto tcp`, - and delete the rule afterwards. Debian's build (§4) needs this step and - no other here. -2. Grant `PhoenixBuilder` on `gandalf`'s storage and bridge, and trust - `gandalf`'s CA on `phoenix`, as §2 did for `Saruman`. Unless `gandalf` joins - `Saruman` in a cluster, it is its own API. Point `PROXMOX_URL` at it for - this build, and admit `phoenix` to its 8006 the way ADR-0043 did on - `Saruman`. -3. Build, naming the ISO actually on `gandalf` and **its** storage. - `disk_storage` defaults to `large_data`, which is `Saruman`'s pool, and - the disk creation fails without the override: +> **Ran 2026-11-09** ([#790](https://github.com/Gerrrt/HomeLab/issues/790), +> [#421](https://github.com/Gerrrt/HomeLab/issues/421)). Steps 2–5 below are +> what ran. The first build took 10 min 33 s and the `-force` rebuild +> 10 min 41 s. `packer-smoke.sh 902` passed after each. `gandalf`'s 10 s +> `boot_wait` was enough with no boot order, unlike 910 on `Saruman`. The +> clone `ragnarok` then read `running` with 0 failed units and no account +> expiring at the epoch, the check +> [#1196](https://github.com/Gerrrt/HomeLab/issues/1196) owed 902. + +1. **Port 8800 on `phoenix`.** If `ufw` is active, run + `sudo ufw allow from 10.0.30.0/24 to any port 8800 proto tcp` for the build + and delete the rule afterwards. On 2026-11-09 `ufw` was inactive on + `phoenix`, so there was nothing to open. Debian's build (§4) needs this + step and no other here. +2. **The user, the roles and the token, on `gandalf`, as root.** `gandalf` is + not clustered with `Saruman` + ([ADR-0039](../adr/0039-decline-proxmox-clustering-while-ifrit-is-the-range.md)), + so it is its own API with its own users. The role is + [`build-the-jumpbox.md`](build-the-jumpbox.md) §4's plus §2's + `VM.GuestAgent.Audit`. The storage is `range`, and the bridges are both + `vmbr0` and `vmbr1`, because the attack VM's second NIC is on the range + bridge: ```bash - packer build -only='kali.proxmox-iso.kali' \ - -var kali_iso_file=local:iso/ \ - -var disk_storage= \ - packer/ + pveum role add PhoenixBuilder --privs "VM.Allocate VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Cloudinit VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Console VM.PowerMgmt VM.Audit VM.Snapshot VM.GuestAgent.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use" + pveum role add PhoenixIsoCleanup --privs "Datastore.Allocate" + pveum user add phoenix@pve --comment "deployment host, ADR-0043" + pveum acl modify /vms --users phoenix@pve --roles PhoenixBuilder + pveum acl modify /storage/local --users phoenix@pve --roles PhoenixBuilder,PhoenixIsoCleanup + pveum acl modify /storage/range --users phoenix@pve --roles PhoenixBuilder + pveum acl modify /sdn/zones/localnetwork/vmbr0 --users phoenix@pve --roles PhoenixBuilder + pveum acl modify /sdn/zones/localnetwork/vmbr1 --users phoenix@pve --roles PhoenixBuilder + pveum acl modify /nodes/gandalf --users phoenix@pve --roles PVEAuditor + pveum acl list | grep phoenix + pveum user token add phoenix@pve builder --privsep 0 ``` -4. Prove it on `gandalf`, as §6 does on `Saruman`: + Paste this into the node's **Shell** in the web UI, or into an SSH session + from Hicks. On 2026-11-09 a first paste went into a zsh on `phoenix`, + where `pveum` does not exist. The paste handling there also turned every + space after the first line into a literal `\s`. Nothing ran, and nothing + needed undoing. No privilege beyond these was asked for by the two + builds, the two smoke tests or `scripts/make-attack-vm.sh`. + + **The installer goes on `gandalf`'s own `local`.** It is the same 2026.2 + ISO as 910's, checked against the hash + [`collect-iso-store-state.sh`](../../scripts/collect-iso-store-state.sh) + pins: + + ```bash + cd /var/lib/vz/template/iso && wget -nc https://cdimage.kali.org/kali-2026.2/kali-linux-2026.2-installer-amd64.iso && echo "6dbefacc95e3b556c19c48e8bae39b8b505e2d3a1aba0bfb7ab62b036c3d2ba3 kali-linux-2026.2-installer-amd64.iso" | sha256sum -c - + ``` + +3. **`phoenix` trusts `gandalf`'s CA and holds its token apart.** `gandalf` + serves its own `PVE Cluster Manager CA` certificate, naming `10.0.30.30` + (read 2026-11-09, so it needs no fix of the kind §2 records for + `Saruman`). Relay the CA from Hicks: + `ssh gandalf cat /etc/pve/pve-root-ca.pem | ssh phoenix 'cat > ~/gandalf-pve-root-ca.crt'`. + Install it on `phoenix` as + `/usr/local/share/ca-certificates/gandalf-pve-root-ca.crt`, as §2 does + for `Saruman`'s. The token goes in **`~/.config/proxmox/gandalf.env`**, + mode 600, with its own `PROXMOX_URL=https://10.0.30.30:8006/api2/json`, + `PROXMOX_TOKEN_ID=phoenix@pve!builder` and `PROXMOX_TOKEN_SECRET`. It is + not added to `phoenix.env`, so a build for `Saruman` can never be pointed + at `gandalf` by accident, or the reverse. Prove it with §2's `curl …/version` + line, with `gandalf.env` sourced in place of `phoenix.env`. +4. **Build.** Source `phoenix.env` first and `gandalf.env` over it. The + variables file refuses an empty `PKR_VAR_build_password` for every source, + Kali included, and that password lives in `phoenix.env`. The `PROXMOX_*` + lines in `gandalf.env` then win. `disk_storage` defaults to `large_data`, + which is `Saruman`'s pool, so name `range`: + + ```bash + set -a; . ~/.config/proxmox/phoenix.env; . ~/.config/proxmox/gandalf.env; set +a + packer init packer/ + packer build -force -only='kali.proxmox-iso.kali' -var disk_storage=range packer/ + ``` + + `kali_iso_file` defaults to the ISO from step 2. +5. **Prove it**, with `gandalf.env` sourced: ```bash PROXMOX_NODE=gandalf scripts/packer-smoke.sh 902 ``` - Then run §8's second `-force` build and smoke test, which is #790's - acceptance. + Then run the build and the smoke test a second time, which is #790's + acceptance and §8's rule. ## 5b. Kali on `Saruman`, for `garuda` diff --git a/docs/runbooks/build-the-playground.md b/docs/runbooks/build-the-playground.md index 8f97fc9b..dbef6fb0 100644 --- a/docs/runbooks/build-the-playground.md +++ b/docs/runbooks/build-the-playground.md @@ -43,7 +43,7 @@ decision and on this runbook. The machine is ready too. `gandalf`, a refurbished ThinkCentre M80q, was bought on 2026-10-05 and has its 32 GB and its SATA range disk ([ADR-0096](../adr/0096-fit-ifrits-range-disk-as-a-sata-ssd-in-the-tinys-bay.md)) -fitted. **§1, §2 and §3 were done on 2026-11-07, and §4 on 2026-11-09** +fitted. **§1, §2 and §3 were done on 2026-11-07, and §4 and §5 on 2026-11-09** ([#421](https://github.com/Gerrrt/HomeLab/issues/421)), and the parts as fitted are in [`hardware.md`](../hardware.md). The node is named `gandalf`, and `ifrit` names its disabled Intel AMT @@ -588,6 +588,30 @@ has never fired is a rule nobody knows can. One VM, and it is the only guest with a leg on both bridges. +> **Made on `gandalf` 2026-11-09** ([#421](https://github.com/Gerrrt/HomeLab/issues/421)): +> **`ragnarok`, VMID 130**, a full clone of `tpl-kali` (902). 902 was built +> on `gandalf` from `phoenix` the same night +> ([`build-the-lab-templates.md`](build-the-lab-templates.md) §5, +> [#790](https://github.com/Gerrrt/HomeLab/issues/790)). Everything below is +> one command on `phoenix`: +> +> ```bash +> set -a; . ~/.config/proxmox/gandalf.env; set +a +> scripts/make-attack-vm.sh +> ``` +> +> [`make-attack-vm.sh`](../../scripts/make-attack-vm.sh) clones onto `range` and sets +> 4 cores and 8 GiB. ADR-0017's "roughly half" of 32 GB counted BloodHound, +> which [ADR-0081](../adr/0081-run-bloodhound-ce-on-a-saruman-guest.md) moved +> to `eden`. It gives `net0` on `vmbr0` DHCP and `net1` on `vmbr1` +> `172.30.30.10/24` with no gateway, both with the guest firewall flag off. It +> reads the config back, then writes `99-no-forward.conf` over SSH and runs +> §8's third check. It refuses to run if VMID 130 exists, because the range is +> rebuilt, not repaired. On its first run, `ragnarok` took `10.0.30.111`. Both +> forwarding sysctls read `0`, neither table had NAT, and the one default +> route was via `10.0.30.1` on `eth0`. `eth1` held `172.30.30.10/24` and +> nothing else. Its cloud-init user is `kali`, with `phoenix`'s key. + - **Its disk on `range`**, the SATA SSD's pool from §1, like every guest here. - **`net0` on `vmbr0`** — ImaginationLAN, DHCP from the pool. Nothing points at this address, so it does not get a reservation. @@ -619,6 +643,16 @@ narrow `50 → 30` without the list: Proxmox on `10.0.30.30:8006`, the C2 operator's interface on the attack VM, and whatever the lab's Grafana and RDP into the estate already need on `Saruman`. +As of 2026-11-09 the list is: + +- Proxmox on `10.0.30.30:8006` and SSH on `10.0.30.30:22`, ADR-0014's rules. +- SSH to `ragnarok` on its lease, port 22. + +Sliver ([ADR-0095](../adr/0095-run-sliver-as-the-range-c2-on-the-attack-vm.md)) +is not installed yet. Its operator port joins the list when it is. The lease +is not reserved, so a rule written for #228 names the segment, not the +address. + --- ## 6. The targets diff --git a/packer/README.md b/packer/README.md index 0cdb4116..d155a254 100644 --- a/packer/README.md +++ b/packer/README.md @@ -5,14 +5,14 @@ [![Packer](https://img.shields.io/badge/Packer-02A8EF?style=plastic&logo=packer&logoColor=white)](https://developer.hashicorp.com/packer) [![Proxmox VE](https://img.shields.io/badge/Proxmox%20VE-E57000?style=plastic&logo=proxmox&logoColor=white)](https://www.proxmox.com/en/proxmox-virtual-environment) -The lab's VM templates, built from `phoenix` through `Saruman`'s API -([ADR-0074]). How to run them is [`build-the-lab-templates.md`][runbook]; this +The lab's VM templates, built from `phoenix` through `Saruman`'s API, and +902 through `gandalf`'s ([ADR-0074]). How to run them is [`build-the-lab-templates.md`][runbook]; this page is the map. | VMID | Template | Source | Installer driven by | Node, and status | | --- | --- | --- | --- | --- | | 901 | `tpl-ubuntu-2604` | `ubuntu.pkr.hcl` | autoinstall, `cidata` disc | `Saruman`, built and usable | -| 902 | `tpl-kali` | `kali.pkr.hcl` | Debian preseed, Packer HTTP | `gandalf`, installed 2026-11-07; the build waits on build-the-lab-templates.md §5 ([#790](https://github.com/Gerrrt/HomeLab/issues/790)) | +| 902 | `tpl-kali` | `kali.pkr.hcl` | Debian preseed, Packer HTTP | `gandalf`, built twice and smoke-tested, 2026-11-09, through `gandalf`'s own API ([#790](https://github.com/Gerrrt/HomeLab/issues/790)); the attack VM's base | | 903 | `tpl-debian-13` | `debian.pkr.hcl` | Debian preseed, Packer HTTP | `Saruman`, built twice and smoke-tested, 2026-10-08 ([#920](https://github.com/Gerrrt/HomeLab/issues/920)) | | 904 | `tpl-fedora-server` | `fedora.pkr.hcl` | Kickstart, `OEMDRV` disc | `Saruman`, built twice and smoke-tested, 2026-10-08 ([#920](https://github.com/Gerrrt/HomeLab/issues/920)) | | 905 | `tpl-opensuse-tw` | `opensuse.pkr.hcl` | AutoYaST, `OEMDRV` disc | `Saruman`, built twice and smoke-tested, 2026-10-09 ([#920](https://github.com/Gerrrt/HomeLab/issues/920)) | diff --git a/packer/kali.pkr.hcl b/packer/kali.pkr.hcl index e95152bc..2d148f67 100644 --- a/packer/kali.pkr.hcl +++ b/packer/kali.pkr.hcl @@ -2,16 +2,16 @@ # build on Saruman for garuda (#921). `-only` picks one: kali.proxmox-iso.kali # or kali.proxmox-iso.kali-saruman. # -# WRITTEN, NOT BUILT (ADR-0074 part 2, #790). The attack VM lives on gandalf, -# installed 2026-11-07 with no template on it yet, and a template belongs to the -# node it was built on. CI proves this file parses; the first build on gandalf -# proves the preseed. +# BUILT on gandalf 2026-11-09, twice, each build smoke-tested (#790). It goes +# through gandalf's own API with gandalf's own token (gandalf.env, not +# phoenix.env), and its disk on `range`: build-the-lab-templates.md §5. The +# attack VM, ragnarok, is cloned from it by scripts/make-attack-vm.sh. # # The one source here that uses Packer's HTTP server. Debian's installer reads # a preseed from the install medium or from a URL, not from a second disc, so # the seed is served from phoenix for the few seconds the installer fetches it. -# That needs phoenix's own firewall to admit the guest's DHCP address on the -# port below — build-the-lab-templates.md §5 says how, for the build on gandalf. +# If phoenix's own firewall is on, it must admit the guest's DHCP address on +# the port below; build-the-lab-templates.md §5 says how. source "proxmox-iso" "kali" { proxmox_url = var.proxmox_url @@ -106,8 +106,8 @@ source "proxmox-iso" "kali" { # workstation (#921, ADR-0091). The same Kali and preseed as 902, built on # Saruman instead: a template belongs to the node it was built on, and 902 # stays gandalf's. Its ISO is pinned by name, the 2026.2 installer on smaug-iso -# that scripts/collect-iso-store-state.sh hashes; 902's is still chosen at its -# first build on gandalf, because gandalf cannot mount smaug-iso. +# that scripts/collect-iso-store-state.sh hashes. 902's is the same installer, +# on gandalf's own local, because gandalf cannot mount smaug-iso. source "proxmox-iso" "kali-saruman" { proxmox_url = var.proxmox_url diff --git a/packer/variables.pkr.hcl b/packer/variables.pkr.hcl index 5102bdff..b3207b3d 100644 --- a/packer/variables.pkr.hcl +++ b/packer/variables.pkr.hcl @@ -157,13 +157,14 @@ variable "arch_iso_file" { default = "smaug-iso:iso/archlinux-2026.10.01-x86_64.iso" } -# A placeholder name: Kali is written and not yet built (ADR-0074 part 2), and -# the first build on gandalf sets this to the ISO it actually has. It stays on -# local: the ISO store's export and pass admit Saruman alone (ADR-0072), so -# gandalf cannot mount it. +# 902's ISO, on gandalf's own local storage: the ISO store's export and pass +# admit Saruman alone (ADR-0072), so gandalf cannot mount it. The same 2026.2 +# installer as 910's, checked against the hash collect-iso-store-state.sh pins +# when it was fetched onto gandalf, 2026-11-09 (#790). A newer Kali is a new +# file name here, never an overwrite. variable "kali_iso_file" { type = string - default = "local:iso/kali-linux-installer-amd64.iso" + default = "local:iso/kali-linux-2026.2-installer-amd64.iso" } # 910's ISO, for garuda (#921). Saruman mounts smaug-iso, so this one sits diff --git a/scripts/README.md b/scripts/README.md index 5b5fe6bf..b0a6a281 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -156,6 +156,7 @@ They need the running estate, so CI cannot run them. Timers do. | `export-dashboards.sh`, `export_dashboards.py` | `dashboards-export` | Folds UI edits in the running Grafana back over the committed JSON | | `capture-screenshots.sh` | `screenshots` | Renders the dashboards into [`docs/images/`](../docs/images/README.md) | | `packer-smoke.sh` | — | Clones a Packer template, boots it, checks it and destroys it ([`packer/`](../packer/README.md)) | +| `make-attack-vm.sh` | — | Makes the range's attack VM, `ragnarok`, on `gandalf` from 902, and proves it neither forwards nor NATs ([`build-the-playground.md`](../docs/runbooks/build-the-playground.md) §5) | | `stage-agent-msis.sh` | — | Stages the Wazuh and Velociraptor MSIs the lab pipeline installs, and the two Debian packages `garuda` installs, into `${SOC_DATA_DIR}/agent-msi` on `odin`, which the SOC stack's Caddy serves. The vendor packages are checked against their pins; the Velociraptor `.deb` is built by the server and its hash printed ([#1068](https://github.com/Gerrrt/HomeLab/issues/1068), [ADR-0092](../docs/adr/0092-enrol-garuda-in-odins-soc-with-debian-packages-staged-on-odin.md)) | | `lab-known-hosts.sh` | — | Reads the lab domain's SSH host keys through the guest agent into `ansible/.known_hosts`, after any rebuild ([ADR-0085](../docs/adr/0085-check-the-lab-domains-host-keys-against-keys-read-through-the-guest-agent.md)) | | `gen_population.py` | — | The lab domain's users, into `ansible/population/` | diff --git a/scripts/make-attack-vm.sh b/scripts/make-attack-vm.sh new file mode 100755 index 00000000..d6279295 --- /dev/null +++ b/scripts/make-attack-vm.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# +# Make the range's attack VM, ragnarok, on gandalf from tpl-kali (902), and +# prove it does not forward (#421, build-the-playground.md §5). +# +# WHY A SCRIPT. The range is rebuilt, not repaired (ADR-0017), so the one guest +# on it worth configuring carefully is one command to make again. The +# isolation it depends on is in what this sets, and §8's third check runs at +# the end of every make rather than when someone remembers to. +# +# WHAT IT MAKES. A full clone of 902 at VMID 130, its disk on `range`: +# net0 vmbr0, ImaginationLAN, DHCP from the pool. No reservation: nothing +# points at it (§5). +# net1 vmbr1, the range bridge, 172.30.30.10/24 and NO gateway. +# Both with the guest firewall flag off (§4's last step). Cloud-init user +# `kali`, phoenix's key. Then, over SSH: /etc/sysctl.d/99-no-forward.conf, +# `sysctl --system`, and §8 check three: both forwarding sysctls 0 and no NAT +# in nft or legacy iptables. +# +# WHAT IT NEEDS. gandalf.env sourced (PROXMOX_URL at 10.0.30.30, the +# phoenix@pve!builder token on gandalf), gandalf's CA in phoenix's trust store +# (build-the-lab-templates.md §5), curl and jq. +# +# Usage: +# set -a; . ~/.config/proxmox/gandalf.env; set +a +# scripts/make-attack-vm.sh +set -euo pipefail + +die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } +info() { printf '\033[0;34m--\033[0m %s\n' "$*"; } +ok() { printf '\033[0;32m PASS\033[0m %s\n' "$*"; } + +NODE=gandalf +TEMPLATE=902 +VMID=130 +NAME=ragnarok +RANGE_ADDR=172.30.30.10/24 +SSH_KEY="${SSH_KEY:-${HOME}/.ssh/id_ed25519}" + +for v in PROXMOX_URL PROXMOX_TOKEN_ID PROXMOX_TOKEN_SECRET; do + [[ -n "${!v:-}" ]] || die "${v} is unset — set -a; . ~/.config/proxmox/gandalf.env; set +a" +done +[[ "${PROXMOX_URL}" == *10.0.30.30* ]] || die "PROXMOX_URL is not gandalf's: source gandalf.env, not phoenix.env" +command -v curl >/dev/null || die "curl is required" +command -v jq >/dev/null || die "jq is required (apt install jq)" + +# The token goes to curl on stdin, never as -H, as in packer-smoke.sh (#846). +api() { + local method="$1" path="$2"; shift 2 + printf 'header = "Authorization: PVEAPIToken=%s=%s"\n' "${PROXMOX_TOKEN_ID}" "${PROXMOX_TOKEN_SECRET}" \ + | curl -K - -fsS -X "${method}" "$@" "${PROXMOX_URL}${path}" +} + +wait_task() { + local upid="$1" + for _ in $(seq 1 360); do + if [[ "$(api GET "/nodes/${NODE}/tasks/${upid}/status" | jq -r '.data.status')" == stopped ]]; then + api GET "/nodes/${NODE}/tasks/${upid}/status" | jq -e '.data.exitstatus == "OK"' >/dev/null \ + || die "task ${upid} did not finish OK" + return 0 + fi + sleep 5 + done + die "task ${upid} still running after 30 minutes" +} + +api GET "/nodes/${NODE}/qemu/${TEMPLATE}/config" | jq -e '.data.template == 1' >/dev/null \ + || die "VM ${TEMPLATE} on ${NODE} is not a template: build tpl-kali first" +if api GET "/nodes/${NODE}/qemu/${VMID}/status/current" >/dev/null 2>&1; then + die "VMID ${VMID} already exists on ${NODE}. The range is rebuilt, not repaired: destroy it first" +fi + +info "full clone ${TEMPLATE} -> ${VMID} (${NAME}) on range" +wait_task "$(api POST "/nodes/${NODE}/qemu/${TEMPLATE}/clone" \ + --data-urlencode "newid=${VMID}" --data-urlencode "name=${NAME}" \ + --data-urlencode "full=1" --data-urlencode "storage=range" | jq -r '.data')" +ok "cloned" + +# sshkeys is URL-encoded inside the form value, as packer-smoke.sh does. +keys="$(jq -rn --arg k "$(cat "${SSH_KEY}.pub")" '$k | @uri')" +api PUT "/nodes/${NODE}/qemu/${VMID}/config" \ + --data-urlencode "cores=4" --data-urlencode "memory=8192" \ + --data-urlencode "net0=virtio,bridge=vmbr0,firewall=0" \ + --data-urlencode "net1=virtio,bridge=vmbr1,firewall=0" \ + --data-urlencode "ipconfig0=ip=dhcp" \ + --data-urlencode "ipconfig1=ip=${RANGE_ADDR}" \ + --data-urlencode "ciuser=kali" --data-urlencode "sshkeys=${keys}" \ + --data-urlencode "tags=range;attack" \ + --data-urlencode "description=The range's attack VM (#421, build-the-playground.md §5). Made by scripts/make-attack-vm.sh." \ + >/dev/null +ok "net0 vmbr0 DHCP, net1 vmbr1 ${RANGE_ADDR} no gateway, firewall flags off" + +# The one thing a typo here would break is the isolation, so read it back. +cfg="$(api GET "/nodes/${NODE}/qemu/${VMID}/config")" +jq -e '(.data.net1 | test("bridge=vmbr1")) and (.data.net0 | test("bridge=vmbr0")) + and ((.data.ipconfig1 // "") | test("gw") | not)' <<<"${cfg}" >/dev/null \ + || die "the config did not read back as written: $(jq -c '.data | {net0,net1,ipconfig0,ipconfig1}' <<<"${cfg}")" +ok "config reads back" + +info "starting" +wait_task "$(api POST "/nodes/${NODE}/qemu/${VMID}/status/start" | jq -r '.data')" + +info "waiting for the guest agent to report the VLAN 30 lease" +addr="" +for _ in $(seq 1 60); do + addr="$(api GET "/nodes/${NODE}/qemu/${VMID}/agent/network-get-interfaces" 2>/dev/null \ + | jq -r '[.data.result[]? | .["ip-addresses"][]? | select(."ip-address-type" == "ipv4") + | ."ip-address" | select(startswith("10.0.30."))][0] // empty' || true)" + [[ -n "${addr}" ]] && break + sleep 10 +done +[[ -n "${addr}" ]] || die "no 10.0.30.x address from the guest agent after 10 minutes" +ok "agent reports ${addr}" + +SSH=(ssh -i "${SSH_KEY}" -o BatchMode=yes -o StrictHostKeyChecking=no + -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=10) +for _ in $(seq 1 18); do + "${SSH[@]}" "kali@${addr}" true 2>/dev/null && break + sleep 10 +done +"${SSH[@]}" "kali@${addr}" true || die "kali@${addr} admitted no SSH login in three minutes" +ok "phoenix's key opens kali@${addr}" + +info "forwarding off, persistently (§5), then §8 check three" +# shellcheck disable=SC2016 # expanded on the guest, not here +"${SSH[@]}" "kali@${addr}" 'sudo sh -eu -c " +printf \"net.ipv4.ip_forward = 0\nnet.ipv6.conf.all.forwarding = 0\n\" > /etc/sysctl.d/99-no-forward.conf +sysctl --system >/dev/null +"' +out="$("${SSH[@]}" "kali@${addr}" 'sudo sh -c " +sysctl net.ipv4.ip_forward net.ipv6.conf.all.forwarding +nft list ruleset 2>/dev/null | grep -Ei \"masquerade|snat\" || echo \"no NAT\" +iptables -t nat -S 2>/dev/null | grep -- \"-j MASQUERADE\" || echo \"no legacy NAT\" +ip -br addr; ip route +"')" +printf '%s\n' "${out}" +grep -qx 'net.ipv4.ip_forward = 0' <<<"${out}" || die "ip_forward is not 0" +grep -qx 'net.ipv6.conf.all.forwarding = 0' <<<"${out}" || die "IPv6 forwarding is not 0" +grep -qx 'no NAT' <<<"${out}" || die "nft has a NAT rule" +grep -qx 'no legacy NAT' <<<"${out}" || die "iptables-legacy has a MASQUERADE rule" +grep -q "${RANGE_ADDR%/*}/24" <<<"${out}" || die "${RANGE_ADDR} is not on any interface" +[[ "$(grep -c '^default' <<<"${out}")" == 1 ]] || die "expected exactly one default route" +grep -q '^default via 10\.0\.30\.1 ' <<<"${out}" || die "the default route is not via 10.0.30.1" +ok "no forwarding, no NAT, one default route via 10.0.30.1, ${RANGE_ADDR} on the range" + +printf '\n\033[0;32m%s is up at %s and %s\033[0m\n' "${NAME}" "${addr}" "${RANGE_ADDR%/*}"