Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ hole from the monitoring VLAN into the monitored one.
| --- | --- | --- | --- |
| `prometheus` (10.0.99.20) | 🔴 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, blackbox-exporter, speedtest-tracker, docker-socket-proxy, Alloy |
| `Saruman` (10.0.30.20) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, hosting twenty-one guests: `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)); `phoenix`, built 2026-09-20 ([#436](https://github.com/Gerrrt/HomeLab/issues/436)); `odin`, built 2026-09-27 for the security tooling (ADR-0030); `fenrir`, built 2026-09-30 as the Zeek sensor ([#437](https://github.com/Gerrrt/HomeLab/issues/437)); `golem`, built 2026-10-03 as the backup server ([#485](https://github.com/Gerrrt/HomeLab/issues/485)); `eden`, built 2026-10-07 as the BloodHound guest ([#451](https://github.com/Gerrrt/HomeLab/issues/451)), off between sessions; `garuda`, built 2026-10-10 as the analyst workstation ([#921](https://github.com/Gerrrt/HomeLab/issues/921)); and ADR-0029's six-machine domain, built by hand 2026-09-24 to 2026-09-25 ([#414](https://github.com/Gerrrt/HomeLab/issues/414)), of which `carbuncle` and `siren` run per session; and all eight of [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)'s on-demand dotfiles test VMs, `dot-debian`, `dot-fedora` and `dot-windows`, built 2026-10-08, and `dot-opensuse`, `dot-arch`, `dot-alpine`, `dot-gentoo` and `dot-nixos`, built 2026-10-09 ([#920](https://github.com/Gerrrt/HomeLab/issues/920)). Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest |
| `gandalf` (10.0.30.30) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, the range host, installed 2026-11-07 ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). No guests yet. Guest disks go on the ZFS pool `range`, targets go on the portless `vmbr1`, and it is off between sessions ([ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md), [ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)) |
| `gandalf` (10.0.30.30) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, the range host, installed 2026-11-07 ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). One guest, the attack VM `ragnarok` (VMID 130), made 2026-11-09 by [`scripts/make-attack-vm.sh`](../scripts/make-attack-vm.sh) from template 902 and dual-homed: a DHCP lease on VLAN 30, and `172.30.30.10` on `vmbr1`. No targets yet. Guest disks go on the ZFS pool `range`, targets go on the portless `vmbr1`, and it is off between sessions ([ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md), [ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)) |
| `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy and its Docker socket proxy, and Caddy as the ingest proxy in front of the first two ([#834](https://github.com/Gerrrt/HomeLab/issues/834)) — the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) |
| `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. Built 2026-09-27 by [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), the stack authored and CI-validated ahead of the guest the way `stacks/lab` was ahead of `alexander` ([#266](https://github.com/Gerrrt/HomeLab/issues/266), [#267](https://github.com/Gerrrt/HomeLab/issues/267)); the six domain machines enrol as agents by GPO, which is the day those two issues close. Its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) |
| `fenrir` (10.0.30.90) | 🟢 30 | [`stacks/sensor`](../stacks/sensor) | Zeek on a `tc` mirror of `Saruman`'s lab bridge, so that the domain's east-west traffic, which crosses no router and so never reaches Suricata on `morpheus`, is observed at all ([#437](https://github.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)). A guest on `Saruman`, VMID 190, with a second NIC alone on the port-less `vmbr1` that receives only the mirror's copies. Its logs go to `alexander`, never to VLAN 99; whether the mirror is delivering is answered on the hypervisor by `homelab_zeek_mirror_active`. Built 2026-09-30 by [`build-the-sensor-guest.md`](runbooks/build-the-sensor-guest.md), and proved by rebooting `Saruman` with the mirror's timer disabled: the gauge fell to 0 and `ZeekMirrorInactive` fired. Alloy agent (Docker) |
Expand Down
33 changes: 33 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,39 @@ docstring gives: it is a record, not a claim about now.

## 2026-11-09

- **[#421](https://github.com/Gerrrt/HomeLab/issues/421),
[#790](https://github.com/Gerrrt/HomeLab/issues/790): `tpl-kali` (902) is
built on `gandalf`, and the attack VM `ragnarok` is up,
`build-the-playground.md` §5.**
- **Access.** `gandalf` is not clustered, so it got its own `phoenix@pve`
user, `PhoenixBuilder` and `PhoenixIsoCleanup` roles and
`phoenix@pve!builder` token. The role is `build-the-jumpbox.md` §4's plus
`VM.GuestAgent.Audit`. Its ACLs cover `range`, `local`, and both
`vmbr0` and `vmbr1`. The two builds, the two smoke tests and the attack
VM asked for nothing more. `phoenix` trusts `gandalf`'s own cluster CA
and keeps the token in `gandalf.env`, apart from `phoenix.env`.
- **The builds.** The ISO is the 2026.2 installer already pinned for 910,
fetched onto `gandalf`'s `local` and checked against that hash. The
first build stopped at validation, because the variables file refuses
an empty `PKR_VAR_build_password` even for Kali. Sourcing `phoenix.env`
under `gandalf.env` fixed it, and §5 says so now. The build took
10 min 33 s and the `-force` rebuild 10 min 41 s. `packer-smoke.sh 902`
passed after each, which is #790's acceptance. 910's boot-order and 20 s
fixes were not needed here. `ufw` is inactive on `phoenix`, so no port
was opened.
- **The attack VM.** `ragnarok`, VMID 130, is a full clone on `range` with
4 cores and 8 GiB, made by the new `scripts/make-attack-vm.sh`. `net0` is
on `vmbr0` by DHCP and took `10.0.30.111`. `net1` is on `vmbr1` at
`172.30.30.10/24` with no gateway. Both have the firewall flag off. The
script wrote `99-no-forward.conf` and ran §8's third check: both
sysctls `0`, no NAT in nft or iptables, and one default route, via
`10.0.30.1`.
- **#1196's check.** On the fresh clone it read `running`, 0 failed units
and 0 accounts expiring at the epoch, the check
[#1196](https://github.com/Gerrrt/HomeLab/issues/1196) owed 902.
- **What is not done.** Sliver is not installed, there are no targets
(§6), and §8's first, second and fourth checks and §9 have not run.

- **[#421](https://github.com/Gerrrt/HomeLab/issues/421): `gandalf`'s
management plane is closed, `build-the-playground.md` §4.** The firewall
had been off since the install, so ADR-0014's rules did not exist yet and
Expand Down
Binary file modified docs/diagrams/current/network.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/diagrams/current/network.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 3 additions & 2 deletions docs/network.md
Original file line number Diff line number Diff line change
Expand Up @@ -713,7 +713,8 @@ Where things get broken on purpose.
away before the attack VM starts taking leases from the same pool.
- A second hypervisor, `gandalf`, carries the attack tooling and the
deliberately-vulnerable targets. It is installed (Proxmox VE 9,
2026-11-07) and has no guests yet. It sits on this segment single-homed, on
2026-11-07) and has one guest, the attack VM `ragnarok` (VMID 130, made
2026-11-09). It sits on this segment single-homed, on
`neo`'s untagged `ether20`, at `10.0.30.30`: a static below `.100`, with a
reservation. Its Intel AMT, which would be the name `ifrit`, is disabled
([ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)).
Expand All @@ -726,7 +727,7 @@ Where things get broken on purpose.
is the third private block in the house and the only one that is not routed
anywhere, which is what makes a `172.30.30.x` source in a firewall block a
leak reporting itself. The attack VM takes a lease from the pool like any
other guest. The build is
other guest, with no reservation, and holds `172.30.30.10` on the bridge. The build is
[`build-the-playground.md`](runbooks/build-the-playground.md).

- **The WireGuard peers live on `172.31.0.0/24`, and it is routed rather than
Expand Down
12 changes: 7 additions & 5 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,8 +240,8 @@ Closes on BloodHound running where nothing attacks it.
[ADR-0074](adr/0074-build-the-lab-templates-with-packer-from-phoenix.md):
901 Ubuntu, 911 Windows 11 and 912 Server 2025 built from `smaug-iso`,
each clone passing `scripts/packer-smoke.sh`, SSH included,
→ [runbook](runbooks/build-the-lab-templates.md); Kali's
template waits for its build on `gandalf`, [#790](https://github.com/Gerrrt/HomeLab/issues/790)) →
→ [runbook](runbooks/build-the-lab-templates.md); Kali's 902
built on `gandalf` 2026-11-09, [#790](https://github.com/Gerrrt/HomeLab/issues/790)) →
[#445](https://github.com/Gerrrt/HomeLab/issues/445) OpenTofu (done
2026-10-03,
[ADR-0076](adr/0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md):
Expand Down Expand Up @@ -286,9 +286,11 @@ Gated on the domain, on a household observation, or on something to publish.

- **[#421](https://github.com/Gerrrt/HomeLab/issues/421) Build the range on
`gandalf`.** Gated on #414 and the SOC: an attack VM pointed at an
uninstrumented estate teaches nothing. Both are met. The runbook's §1–§4
are behind it: the host, its bridges, `Saruman` at `.20` and the closed
management plane. Next is §5, the attack VM, then §6–§9, in order.
uninstrumented estate teaches nothing. Both are met. The runbook's §1–§5
are behind it: the host, its bridges, `Saruman` at `.20`, the closed
management plane, and the attack VM `ragnarok` on template 902. Next is §6,
the targets, then §7–§9, in order. Sliver goes on `ragnarok` before the
first exercise.
The name is
[ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)'s.
→ [runbook](runbooks/build-the-playground.md)
Expand Down
3 changes: 3 additions & 0 deletions docs/runbooks/build-the-analyst-workstation.md
Original file line number Diff line number Diff line change
Expand Up @@ -553,6 +553,9 @@ the console password.
- **Still owed:** the same check on 902 (gandalf's Kali, same preseed) when it
is next built — [#1196](https://github.com/Gerrrt/HomeLab/issues/1196),
tied to [#790](https://github.com/Gerrrt/HomeLab/issues/790).
- **Paid 2026-11-09:** 902 was built twice on `gandalf`, and its clone
`ragnarok` read `running`, 0 failed units and 0 accounts at the epoch
([`build-the-lab-templates.md`](build-the-lab-templates.md) §5).
- **§12, phase 2, 2026-10-10**
([#1163](https://github.com/Gerrrt/HomeLab/pull/1163),
[ADR-0092](../adr/0092-enrol-garuda-in-odins-soc-with-debian-packages-staged-on-odin.md)).
Expand Down
101 changes: 80 additions & 21 deletions docs/runbooks/build-the-lab-templates.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Runbook: Build the lab's VM templates with Packer, from `phoenix`

**Target:** templates 901, 903–912 on `Saruman`, built from
`phoenix` through the Proxmox API. 902, Kali, is written here and built on
`gandalf`, the range host, installed 2026-11-07 and not yet built on. 910 is the same Kali build on `Saruman`, for
`phoenix` through the Proxmox API. 902, Kali, is built on `gandalf`, the
range host, through `gandalf`'s own API (§5, first built 2026-11-09). 910 is the same Kali build on `Saruman`, for
`garuda` (§5b, [ADR-0091](../adr/0091-put-a-kali-purple-analyst-workstation-on-saruman.md)). 903 (Debian), 904 (Fedora), 905 (openSUSE
Tumbleweed), 906 (Arch), 907 (Alpine), 908 (Gentoo) and 909 (NixOS) are the
dotfiles OS layers' templates
Expand Down Expand Up @@ -488,34 +488,93 @@ node. The preseed is served over Packer's HTTP server on port 8800 of
`phoenix`, because Debian's installer reads one from a URL or its own medium,
not from a second disc.

1. Admit the guest to that port on `phoenix` for the length of the build. If
`ufw` is active, run `sudo ufw allow from 10.0.30.0/24 to any port 8800 proto tcp`,
and delete the rule afterwards. Debian's build (§4) needs this step and
no other here.
2. Grant `PhoenixBuilder` on `gandalf`'s storage and bridge, and trust
`gandalf`'s CA on `phoenix`, as §2 did for `Saruman`. Unless `gandalf` joins
`Saruman` in a cluster, it is its own API. Point `PROXMOX_URL` at it for
this build, and admit `phoenix` to its 8006 the way ADR-0043 did on
`Saruman`.
3. Build, naming the ISO actually on `gandalf` and **its** storage.
`disk_storage` defaults to `large_data`, which is `Saruman`'s pool, and
the disk creation fails without the override:
> **Ran 2026-11-09** ([#790](https://github.com/Gerrrt/HomeLab/issues/790),
> [#421](https://github.com/Gerrrt/HomeLab/issues/421)). Steps 2–5 below are
> what ran. The first build took 10 min 33 s and the `-force` rebuild
> 10 min 41 s. `packer-smoke.sh 902` passed after each. `gandalf`'s 10 s
> `boot_wait` was enough with no boot order, unlike 910 on `Saruman`. The
> clone `ragnarok` then read `running` with 0 failed units and no account
> expiring at the epoch, the check
> [#1196](https://github.com/Gerrrt/HomeLab/issues/1196) owed 902.

1. **Port 8800 on `phoenix`.** If `ufw` is active, run
`sudo ufw allow from 10.0.30.0/24 to any port 8800 proto tcp` for the build
and delete the rule afterwards. On 2026-11-09 `ufw` was inactive on
`phoenix`, so there was nothing to open. Debian's build (§4) needs this
step and no other here.
2. **The user, the roles and the token, on `gandalf`, as root.** `gandalf` is
not clustered with `Saruman`
([ADR-0039](../adr/0039-decline-proxmox-clustering-while-ifrit-is-the-range.md)),
so it is its own API with its own users. The role is
[`build-the-jumpbox.md`](build-the-jumpbox.md) §4's plus §2's
`VM.GuestAgent.Audit`. The storage is `range`, and the bridges are both
`vmbr0` and `vmbr1`, because the attack VM's second NIC is on the range
bridge:

```bash
packer build -only='kali.proxmox-iso.kali' \
-var kali_iso_file=local:iso/<the iso> \
-var disk_storage=<gandalf's guest storage> \
packer/
pveum role add PhoenixBuilder --privs "VM.Allocate VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Cloudinit VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Console VM.PowerMgmt VM.Audit VM.Snapshot VM.GuestAgent.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use"
pveum role add PhoenixIsoCleanup --privs "Datastore.Allocate"
pveum user add phoenix@pve --comment "deployment host, ADR-0043"
pveum acl modify /vms --users phoenix@pve --roles PhoenixBuilder
pveum acl modify /storage/local --users phoenix@pve --roles PhoenixBuilder,PhoenixIsoCleanup
pveum acl modify /storage/range --users phoenix@pve --roles PhoenixBuilder
pveum acl modify /sdn/zones/localnetwork/vmbr0 --users phoenix@pve --roles PhoenixBuilder
pveum acl modify /sdn/zones/localnetwork/vmbr1 --users phoenix@pve --roles PhoenixBuilder
pveum acl modify /nodes/gandalf --users phoenix@pve --roles PVEAuditor
pveum acl list | grep phoenix
pveum user token add phoenix@pve builder --privsep 0
```

4. Prove it on `gandalf`, as §6 does on `Saruman`:
Paste this into the node's **Shell** in the web UI, or into an SSH session
from Hicks. On 2026-11-09 a first paste went into a zsh on `phoenix`,
where `pveum` does not exist. The paste handling there also turned every
space after the first line into a literal `\s`. Nothing ran, and nothing
needed undoing. No privilege beyond these was asked for by the two
builds, the two smoke tests or `scripts/make-attack-vm.sh`.

**The installer goes on `gandalf`'s own `local`.** It is the same 2026.2
ISO as 910's, checked against the hash
[`collect-iso-store-state.sh`](../../scripts/collect-iso-store-state.sh)
pins:

```bash
cd /var/lib/vz/template/iso && wget -nc https://cdimage.kali.org/kali-2026.2/kali-linux-2026.2-installer-amd64.iso && echo "6dbefacc95e3b556c19c48e8bae39b8b505e2d3a1aba0bfb7ab62b036c3d2ba3 kali-linux-2026.2-installer-amd64.iso" | sha256sum -c -
```

3. **`phoenix` trusts `gandalf`'s CA and holds its token apart.** `gandalf`
serves its own `PVE Cluster Manager CA` certificate, naming `10.0.30.30`
(read 2026-11-09, so it needs no fix of the kind §2 records for
`Saruman`). Relay the CA from Hicks:
`ssh gandalf cat /etc/pve/pve-root-ca.pem | ssh phoenix 'cat > ~/gandalf-pve-root-ca.crt'`.
Install it on `phoenix` as
`/usr/local/share/ca-certificates/gandalf-pve-root-ca.crt`, as §2 does
for `Saruman`'s. The token goes in **`~/.config/proxmox/gandalf.env`**,
mode 600, with its own `PROXMOX_URL=https://10.0.30.30:8006/api2/json`,
`PROXMOX_TOKEN_ID=phoenix@pve!builder` and `PROXMOX_TOKEN_SECRET`. It is
not added to `phoenix.env`, so a build for `Saruman` can never be pointed
at `gandalf` by accident, or the reverse. Prove it with §2's `curl …/version`
line, with `gandalf.env` sourced in place of `phoenix.env`.
4. **Build.** Source `phoenix.env` first and `gandalf.env` over it. The
variables file refuses an empty `PKR_VAR_build_password` for every source,
Kali included, and that password lives in `phoenix.env`. The `PROXMOX_*`
lines in `gandalf.env` then win. `disk_storage` defaults to `large_data`,
which is `Saruman`'s pool, so name `range`:

```bash
set -a; . ~/.config/proxmox/phoenix.env; . ~/.config/proxmox/gandalf.env; set +a
packer init packer/
packer build -force -only='kali.proxmox-iso.kali' -var disk_storage=range packer/
```

`kali_iso_file` defaults to the ISO from step 2.
5. **Prove it**, with `gandalf.env` sourced:

```bash
PROXMOX_NODE=gandalf scripts/packer-smoke.sh 902
```

Then run §8's second `-force` build and smoke test, which is #790's
acceptance.
Then run the build and the smoke test a second time, which is #790's
acceptance and §8's rule.

## 5b. Kali on `Saruman`, for `garuda`

Expand Down
Loading
Loading