Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ documents for different readers.
can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed
identically everywhere. [How](docs/architecture.md#observability-data-flow)
- **Dashboards and alerting as code.** 8 provisioned dashboards, 156 panels, and
179 alert rules — 159 metric-based in Prometheus, 20 log-based in Loki — sharing
180 alert rules — 159 metric-based in Prometheus, 21 log-based in Loki — sharing
one Alertmanager routing tree. No dashboard exists only in a database.
- **Secrets encrypted in-repo with SOPS + age.** Per-device credentials,
decrypted at deploy time into gitignored paths, with `git log` showing which
Expand Down Expand Up @@ -203,7 +203,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology
│ ├── observability/ # the estate's stack on prometheus — ten services
│ │ ├── prometheus/ # config, file_sd targets, 159 alert rules
│ │ ├── alertmanager/ # routing and inhibition
│ │ ├── loki/ # single-binary config + 20 LogQL rules
│ │ ├── loki/ # single-binary config + 21 LogQL rules
│ │ ├── alloy/ # the agent config directory, shipped to every host
│ │ ├── snmp-exporter/ # generator.yaml is the source of truth
│ │ └── grafana/ # provisioning + 8 dashboards
Expand Down
19 changes: 19 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,25 @@ docstring gives: it is a record, not a claim about now.

## 2026-11-09

- **[#421](https://github.com/Gerrrt/HomeLab/issues/421): `RangeSubnetLeaked`
is written, and `build-the-playground.md` §8 ran on `gandalf`.**
- **The rule** is ADR-0014's optional one: any `filterlog` line, block or
pass, whose source is in `172.30.30.0/24`. It is critical and fires at
once. That makes 21 Loki rules and 180 in all. §9 will prove it by
causing a leak, once it is deployed.
- **§8 ran before §6**, against the roadmap's order, because §9 must run
before any vulnerable image exists on the machine. Checks one and two
passed: `vmbr1` holds only `ragnarok`'s tap and has no address, and
there is no VLAN-aware bridge. Check three passed at `ragnarok`'s make.
- **Check four failed first,** on a throwaway clone of `tpl-kali` at
`172.30.30.199`. The template's `interfaces.d/eth0` is a DHCP stanza the
preseed writes for the Packer build. In a static clone, ifupdown raised
it beside cloud-init's stanza. dhcpcd fell back to `169.254.x.x` and
added `default dev eth0 scope link`, so the pings timed out instead of
being unreachable. Nothing left the bridge. With the stanza gone, the
check passed. `ragnarok` takes a real lease on `eth0` and is not
affected. #1341 makes the build remove the stanza. 902 keeps it until
it is rebuilt.
- **`tpl-kali`'s build now removes its own eth0 DHCP stanza before
templating.** The preseed writes `/etc/network/interfaces.d/eth0` so the
build has a network ([#790](https://github.com/Gerrrt/HomeLab/issues/790)),
Expand Down
10 changes: 9 additions & 1 deletion docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -326,6 +326,14 @@ rather than from the segment. A `10.0.30.x` source is the lab, a `172.31.x`
source is a remote peer, and a `172.30.x` source is neither — that is `gandalf`'s
range bridge escaping, which is a different incident.

**That incident has its own alert, `RangeSubnetLeaked`.** ADR-0014 chose
`172.30.30.0/24` so that a forwarded packet from the range would miss the
`→ any` pass, hit default deny and be logged: "a leak reports itself". The
rule reads any `filterlog` line, block or pass, whose *source* is in that
subnet. A range address as a destination is followed by a port, not a second
address, so it does not match. `build-the-playground.md` §9 proves the rule
by causing a leak.

A tripwire that never fires is indistinguishable from a broken one, which is
this whole family of defect, so the logging path was proven rather than assumed:
logging was briefly enabled on the lowest-volume terminal egress rule, and 49 of
Expand All @@ -344,7 +352,7 @@ separates a quiet stream from a stopped one.

## Alerting

179 rules in total: 159 metric-based in `prometheus/rules/`, and 20 log-based in
180 rules in total: 159 metric-based in `prometheus/rules/`, and 21 log-based in
`loki/rules/`.

### Log-based (Loki ruler)
Expand Down
8 changes: 5 additions & 3 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -299,9 +299,11 @@ Gated on the domain, on a household observation, or on something to publish.
`gandalf`.** Gated on #414 and the SOC: an attack VM pointed at an
uninstrumented estate teaches nothing. Both are met. The runbook's §1–§5
are behind it: the host, its bridges, `Saruman` at `.20`, the closed
management plane, and the attack VM `ragnarok` on template 902. Next is §6,
the targets, then §7–§9, in order. Sliver goes on `ragnarok` before the
first exercise.
management plane, and the attack VM `ragnarok` on template 902. §8 ran
before any target, because §9's leak proof must run before any vulnerable
image is on the machine. Next is §9, which waits on `RangeSubnetLeaked`
being deployed so that the leak proves the alert as well as the query.
Then §6, the targets. Sliver goes on `ragnarok` before the first exercise.
The name is
[ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)'s.
→ [runbook](runbooks/build-the-playground.md)
Expand Down
31 changes: 29 additions & 2 deletions docs/runbooks/build-the-playground.md
Original file line number Diff line number Diff line change
Expand Up @@ -723,6 +723,29 @@ has been rerun.
ADR-0014 makes four claims. Each one is checkable, and none of them is checked
by anything else in this repository.

> **Ran on `gandalf` 2026-11-09** ([#421](https://github.com/Gerrrt/HomeLab/issues/421)),
> before any target existed, and so before §6, as §9 requires.
>
> - **One:** `vmbr0` carried `nic0` and `ragnarok`'s `tap130i0`, and `vmbr1`
> carried `tap130i1` and nothing else. `vmbr1` had `0` addresses.
> - **Two:** `0`.
> - **Three:** `make-attack-vm.sh` ran it on `ragnarok` the same night (§5).
> - **Four:** on a throwaway clone of `tpl-kali` at `172.30.30.199`, VMID 199,
> which then served §9. **It failed first.** The template carries
> `/etc/network/interfaces.d/eth0`, a DHCP stanza its preseed writes so the
> Packer build has a network. ifupdown raised it beside cloud-init's static
> stanza. dhcpcd got no lease on a bridge with no DHCP server, so it fell
> back to `169.254.x.x` with a `default dev eth0 scope link` route. Both
> pings then timed out instead of being unreachable. The packets went
> nowhere, because nothing on `vmbr1` routes, but that is the "filtered,
> not unrouted" state this check exists to catch. With the stanza removed
> and dhcpcd stopped, the guest had no default route, and both pings gave
> `Network is unreachable` while `172.30.30.10` answered. `ragnarok` is not
> affected, because its `eth0` takes a real lease. Since #1341 the build
> removes the stanza before templating. **Until 902 is rebuilt, a
> static-only clone of it on `vmbr1` needs that file removed before this
> check passes.**

**One — `gandalf` is single-homed, and `vmbr1` has no port.** On the host:

```bash
Expand Down Expand Up @@ -796,8 +819,10 @@ the machine.**
| jq '.data.result'
```

A row with a `172.30.30.x` source is the property working. **An empty result
is not a pass** — it means the leak was silent, and you should find out
A row with a `172.30.30.x` source is the property working, and the alert
`RangeSubnetLeaked` should fire within a ruler cycle. Check it in
Alertmanager or on the security channel, where its `category: security`
label routes it, not on the urgent one. **An empty result is not a pass** — it means the leak was silent, and you should find out
whether the packet was forwarded at all before concluding anything.
4. Turn forwarding back off, remove the route, delete the guest, and rerun §8's
third check.
Expand All @@ -822,6 +847,8 @@ Addresses are parsed at query time rather than indexed, per
| Everything on `vmbr1` can reach everything else and nothing outside | Working as designed | — |
| `Saruman` stops appearing in Loki after §3 | The pass rule still names `.110` | §3, step 2 |
| Kea hands `10.0.30.30` to something else | The reservation was not added | §2 |
| `RangeSubnetLeaked` fires outside a §9 run | `ragnarok` is forwarding, or a target has a NIC on `vmbr0` | Power `gandalf` off; [`respond-to-a-security-alert.md`](respond-to-a-security-alert.md#rangesubnetleaked) |
| A static target on `vmbr1` has a `169.254.x.x` address and a `default dev` route | A DHCP stanza beside the static one, such as `interfaces.d/eth0` in a `tpl-kali` built before #1341, so dhcpcd fell back to link-local | Remove the stanza and stop dhcpcd, then §8 check four |
| The §9 query returns nothing | Either the leak never happened, or firewall logs are not arriving | Check the pipeline first: [`ship-firewall-logs.md`](ship-firewall-logs.md) §7 |

---
Expand Down
23 changes: 23 additions & 0 deletions docs/runbooks/respond-to-a-security-alert.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,29 @@ tooling, so it is the more serious of the two.
also means the ADR-0042 blocks are missing, or ordered below the
catch-all ([`open-the-remote-path.md`](open-the-remote-path.md)).

## RangeSubnetLeaked

pfSense logged a packet whose source is in `172.30.30.0/24`, the range bridge
on `gandalf`. That bridge has no physical port and no gateway (ADR-0017), so
the packet was forwarded out by a guest that should not forward. Either
`ragnarok` has forwarding on, or a target has a NIC on `vmbr0`.

```logql
{app="filterlog"} |~ `,172\.30\.30\.[0-9]+,[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+,`
```

- **Unless [`build-the-playground.md`](build-the-playground.md) §9 is being
run on purpose right now, contain first.** Power `gandalf` off. It is off
between sessions anyway (§7), and nothing on it is kept.
- **Then find which guest.** The source address names the target. On
`ragnarok`, run §8's third check: forwarding, and NAT in either table.
Docker resets `ip_forward` at every daemon start. On `gandalf`, run
`qm config <vmid> | grep ^net` for each target: every NIC must say
`bridge=vmbr1`.
- **A `pass` rather than a `block`** also means a firewall rule now passes a
source outside `10.0.30.0/24` on `igc0.30`. Treat that as the
`LabSegmentReachedInternalNetwork` case above: find the rule and revert it.

## UnknownDeviceOnManagementSegment

A MAC address took a DHCP lease on VLAN 99 for the first time in seven days.
Expand Down
2 changes: 1 addition & 1 deletion docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ What this network is actually built to survive:
| Someone on the lab segment reaching the SOC's web UIs | **Scoped by address, and accepted.** The Wazuh dashboard (`443`) and the Velociraptor GUI (`8889`) on `odin` are published by the stack's `caddy` service (container `soc-agent-msi`), not by their own containers, and its Caddyfile answers Hicks's `10.0.50.0/24` and refuses everyone else ([#1139](https://github.com/Gerrrt/HomeLab/issues/1139)). Until then both answered all of VLAN 30, the six domain guests included, because traffic between VLAN 30 hosts never crosses pfSense — so no firewall rule could have narrowed them. WireGuard peers are not in the list. The residuals: Hicks is a whole /24 with no per-host source yet ([ADR-0031](adr/0031-narrow-hicks-to-a-named-list-on-winterfell-and-leave-the-lab-open.md)), and an address on VLAN 30 can be spoofed, so this narrows who reaches the login pages and does not replace them. And the hop from Caddy to Velociraptor skips certificate verification: the GUI's certificate is signed by the CA inside the gitignored `server.config.yaml` on `odin`, and the hop is container to container on the `soc` bridge. The dashboard hop is verified against the Wazuh generator's root |
| A range target with a path out | It has none — `gandalf`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) |
| Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) |
| Losing visibility of a failure | 179 alert rules, 30 days of metrics and logs |
| Losing visibility of a failure | 180 alert rules, 30 days of metrics and logs |
| Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana |
| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What happens when the cut outlasts the pack is armed and not proved.** Since 2026-09-23 the firewall's NUT server reads `mjolnir`, and `Saruman` and `smaug` subscribe to it on their own gateway addresses, so a real cut would now try to run the sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides: `Saruman` and `smaug` halt first and `morpheus` last. Nobody has watched it run. Since the move into the 32U rack on 2026-11-06 the PDU also carries `trinity`, `gandalf` (off until its build) and the console, with `smaug` racked and on the PDU directly rather than by its old long cord to the media room. [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) makes `trinity` a third subscriber and bounds `Saruman`'s guest stop, and neither is applied yet, so `trinity` still stops uncleanly when the pack empties; `gandalf`, the console and `neo` are left to the pack by decision. The pack itself is unmeasured: about 47 minutes at 21 % load is the card's own estimate. [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) §5 to §7 are what is left — the forced-shutdown proof and the one mains pull that measures the pack ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) |
| The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet off the estate.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data. That tier has held real photographs since 2026-09-28, and their off-estate copy is built but has no holder yet ([#455](https://github.com/Gerrrt/HomeLab/issues/455)) — see below |
Expand Down
2 changes: 1 addition & 1 deletion stacks/observability/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ alertmanager/
alertmanager.yaml severity + category routing, inhibition
loki/
loki-config.yaml single-binary, filesystem, 30-day retention
rules/*.rules.yaml 20 LogQL rules, security and watchdog, evaluated by Loki's ruler
rules/*.rules.yaml 21 LogQL rules, security and watchdog, evaluated by Loki's ruler
alloy/ the agent config — Alloy loads the directory
config.alloy every monitored host
docker.alloy hosts with a Docker socket
Expand Down
39 changes: 39 additions & 0 deletions stacks/observability/loki/rules/security.rules.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -481,6 +481,45 @@ groups:
peer, which also means the ADR-0042 blocks are missing or ordered
below the catch-all. Treat as a live segmentation failure.

- alert: RangeSubnetLeaked
# ADR-0014's optional rule, landed with #421 once the subnet was real.
# 172.30.30.0/24 is gandalf's range bridge, vmbr1, which has no
# physical port and no gateway (ADR-0017). No packet from it should
# ever reach morpheus. The address was chosen so that one which does
# arrives on igc0.30 with a source outside 10.0.30.0/24, misses the
# `→ any` pass, hits default deny and is logged: "a leak reports
# itself". That is the block this fires on. A pass would be worse,
# and it fires on that too.
#
# The pattern anchors the address as the SOURCE: an address followed
# by another address. A 172.30.30.x destination is followed by a
# port, so a reply into the range never matches. And nothing on the
# estate sends there: morpheus has no route to the subnet.
#
# build-the-playground.md §9 proves the rule by causing a leak.
expr: |
sum by (interface, action) (
count_over_time(
{app="filterlog"}
|~ `,172\.30\.30\.[0-9]+,[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+,` [5m]
)
) > 0
for: 0s
labels:
component: logs
severity: critical
category: security
annotations:
runbook_url: "https://github.com/Gerrrt/HomeLab/blob/main/docs/runbooks/respond-to-a-security-alert.md#rangesubnetleaked"
summary: "Range leak: a 172.30.30.x source was logged ({{ $labels.action }}) on {{ $labels.interface }}"
description: >-
A packet from the range's target subnet reached the firewall.
ADR-0017 gives that bridge no physical port and no gateway, so it
got out through a guest that forwarded it. Either ragnarok has
forwarding on, or a target has a NIC on vmbr0. Unless this is
build-the-playground.md §9 being run on purpose, treat it as an
incident: power gandalf off, then find which.

- alert: IoTAttemptedLateralMovement
# The mirror image: correctly BLOCKED attempts out of the IoT segment.
# The control worked. The point is that a camera or a $20 Tuya device
Expand Down
Loading
Loading