diff --git a/docs/changelog.md b/docs/changelog.md index 366c2432..b8327181 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -19,6 +19,25 @@ docstring gives: it is a record, not a claim about now. ## 2026-11-09 +- **[#574](https://github.com/Gerrrt/HomeLab/issues/574): the current-state + documents now say the UPS shutdown is armed and not proved.** + `security.md`'s mains-loss row still said that as of 2026-09-20 nothing + shuts down on the UPS's signal. That stopped being true on 2026-09-23, when + steps 0 to 4 of + [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) put NUT on + `morpheus` and subscribed `Saruman` and `smaug`. The row now says the + sequence is armed, `trinity` is not subscribed yet, and the pack is + unmeasured. `network.md` had never listed the two NUT pass/block pairs + built that day: `Allow NUT from Saruman` on `igc0.30` and `Allow NUT from + smaug` on `igc0.40`, each above a `Block NUT to pfSense` for the rest of + its segment, both to the gateway's own address on `3493`. Both are now in + their segment's notes. Read over the TrueNAS API the same day: `smaug` + reports no active alerts, and its UPS reporting graphs hold no data + points. That is expected for a NUT secondary, which runs no local `upsd` + for the graphs to read, so it says nothing about whether the subscription + is up. The rack visit in runbook §5 to §7 still owes the forced-shutdown + proof and the mains pull. + - **The in-house ntfy topics are renamed so a phone can tell them apart.** The iOS ntfy app shows a subscription by its topic name and has no display name ([binwiederhier/ntfy#1314](https://github.com/binwiederhier/ntfy/issues/1314)), diff --git a/docs/network.md b/docs/network.md index ad5c9288..b45f7e82 100644 --- a/docs/network.md +++ b/docs/network.md @@ -435,6 +435,18 @@ Televisions and consoles. Internet only. 2026-10-03 by [`build-the-backup-guest.md`](runbooks/build-the-backup-guest.md) §4, directly above the same block. `golem` has mounted the share through it, and `alexander` is refused. +- **One pair lets `smaug` initiate, and only to its own gateway.** + `Allow NUT from smaug`, `10.0.40.30 → 10.0.40.1:3493`, sits above + `Block NUT to pfSense`, CasaBonita net → `10.0.40.1:3493`, and both sit + above *Allow internet* on `igc0.40`, in the shape of the existing *Block SSH + to pfSense*. They were created on 2026-09-23 by + [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) §2.3, so that + TrueNAS's UPS service can subscribe to the firewall's NUT server + ([ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md)). + The destination is the firewall's address on this segment, not another + segment, so the *Reaches* column is unchanged. The pass narrows rather than + opens: before it, *Allow internet* already passed the whole segment to that + address, and the block is what keeps the televisions off the listener. - **Three rules from `trinity` to `mediatv`, in force since 2026-10-19:** `10.0.99.40 → 10.0.40.40`, passing `3001/tcp` for Home Assistant's LG webOS integration and `9/udp` for the magic packet that wakes the TV, and @@ -523,6 +535,15 @@ Where things get broken on purpose. ([ADR-0053](adr/0053-run-pbs-on-saruman-with-its-datastore-on-smaug-over-nfs.md)). [`build-the-backup-guest.md`](runbooks/build-the-backup-guest.md) §4 asked for this block to be named here. +- **`Saruman` reaches its own gateway for NUT, and nothing else on the segment + does.** `Allow NUT from Saruman`, `10.0.30.20 → 10.0.30.1:3493`, sits above + `Block NUT to pfSense`, ImaginationLAN net → `10.0.30.1:3493`, both above + *Allow internet* on `igc0.30`. They were created on 2026-09-23 by + [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) §2.1–2.2 + ([ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md)) + with `.110` as the source, and replaced by a `.20` copy when the hypervisor + moved on 2026-11-07. The block is what covers the lab's guests, the attack + VM included. - `shiva` and `Saruman` are the same physical box: `shiva` is the iLO BMC on its dedicated port, `Saruman` is the Proxmox install. They are separate addresses and separate names, and conflating them is a mistake this document previously diff --git a/docs/runbooks/shut-down-on-the-ups.md b/docs/runbooks/shut-down-on-the-ups.md index a310a0d5..5414aada 100644 --- a/docs/runbooks/shut-down-on-the-ups.md +++ b/docs/runbooks/shut-down-on-the-ups.md @@ -1197,7 +1197,7 @@ this work exists to catch. | This file's status block | Built on *date*; sequence takes *N*; pack measured at *M* minutes at *L* % load | | [`hardware.md`](../hardware.md) Rack paragraph | Replace "not measured" with the measured number | | [`hardware.md`](../hardware.md) Accessories, UPS entry | The low battery duration you set in 6.5, and that 0.1 read 8 minutes before any change here (factory value not observed). Also the card's second SNMPv3 access-control entry, `10.0.99.1` read-only, from 1.0 | -| [`security.md`](../security.md) mains-loss row | It no longer says nothing shuts down on the signal | +| [`security.md`](../security.md) mains-loss row | It says the sequence is proved, not armed and unproved, and gives the measured runtime in place of the card's estimate | | [`fit-the-ups-battery.md`](fit-the-ups-battery.md) §5 | Whether the card's runtime series tracked your measurement | | §1.4 and §3.7 of this file | `HOSTSYNC` stays at 600 as long as 3.6's bound is 9½ minutes. Lower it only if the bound itself falls. A measured run shorter than the bound is not a reason to lower it | | [`hardware.md`](../hardware.md) outlet-group paragraph | The three groups' turn-off, turn-on and load-shedding settings read under *Before you start* | diff --git a/docs/security.md b/docs/security.md index 1573933f..6b59cb9c 100644 --- a/docs/security.md +++ b/docs/security.md @@ -26,7 +26,7 @@ What this network is actually built to survive: | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | | Losing visibility of a failure | 179 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | -| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What the UPS cannot answer is what happens when the cut outlasts the pack: as of 2026-09-20 nothing shuts down on its signal, and everything on the PDU — `morpheus`, `Saruman`, `neo` and `smaug`, and since the move into the 32U rack on 2026-11-06 also `trinity`, `gandalf` (off until its build) and the console, with `smaug` racked and on the PDU directly rather than by its old long cord to the media room — stops uncleanly when the pack empties, about 47 minutes in at 21 % load by the card's own unmeasured estimate.** [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides that the firewall's NUT server halts `Saruman` and `smaug` first and itself last, [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) adds `trinity` to the first group and leaves `gandalf` and the console to the pack, and [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) is the build, the forced-shutdown proof and the one mains pull that measures the pack; until those are done the decision is a configuration nobody has tested ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) | +| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What happens when the cut outlasts the pack is armed and not proved.** Since 2026-09-23 the firewall's NUT server reads `mjolnir`, and `Saruman` and `smaug` subscribe to it on their own gateway addresses, so a real cut would now try to run the sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides: `Saruman` and `smaug` halt first and `morpheus` last. Nobody has watched it run. Since the move into the 32U rack on 2026-11-06 the PDU also carries `trinity`, `gandalf` (off until its build) and the console, with `smaug` racked and on the PDU directly rather than by its old long cord to the media room. [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) makes `trinity` a third subscriber and bounds `Saruman`'s guest stop, and neither is applied yet, so `trinity` still stops uncleanly when the pack empties; `gandalf`, the console and `neo` are left to the pack by decision. The pack itself is unmeasured: about 47 minutes at 21 % load is the card's own estimate. [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) §5 to §7 are what is left — the forced-shutdown proof and the one mains pull that measures the pack ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet off the estate.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data. That tier has held real photographs since 2026-09-28, and their off-estate copy is built but has no holder yet ([#455](https://github.com/Gerrrt/HomeLab/issues/455)) — see below | What it explicitly does **not** defend against: a determined attacker with