Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,25 @@ docstring gives: it is a record, not a claim about now.

## 2026-11-09

- **[#574](https://github.com/Gerrrt/HomeLab/issues/574): the current-state
documents now say the UPS shutdown is armed and not proved.**
`security.md`'s mains-loss row still said that as of 2026-09-20 nothing
shuts down on the UPS's signal. That stopped being true on 2026-09-23, when
steps 0 to 4 of
[`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) put NUT on
`morpheus` and subscribed `Saruman` and `smaug`. The row now says the
sequence is armed, `trinity` is not subscribed yet, and the pack is
unmeasured. `network.md` had never listed the two NUT pass/block pairs
built that day: `Allow NUT from Saruman` on `igc0.30` and `Allow NUT from
smaug` on `igc0.40`, each above a `Block NUT to pfSense` for the rest of
its segment, both to the gateway's own address on `3493`. Both are now in
their segment's notes. Read over the TrueNAS API the same day: `smaug`
reports no active alerts, and its UPS reporting graphs hold no data
points. That is expected for a NUT secondary, which runs no local `upsd`
for the graphs to read, so it says nothing about whether the subscription
is up. The rack visit in runbook §5 to §7 still owes the forced-shutdown
proof and the mains pull.

- **The in-house ntfy topics are renamed so a phone can tell them apart.**
The iOS ntfy app shows a subscription by its topic name and has no display
name ([binwiederhier/ntfy#1314](https://github.com/binwiederhier/ntfy/issues/1314)),
Expand Down
21 changes: 21 additions & 0 deletions docs/network.md
Original file line number Diff line number Diff line change
Expand Up @@ -435,6 +435,18 @@ Televisions and consoles. Internet only.
2026-10-03 by [`build-the-backup-guest.md`](runbooks/build-the-backup-guest.md)
§4, directly above the same block. `golem` has mounted the share through
it, and `alexander` is refused.
- **One pair lets `smaug` initiate, and only to its own gateway.**
`Allow NUT from smaug`, `10.0.40.30 → 10.0.40.1:3493`, sits above
`Block NUT to pfSense`, CasaBonita net → `10.0.40.1:3493`, and both sit
above *Allow internet* on `igc0.40`, in the shape of the existing *Block SSH
to pfSense*. They were created on 2026-09-23 by
[`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) §2.3, so that
TrueNAS's UPS service can subscribe to the firewall's NUT server
([ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md)).
The destination is the firewall's address on this segment, not another
segment, so the *Reaches* column is unchanged. The pass narrows rather than
opens: before it, *Allow internet* already passed the whole segment to that
address, and the block is what keeps the televisions off the listener.
- **Three rules from `trinity` to `mediatv`, in force since 2026-10-19:**
`10.0.99.40 → 10.0.40.40`, passing `3001/tcp` for Home Assistant's LG
webOS integration and `9/udp` for the magic packet that wakes the TV, and
Expand Down Expand Up @@ -523,6 +535,15 @@ Where things get broken on purpose.
([ADR-0053](adr/0053-run-pbs-on-saruman-with-its-datastore-on-smaug-over-nfs.md)).
[`build-the-backup-guest.md`](runbooks/build-the-backup-guest.md) §4
asked for this block to be named here.
- **`Saruman` reaches its own gateway for NUT, and nothing else on the segment
does.** `Allow NUT from Saruman`, `10.0.30.20 → 10.0.30.1:3493`, sits above
`Block NUT to pfSense`, ImaginationLAN net → `10.0.30.1:3493`, both above
*Allow internet* on `igc0.30`. They were created on 2026-09-23 by
[`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) §2.1–2.2
([ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md))
with `.110` as the source, and replaced by a `.20` copy when the hypervisor
moved on 2026-11-07. The block is what covers the lab's guests, the attack
VM included.
- `shiva` and `Saruman` are the same physical box: `shiva` is the iLO BMC on its
dedicated port, `Saruman` is the Proxmox install. They are separate addresses
and separate names, and conflating them is a mistake this document previously
Expand Down
2 changes: 1 addition & 1 deletion docs/runbooks/shut-down-on-the-ups.md
Original file line number Diff line number Diff line change
Expand Up @@ -1197,7 +1197,7 @@ this work exists to catch.
| This file's status block | Built on *date*; sequence takes *N*; pack measured at *M* minutes at *L* % load |
| [`hardware.md`](../hardware.md) Rack paragraph | Replace "not measured" with the measured number |
| [`hardware.md`](../hardware.md) Accessories, UPS entry | The low battery duration you set in 6.5, and that 0.1 read 8 minutes before any change here (factory value not observed). Also the card's second SNMPv3 access-control entry, `10.0.99.1` read-only, from 1.0 |
| [`security.md`](../security.md) mains-loss row | It no longer says nothing shuts down on the signal |
| [`security.md`](../security.md) mains-loss row | It says the sequence is proved, not armed and unproved, and gives the measured runtime in place of the card's estimate |
| [`fit-the-ups-battery.md`](fit-the-ups-battery.md) §5 | Whether the card's runtime series tracked your measurement |
| §1.4 and §3.7 of this file | `HOSTSYNC` stays at 600 as long as 3.6's bound is 9½ minutes. Lower it only if the bound itself falls. A measured run shorter than the bound is not a reason to lower it |
| [`hardware.md`](../hardware.md) outlet-group paragraph | The three groups' turn-off, turn-on and load-shedding settings read under *Before you start* |
Expand Down
2 changes: 1 addition & 1 deletion docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ What this network is actually built to survive:
| Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) |
| Losing visibility of a failure | 179 alert rules, 30 days of metrics and logs |
| Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana |
| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What the UPS cannot answer is what happens when the cut outlasts the pack: as of 2026-09-20 nothing shuts down on its signal, and everything on the PDU — `morpheus`, `Saruman`, `neo` and `smaug`, and since the move into the 32U rack on 2026-11-06 also `trinity`, `gandalf` (off until its build) and the console, with `smaug` racked and on the PDU directly rather than by its old long cord to the media room — stops uncleanly when the pack empties, about 47 minutes in at 21 % load by the card's own unmeasured estimate.** [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides that the firewall's NUT server halts `Saruman` and `smaug` first and itself last, [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) adds `trinity` to the first group and leaves `gandalf` and the console to the pack, and [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) is the build, the forced-shutdown proof and the one mains pull that measures the pack; until those are done the decision is a configuration nobody has tested ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) |
| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What happens when the cut outlasts the pack is armed and not proved.** Since 2026-09-23 the firewall's NUT server reads `mjolnir`, and `Saruman` and `smaug` subscribe to it on their own gateway addresses, so a real cut would now try to run the sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides: `Saruman` and `smaug` halt first and `morpheus` last. Nobody has watched it run. Since the move into the 32U rack on 2026-11-06 the PDU also carries `trinity`, `gandalf` (off until its build) and the console, with `smaug` racked and on the PDU directly rather than by its old long cord to the media room. [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) makes `trinity` a third subscriber and bounds `Saruman`'s guest stop, and neither is applied yet, so `trinity` still stops uncleanly when the pack empties; `gandalf`, the console and `neo` are left to the pack by decision. The pack itself is unmeasured: about 47 minutes at 21 % load is the card's own estimate. [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) §5 to §7 are what is left — the forced-shutdown proof and the one mains pull that measures the pack ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) |
| The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet off the estate.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data. That tier has held real photographs since 2026-09-28, and their off-estate copy is built but has no holder yet ([#455](https://github.com/Gerrrt/HomeLab/issues/455)) — see below |

What it explicitly does **not** defend against: a determined attacker with
Expand Down
Loading