feat(bindings): an offline-first demo on three devices, and OP_GATEWAY in the image

CHANGELOG.md

@@ -115,6 +115,24 @@ archived by series under [docs/changelog/](docs/changelog/); see the
   is removed, the control); a message crosses a middle device to one the
   sender cannot hear, and the recipient's receipt comes back the same way.
 
+- **An offline-first demo on three devices.** `examples/offline-first` runs
+  three services from the container image on two bridge networks, so the
+  middle one is the only way between the other two, and `run.py` runs the
+  scenarios with `offline-protocol-verify` and prints a table: store and
+  forward, the sender killed and restarted while its message is queued, and
+  a message carried through the middle device. Its README is the runbook for
+  the legs that need hardware (the LAN between hosts, the carrier changing
+  under a stream of messages, the relay, a gateway daemon, a phone), with a
+  record of what has been run: scenarios 1, 2 and 4 in containers on one
+  machine, nothing on hardware yet. Scenario 4 fails unless the sender's
+  receipt comes back across the hop. The image gains `OP_GATEWAY` (the
+  gateway daemon, for a configuration with `reticulum_enabled`), two baked
+  configurations beside the default (`config-ble.json`, `config-relay.json`),
+  and the verifier, and its build fails when the installed package has none.
+  Found while running it: a message that a direct stream took and then lost
+  (a device that went away without closing the stream) is retried over
+  direct carriers only and never handed to the mesh (#541).
+
 ### Changed
 
 - **A peer-stream or relay flag the configuration cannot honour is refused.**

bindings/python/docker/Dockerfile

@@ -27,9 +27,13 @@ RUN set -eu; \
     pip install --no-cache-dir --find-links /wheels "${SDK_SPEC}"; \
     rm -rf /wheels; \
     { offline-protocol-service --help | grep -q -- '--http ' && python -c 'import aiohttp, zeroconf'; } \
-        || { echo "the installed offline-protocol-service has no HTTP front: put a wheel built from a checkout in wheels/" >&2; exit 1; }
+        || { echo "the installed offline-protocol-service has no HTTP front: put a wheel built from a checkout in wheels/" >&2; exit 1; }; \
+    command -v offline-protocol-verify >/dev/null \
+        || { echo "the installed package has no offline-protocol-verify: put a wheel built from a checkout in wheels/" >&2; exit 1; }
 
-COPY config.json /etc/offline-protocol/config.json
+# The baked configuration, and two to mount or name with OP_CONFIG: one
+# with Bluetooth LE on, one with the internet relay on.
+COPY config.json config-ble.json config-relay.json /etc/offline-protocol/
 COPY entrypoint.sh /usr/local/bin/offline-protocol-entrypoint
 RUN chmod 0755 /usr/local/bin/offline-protocol-entrypoint \
     && mkdir -p /var/lib/offline-protocol

bindings/python/docker/README.md

@@ -31,8 +31,19 @@ the service these, and the failure each one prevents:
 `config.json` is the engine's `ProtocolConfig`, baked into the image; mount
 another at `/etc/offline-protocol/config.json` or point `OP_CONFIG` at one.
 The default enables the peer stream with encryption required, and leaves
-Bluetooth LE and the internet relay off. `profile` is this host's label and
-part of its storage namespace: change it before the first start, never after.
+Bluetooth LE and the internet relay off. Two more are baked beside it and
+differ from it in one field each: `OP_CONFIG=/etc/offline-protocol/config-ble.json`
+turns Bluetooth LE on (and needs the D-Bus grant above), and
+`config-relay.json` turns the internet transport on for `OP_RELAY`. `profile`
+is this host's label and part of its storage namespace: change it before the
+first start, never after.
+
+The image also has `offline-protocol-verify`, which drives the service in
+the same container and waits for the events that prove a message was held,
+carried and delivered: `docker exec <container> offline-protocol-verify
+--socket /run/offline-protocol/api.sock state`. The
+[offline-first demo](../../../examples/offline-first) runs its scenarios
+with it.
 
 `entrypoint.sh` maps environment variables to flags:
 
@@ -47,6 +58,7 @@ part of its storage namespace: change it before the first start, never after.
 | `OP_HTTP_TOKEN_FILE` | none | `--http-token-file`: the front writes a per-launch token there and requires it; needed off loopback, and on a host where a browser runs (R23 in the threat model). The file is replaced at every start and readable by the container's user only: bind-mount its directory, never the file, and read it as that user |
 | `OP_HTTP_ALIASES` | none | `--http-aliases` |
 | `OP_RELAY` | none | `--relay`, token in `OFFLINE_PROTOCOL_RELAY_TOKEN`; the config must set `internet_enabled`, which the default leaves off, or the service refuses to start |
+| `OP_GATEWAY` | none | `--gateway`, a gateway daemon's `HOST:PORT`; the config must set `reticulum_enabled`, or the service refuses to start |
 | `OP_SOCKET` | `/run/offline-protocol/api.sock` | `--socket` |
 
 Arguments after the image name come after every flag the environment sets,
@@ -73,7 +85,8 @@ docker build bindings/python/docker
 
 Wheels for several architectures may sit there together; pip takes the one
 that matches the image. The build fails if the installed service has no HTTP
-front, which is the case for every release before the `http` extra.
+front or the package has no `offline-protocol-verify`, which is the case for
+every release so far.
 
 The build context is this directory only. Never build from the repository
 root or mount the checkout: `target/` alone fills the build VM.

bindings/python/docker/config-ble.json

@@ -0,0 +1,19 @@
+{
+  "app_id": "offline-protocol-service",
+  "profile": "device",
+  "ble_enabled": true,
+  "wifi_direct_enabled": true,
+  "internet_enabled": false,
+  "reticulum_enabled": false,
+  "nostr_enabled": false,
+  "prefer_online": false,
+  "initial_ttl": 5,
+  "encryption_enabled": true,
+  "require_encryption": true,
+  "auto_key_exchange": true,
+  "store_pending": true,
+  "max_pending_per_peer": 100,
+  "max_pending_global": 1000,
+  "pending_ttl_ms": 604800000,
+  "overflow_policy": "DropOldest"
+}

bindings/python/docker/config-relay.json

@@ -0,0 +1,19 @@
+{
+  "app_id": "offline-protocol-service",
+  "profile": "device",
+  "ble_enabled": false,
+  "wifi_direct_enabled": true,
+  "internet_enabled": true,
+  "reticulum_enabled": false,
+  "nostr_enabled": false,
+  "prefer_online": false,
+  "initial_ttl": 5,
+  "encryption_enabled": true,
+  "require_encryption": true,
+  "auto_key_exchange": true,
+  "store_pending": true,
+  "max_pending_per_peer": 100,
+  "max_pending_global": 1000,
+  "pending_ttl_ms": 604800000,
+  "overflow_policy": "DropOldest"
+}

bindings/python/docker/entrypoint.sh

@@ -15,6 +15,8 @@
 #   OP_HTTP_ALIASES             device alias file for the front
 #   OP_RELAY                    internet relay URL, with a config that sets
 #                               internet_enabled; its token in OFFLINE_PROTOCOL_RELAY_TOKEN
+#   OP_GATEWAY                  gateway daemon HOST:PORT, with a config that sets
+#                               reticulum_enabled
 #   OP_SOCKET                   local API socket (default /run/offline-protocol/api.sock)
 #
 # Any arguments come after every flag the environment sets, so an explicit
@@ -66,6 +68,9 @@ fi
 if [ -n "${OP_RELAY:-}" ]; then
     set -- "$@" --relay "$OP_RELAY"
 fi
+if [ -n "${OP_GATEWAY:-}" ]; then
+    set -- "$@" --gateway "$OP_GATEWAY"
+fi
 
 # The operator's arguments last: the service keeps the last occurrence of a
 # flag, so `docker run IMAGE --http 0.0.0.0:8080 ...` would otherwise lose to

examples/offline-first/.gitignore

@@ -0,0 +1,2 @@
+# The per-device store keys run.py writes: secret, and per checkout.
+.env

examples/offline-first/README.md

@@ -0,0 +1,136 @@
+# Offline first: what the network does when a device is not there
+
+A request with a deadline needs both devices on at once. A message does
+not: the sender holds it on disk until the recipient can be reached, by
+whatever carrier reaches it, through whichever devices are in between, and
+the recipient's own acknowledgement comes back as `message_delivered`. These
+scenarios show that, and each one passes or fails on the engine's events,
+read on the device by
+[`offline-protocol-verify`](../../docs/local-api.md#checking-what-the-network-did-offline-protocol-verify).
+
+| # | Scenario | Passes when |
+|---|---|---|
+| 1 | Store and forward: B off, A sends, B on | B receives it, and A gets `message_delivered` within 60 s of B coming back |
+| 2 | The sender restarts: B off, A sends, A killed (`SIGKILL`) and started, B on | the same, from the restarted A: the queue was on disk |
+| 3 | The carrier changes: the link A and B were using goes away | every message of a `ping` run is delivered, and `message_delivered.transport` names the new carrier (hardware only, below) |
+| 4 | Through the middle: A and C meet once, then only B hears both | C receives A's message at `hop_count` 1 within 30 s, B reports `message_relayed`, and A gets `message_delivered` back the same way |
+
+## In containers, on one machine
+
+```
+a ---- net-ab ---- b ---- net-bc ---- c
+```
+
+[`compose.yml`](compose.yml) runs three devices from the
+[service image](../../bindings/python/docker), each with its own identity
+on its own volume. a and c share no network, so b is the only way between
+them. Peers are static, so the topology is the file's and not multicast
+DNS's.
+
+```bash
+python3 run.py            # builds the image, starts the devices, runs 1, 2 and 4
+python3 run.py --fresh    # new identities first
+python3 run.py --scenario 4
+```
+
+The image installs the package from PyPI unless a Linux wheel built from a
+checkout is in `bindings/python/docker/wheels/`, and its build fails when
+the installed package has no `offline-protocol-verify`, which is true of
+every release so far: until one ships, put a wheel there, or build the image
+another way and pass `--no-build` to use `offline-protocol-service:offline-first`
+as it is. `run.py` writes `.env` with one store key per device the first
+time; keep it, or the devices come back with new addresses. It prints each
+scenario as it finishes and a table at the end, and exits non-zero if any
+failed.
+
+What `run.py` does, so each step can be run by hand with `docker exec
+offline-first-<device> offline-protocol-verify --socket /run/offline-protocol/api.sock ...`:
+
+1. `pair` A and B (the session forms by itself once they hear each other),
+   `docker stop` B, `send` from A, start `await <id> --until delivered` on
+   A, `docker start` B, then `await <id> --until received` on B.
+2. The same with `docker kill --signal KILL` and `docker start` on A between
+   the send and B's return.
+4. `docker network connect` C to net-ab and `pair` A and C; `docker stop` C,
+   disconnect it, `docker start` it; then `watch` on A and B, `send` from A
+   to C, and `await --until received` on C.
+
+Two things in that order matter, and both are the engine's rules rather
+than the script's. A receipt that fires while no client is connected is not
+held, so the wait on the sender starts before the recipient can answer. And
+C is stopped before it leaves net-ab: taken off the network first, it would
+leave A a stream that looks open for about 30 s, and a message A sent down
+it would never be handed to B (see the known gaps).
+
+## On hardware
+
+`run.py --ssh a=user@host --ssh b=user@host --ssh c=user@host` runs the same
+checks over ssh against hosts that already run the service, and asks the
+operator to switch devices off and on, and to move a and c apart, at each
+step. Each host runs the [service image](../../bindings/python/docker) under
+host networking with `OP_LISTEN` set to its own LAN address, or the service
+directly. The verifier has to run where the service's socket is: for the
+image that is inside the container, so add `--remote-exec "docker exec
+<container>"` (`docker-offline-protocol-1` for the image's `compose.yml`
+started from its own directory); for the service run directly, add
+`--socket` with the path it was started with.
+
+The legs that need real radios or more than one machine, and how to run
+each:
+
+- **LAN between hosts.** Two or three boxes on one segment, `OP_LAN=1` so
+  they find each other over DNS-SD. Scenarios 1, 2, 4 as above; for 4, put
+  C on a second segment that only B joins.
+- **The carrier changes (3).** Two boxes with `OP_CONFIG=/etc/offline-protocol/config-ble.json`
+  and the D-Bus grant, so both peer streams and Bluetooth LE are up. Start
+  `offline-protocol-verify ping <B> --every 2 --count 60` on A, then pull
+  the cable (or `ip link set <iface> down`) on B. Expect the messages in
+  flight to arrive about 30 to 90 s late over `ble` (keepalive ends the dead
+  stream in about 30 s, then the next retry picks the carrier that is up),
+  and later ones over `ble` at once. Plug it back in to see `wifiDirect`
+  return.
+- **Relay.** The relay server with a Postgres database and authentication
+  off for a closed test, `config-relay.json` and `OP_RELAY=ws://<relay>:3000/ws`
+  on each box, and no peer stream between them (`OP_LAN=0` and no
+  `OP_PEERS`, or two networks): a live direct link to the recipient is
+  tried before any other carrier, so on one LAN the message comes back over
+  `wifiDirect` and proves nothing about the relay. The receipt names
+  `internet` when it does. Scenario 1 twice: once with A also off when B
+  returns (the relay's mailbox holds the frame), once with the relay
+  unreachable from A (A's outbox holds it). `message_undeliverable` is
+  printed as status on the way: it is the relay saying B is away now, not a
+  failure.
+- **Reticulum.** A gateway daemon and `rnsd` on each of two boxes with a
+  backbone between them (a TCP interface, or a pair of RNodes), a config
+  with `reticulum_enabled` and `OP_GATEWAY=127.0.0.1:4242`, and no peer
+  stream between the boxes, as for the relay. Check the attach
+  first (the transport comes up only after the daemon's capabilities), then
+  scenario 1. The service's gateway client has not met a real daemon yet.
+- **A phone.** The React Native example app with Bluetooth LE on, against a
+  box running `config-ble.json`: the box's `await --until received`, and
+  the app's own delivered state, both directions. An iPhone can also reach
+  a box over the LAN peer stream on the same Wi-Fi.
+
+## Known gaps
+
+- **A message a direct link took and then lost never crosses the mesh**
+  (#541): a message sent down a stream to a device that went away without
+  closing it waits for a direct link to that device.
+- **One phone per Linux box over Bluetooth LE.** The box's peripheral learns
+  which phone wrote to it, but it maps a phone to its user id only while that
+  phone is the one central connected, so with two a reply has no route back
+  through the peripheral.
+- **An image built from 0.28.0 or earlier** never gets the receipt in
+  scenario 4: the engine drops an acknowledgement for a message whose only
+  route was the mesh (#537). Pass `--allow-missing-hop-receipt` to run the
+  rest of the scenario on such an image.
+
+## What has been run
+
+| When | Where | Scenarios | Result |
+|---|---|---|---|
+| 2026-10-08 | `run.py`, Docker 28.3 on one arm64 laptop, the image built from the 0.28.0 Linux wheel with this branch's Python sources, so an engine without #537 | 1, 2, 4 | pass, twice in a row (receipt latency 29 to 45 ms; 4 without A's receipt, which that engine never sends) |
+| 2026-10-08 | `run.py --fresh` then `run.py`, the same machine, the image with the native library built from `main` after #537 and this branch's Python sources | 1, 2, 4 | pass, twice in a row, A's receipt required in 4 and back at hop 1 (latency 33 to 36 ms in 1 and 2) |
+
+Nothing on this page has been run between separate hosts, over Bluetooth
+LE, over a relay, through a gateway daemon, or with a phone.

examples/offline-first/compose.yml

@@ -0,0 +1,61 @@
+# Three devices on one machine, for the offline-first scenarios in README.md.
+#
+#   a ---- net-ab ---- b ---- net-bc ---- c
+#
+# a and c share no network, so b is the only way between them; Docker keeps
+# separate bridge networks apart. Each device is the service image from
+# bindings/python/docker with its own identity on its own volume. No HTTP
+# front: the scenarios use the local API through offline-protocol-verify.
+#
+# Peers are static (a and c dial b, c also dials a when the scenario puts it
+# on net-ab), so the topology is decided here and not by multicast DNS.
+#
+# run.py writes .env with one store key per device the first time; keep it,
+# or the devices come back with new addresses.
+x-device: &device
+  build: ../../bindings/python/docker
+  image: offline-protocol-service:offline-first
+  environment: &environment
+    OP_LAN: "0"
+    OP_HTTP: ""
+    OP_LISTEN: 0.0.0.0:7878
+
+services:
+  a:
+    <<: *device
+    container_name: offline-first-a
+    hostname: a
+    environment:
+      <<: *environment
+      OFFLINE_PROTOCOL_STORE_KEY: ${KEY_A:?run run.py once, or set KEY_A}
+      OP_PEERS: b:7878
+    networks: [net-ab]
+    volumes: [data-a:/var/lib/offline-protocol]
+  b:
+    <<: *device
+    container_name: offline-first-b
+    hostname: b
+    environment:
+      <<: *environment
+      OFFLINE_PROTOCOL_STORE_KEY: ${KEY_B:?run run.py once, or set KEY_B}
+    networks: [net-ab, net-bc]
+    volumes: [data-b:/var/lib/offline-protocol]
+  c:
+    <<: *device
+    container_name: offline-first-c
+    hostname: c
+    environment:
+      <<: *environment
+      OFFLINE_PROTOCOL_STORE_KEY: ${KEY_C:?run run.py once, or set KEY_C}
+      OP_PEERS: b:7878 a:7878
+    networks: [net-bc]
+    volumes: [data-c:/var/lib/offline-protocol]
+
+networks:
+  net-ab:
+  net-bc:
+
+volumes:
+  data-a:
+  data-b:
+  data-c: