feat: storage radius check

[akrem-chabchoub]

feat: implement storage-radius

Summary

Implements the storage-radius check for testing Bee's storage radius increase/decrease behavior. The check fills a cluster's reserves with random data until Bee raises its storage radius, then dilutes the postage batches to verify the radius decreases.

This is a foundational test for Bee's reserve management and eviction logic.

Changes

New Check: pkg/check/storageradius/

Implements the complete storage-radius check with:

• Random chunk upload with deferred delivery tracking via /debugstore
• Postage batch management — creation, reuse by label, and utilization validation
• Storage radius increase verification — waits for radius bump after reserves exceed capacity
• Batch dilution and decrease verification — dilutes batches to depth 32, monitors for radius drop
• Pipeline monitoring — tracks reserve size and pusher backlog growth from baseline
• Cluster state logging — before/after reserve snapshots with per-node breakdown

Key design choices:

• Uses WaitGroup (not errgroup) for batch preparation to avoid 900s hangs on chain reverts
• Measures upload progress from baseline (handles pre-filled clusters)
• Stops uploads when pipeline holds enough chunks, avoiding overshoots
• Skips pusher-idle wait when reserves are 95%+ full (they'll evict backlog on arrival)
• Requires 5 consecutive stable polls to conclude pushers are done

API Enhancements

pkg/bee/api/status.go

• Add IsWarmingUp bool field to detect nodes still in warmup phase

pkg/bee/api/debugstore.go

• Replace flat map[string]int with proper struct for /debugstore response
• Decode nested JSON: upload, pinning, cache, reserve, chunkStore stats
• UploadStat.PendingUpload is the key metric for tracking undelivered chunks

pkg/bee/api/api.go

• Register DebugStoreService in the API client (was never wired up)

Config: config/testnet-bee-playground.yaml

Bee config

• Add payment-threshold: 108000000 (max Bee accepts; prevents accounting lock contention)

Check definition: pg-storage-radius

reserve-capacity: 4000              # requires patched Bee image
target-fill-percent: 1.03           # 3% over capacity to trigger radius
stamps: 8                           # 1 batch per node (8-node cluster)
chunks-per-upload: 512              # ~2 MB per request (avoids proxy timeout)
postage-depth: 22                   # > Bee's bucket depth of 16
postage-amount: 2073600000          # clears minimum validity floor
upload-timeout: 10m                 # per-upload timeout
upload-wave-pause: 5s               # let watcher poll between waves
poll-interval: 2s                   # check frequency
min-radius-wait: 5m                 # minimum radius increase observation time
pushers-idle-wait: 2m               # max wait for backlog to settle before dilution
dilute-depth: 32                    # max Bee depth
dilute-wait: 45m                    # covers 3× 15m reserve worker ticks
timeout: 80m                        # total check timeout

Prerequisites

1. Patched Bee Image

The check requires a custom Bee build with 4 edits (see guide for details):

// pkg/storer/storer.go
DefaultReserveCapacity = 4000       // was: 1 << 22 (4.2M chunks)
ReserveWakeUpDuration: time.Second * 10  // was: 30 minutes

// pkg/storer/reserve.go
func threshold(capacity int) int { return capacity * 8 / 10 }  // was: 5 / 10

// cmd/bee/cmd/cmd.go
time.Second*120                     // was: 5 minutes (warmup shortening)

Build and push:

cd ../bee
make docker-build BEE_IMAGE=<your-registry>/bee:reserve-4000

Then update the node-group image in beekeeper config.

2. Payment Threshold (§2.2)

✅ Already added: payment-threshold: 108000000

Prevents failed to acquire lock when preparing credit during bulk uploads.

3. Ingress Proxy (§2.3)

k3d ships Traefik (not nginx) with ~60s default timeout. Monitor logs for HTTP 499/502 errors during uploads. The check uses 10m timeout but may still hit proxy limits on very slow clusters.

4. Postage Amount (§2.4)

✅ Already set: 2073600000

Clears minimum validity floor: amount > currentPrice × minimumValidityBlocks

Testing

# Build and run the check
make build && make vet && make lint && make test

# Run the check against bee-playground
./dist/beekeeper check \
  --cluster-name=bee-playground \
  --checks=pg-storage-radius \
  --timeout=90m \
  --log-verbosity=info

Cross-check on a node:

curl -s localhost:1633/status | jq '{reserveSize, reserveSizeWithinRadius, storageRadius}'
curl -s localhost:1633/debugstore | jq '.upload'

In Bee's logs, look for:

• unreserve start followed by reserve radius increase → working correctly
• unreserve finished → eviction met target, radius didn't move (chunks were evictable)

Related

See FILL_RESERVE_GUIDE.md for:

• Complete design rationale (Bee's reserve mechanics, why design choices matter)
• Step-by-step rebuild guide from scratch
• Appendix of mistakes worth not repeating (13 items learned during development)

Notes

• This implements the foundational storage-radius check only
• Requires Go 1.26+