workqueue-lab

Watch client-go's workqueue.RateLimitingInterface work under real load: deduplication, the dirty/processing set pair (the internal locking that makes concurrent workers safe), and exponential backoff — first in isolation, then at the end of a real controller pipeline.

📖 New to workqueues? Read HOW-IT-WORKS.md first — the two-sets state machine and the backoff math as diagrams, with real log lines mapped onto them.

Two modes:

mode	what it does
--mode=direct	pure queue mechanics. A generator floods the queue with a tiny key pool (default 50,000 enqueues/sec onto 20 keys) so dedup and requeue-while-processing fire constantly. No cluster needed.
--mode=pipeline	the realistic path: k6/curl → HTTP /trigger → real kube-apiserver write (envtest) → etcd → Cacher fan-out → SharedInformer → queue.Add → workers.

Files: queue.go (the observable queue + workers — start here), mode_direct.go, mode_pipeline.go, main.go (flags + stats).

Prerequisites

• Go ≥ 1.23
• pipeline mode only: envtest binaries (etcd + kube-apiserver). If you ran informer-lab they're already installed; otherwise:

  go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
  setup-envtest use 1.31.x -p path   # one-time download; auto-detected afterwards

Mode 1: direct (pure workqueue mechanics)

go build -o workqueue-lab .

# full-rate: watch dedup collapse half a million Adds
./workqueue-lab --mode=direct

# slow-motion: read every single line — best for learning
./workqueue-lab --mode=direct --rate=30 --keys=3 --work-ms=300 --fail-rate=0.3 --log-sample=1

Ctrl-C to stop. What to look for:

signature	proves
[ENQUEUE] → [DEDUP] "already waiting in the FIFO"	an Add for an already-queued key is a no-op
[REQUEUE-WHILE-PROCESSING] → DEDUP×N → [REDELIVERY] → Get seq=N+1	N Adds while a worker holds the key → exactly one redelivery at Done()
[RATELIMIT] attempt=1 backoff=5ms → attempt=2 backoff=10ms → attempt=3 backoff=20ms	per-key exponential backoff doubling on consecutive failures
result=OK → Forget (backoff reset) then next failure back at attempt=1 backoff=5ms	Forget on success resets the counter
[STATS] adds-absorbed=99.2%	the funnel: 50k Adds/s → ~920 processings/s
[STATS] queue.Len stays ≈ # of hot keys	dedup caps queue depth at key cardinality, not event rate

Same key processed by different workers, but never at the same time — watch seq=N climb per key while the worker id varies. That's the processing set doing your locking for you.

Mode 2: pipeline (k6 → apiserver → informer → queue)

./workqueue-lab --mode=pipeline            # boots envtest (~3s), serves :8080

Trigger endpoint contract (write your k6 script against this)

method + path	POST /trigger/{name} — no body needed
valid names	obj-0 … obj-(N-1), N = --keys (default 20)
what it does	real apiserver write: create the ConfigMap default/{name} if absent, else update it (data.hits increments)
success	200 + {"name":"obj-7","action":"created|updated","resourceVersion":"197"}
bad name	400 + {"error":"name must be obj-N with 0 <= N < 20"}
health	GET /healthz → ok

Sanity check without k6:

curl -X POST localhost:8080/trigger/obj-3
# hammer it: seq 1 500 | xargs -P 20 -I{} curl -s -X POST localhost:8080/trigger/obj-$(({} % 20)) -o /dev/null

Every trigger flows the full path — visible as [TRIGGER] → [ENQUEUE] (informer: MODIFIED) → [WORKER-N] Get/Done. The [STATS] line gains latency percentiles for the endpoint:

[STATS] ... | /trigger reqs=203 p50=1.94ms p95=2.53ms p99=3.10ms (window=203)

Tip: at low load with fast workers, dedup stays at 0% (the queue drains instantly). To make dedup appear in pipeline mode, slow the workers: --work-ms=200 and drive hard.

Log prefix legend

prefix	meaning
[SETUP]	lifecycle (flags, envtest, endpoint, shutdown)
[ENQUEUE]	an external Add() — source in parens: (generator) or (informer: ADDED/MODIFIED/DELETED)
[DEDUP]	Add was a no-op: key already waiting, or already marked for redelivery
[REQUEUE-WHILE-PROCESSING]	Add hit a key a worker currently holds → redelivered once at Done()
[REDELIVERY]	that redelivery actually happening (watched key only)
[WORKER-N]	worker Get/Done with per-key seq number
[RATELIMIT]	AddRateLimited with computed backoff + attempt count
[TRIGGER]	pipeline mode: one HTTP request → one apiserver write
[STATS]	exact totals every 2s (log lines are sampled; stats are not)

Logging fidelity: key default/obj-0 is logged in full; everything else is sampled 1 in --log-sample. --log-sample=1 logs everything.

Flags

flag	default	meaning
--mode	direct	direct | pipeline
--rate	50000	[direct] enqueues/sec
--keys	20	[direct] key cardinality · [pipeline] ConfigMap pool size
--workers	5	worker goroutines
--work-ms	5	simulated reconcile time per item
--fail-rate	0.10	per-item failure probability (drives AddRateLimited)
--port	8080	[pipeline] HTTP port for /trigger
--log-sample	1000	1-in-N sampling for high-volume log lines

Why the workqueue is rarely your bottleneck

Mode 2's own numbers (this machine, 203 requests at 10-way concurrency):

• apiserver write: p50 ≈ 1.9ms, p99 ≈ 3.1ms — network, authn/authz, admission, etcd transaction
• informer → enqueue: ~60µs — Cacher fan-out + decode
• queue Add/Get/Done: ~1µs — two map writes and a slice append, in memory

The queue is ~1000× cheaper than the work on either side of it. Under k6 load you will see /trigger p99 climb (apiserver/etcd write contention on the same 20 objects) long before queue.Len or queue latency becomes interesting. The only queue-side thing worth watching in production is depth (queue.Len in [STATS]): if workers are slower than the deduplicated arrival rate, depth grows — and the fix is faster reconciles or more workers, not a faster queue. Collect your own numbers with k6 and compare: the bottleneck is the write path and your reconcile logic, essentially never the queue.

Related

• informer-lab — the server half of mode 2: LIST→WATCH handoff and Cacher fan-out, decoded on the wire.