Repository navigation
Finish ZeroKMS vs HSM comparison + add comparison diagrams #32
Description
Follow-up to #31, which migrated the ZeroKMS comparison content from cipherstash-js-suite PR #364. Some of it landed as work-in-progress and needs finishing.
1. Fill in the WIP TODOs in zerokms-vs-hsm.mdx
content/stack/reference/comparisons/zerokms-vs-hsm.mdx still carries the original draft's TODO markers:
- Benefits of HSM section — currently just
TODO - Operational complexity —
TODO: key ceremonies, replication, uptime (eg. power), reliability etc - Compliance section —
TODO: Talk about PCI/DSS etc - Physical access "ZeroKMS Advantage" callout —
TODO: this could be worded better - Cryptographic custody / distributed trust link —
TODO(may warrant its own "Trust Model" article)
2. Replace the diagram placeholder
The "Hybrid solutions" section of zerokms-vs-hsm.mdx originally used a D2 diagram, but this repo has no D2 renderer. It was converted to a {/* TODO */} comment with the original D2 source preserved inline:
-
zerokms-vs-hsm.mdx— "Hybrid solutions" client/SDK/HSM diagram
Needs a decision on how diagrams are rendered in this repo (image export, mermaid support, or a diagram component).
Note: the original ZeroKMS vs Cloud KMS page also had a placeholder diagram, but that page was merged into
aws-kms.mdxand the placeholder diagram (which was sample content) was dropped.
Once both are addressed, cipherstash-js-suite PR #364 can be closed.
Activity
coderdan commentedon Jun 23, 2026
Two performance-claim follow-ups surfaced while rewriting the comparison pages for an agent-research audience (#31):
{/* TODO */}inaws-kms.mdx). The FHE page cites an open-source benchmark harness; this figure should too, with conditions. The sub-5ms latency claim is now linked to the Artillery benchmark.cipherstash/kms/index.mdxandreference/what-is-cipherstash.mdxstill say 100×. A reader cross-referencing pages will hit the contradiction — pick one repo-wide once the benchmark is confirmed.coderdan commentedon Jun 28, 2026
Benchmark results — conditioning the "14×" claim
The ZeroKMS-vs-AWS-KMS benchmark now exists: cipherstash/benches#18 (report with charts). It gives us real data to source/reframe the unsourced, internally-contradictory "14× / 100× faster than AWS KMS" claims.
What the data says
The advantage is not a blanket multiple — it's a function of how many values you encrypt/decrypt per request, because the architectures differ at the root: ZeroKMS does a whole batch in one round-trip (bulk API, up to 10k keys); AWS KMS has no bulk API, so it's one call per value.
They're from a developer laptop on home Wi-Fi, which overstates the gap (network penalty hits AWS's per-value calls harder). The conservative, citable numbers come from the pending in-region EC2 run (runbook:
kms-app/EC2.md).Recommendation for the docs
kms/index.mdxandwhat-is-cipherstash.mdxsay 100×; the comparison pages say 14×. Neither is supported as a blanket figure. Pick one conditioned framing repo-wide.coderdan commentedon Jun 28, 2026
In-region benchmark is done — citable numbers
The conservative, in-region run (cipherstash/benches#18, report) is complete: two
c7i.2xlargeinap-southeast-2, with the load generator on a separate instance from the app (the earlier laptop numbers overstated the gap; a single-box attempt was CPU-bound and discarded).Citable figures:
These supersede the laptop directional numbers in my earlier comment. The "≈14×" in the docs is in the right ballpark for mid-batch latency but undersells the throughput story — for bulk operations the gap is closer to two orders of magnitude. Recommendation stands: replace the blanket "14×/100×" with a conditioned statement (e.g. "for bulk operations ZeroKMS is ~16× lower latency and ~85× higher throughput than AWS KMS, which has no bulk API and throttles at scale") and cite cipherstash/benches.
Caveat for whoever writes the copy: ZeroKMS's ~21k throughput ceiling here was bounded by the app instance's CPU, not ZeroKMS — so it's a floor.