From 47df00160d76d47c5d618df398278d04b31155d1 Mon Sep 17 00:00:00 2001 From: Zhongxuan Wang Date: Fri, 10 Jul 2026 17:49:00 -0700 Subject: [PATCH 1/7] docs: document the opt-in response cache Adds a Response Cache page to the adaptive plugin docs: when to use it, a plugins.toml quickstart, per-language plugin configuration, the complete-answers-only storage rules with streaming aggregation and native replay, exact-request key normalization, the response_cache mark and doctor surfaces, a field reference, and the unredacted-at-rest caveat for shared Redis deployments. Links the page from the adaptive About and Configuration pages. Signed-off-by: Zhongxuan Wang --- docs/configure-plugins/adaptive/about.mdx | 3 + .../adaptive/configuration.mdx | 6 +- .../adaptive/response-cache.mdx | 321 ++++++++++++++++++ 3 files changed, 328 insertions(+), 2 deletions(-) create mode 100644 docs/configure-plugins/adaptive/response-cache.mdx diff --git a/docs/configure-plugins/adaptive/about.mdx b/docs/configure-plugins/adaptive/about.mdx index e6570839d..3bdf9b30c 100644 --- a/docs/configure-plugins/adaptive/about.mdx +++ b/docs/configure-plugins/adaptive/about.mdx @@ -22,6 +22,7 @@ The plugin can coordinate: - Adaptive hints injected into outgoing model requests. - Tool-parallelism observation or scheduling behavior. - Adaptive Cache Governor (ACG) prompt-cache planning. +- Opt-in response caching for repeated LLM calls. - Component-local validation policy. ## Use Adaptive When @@ -51,6 +52,8 @@ If instrumentation is not in place yet, start with cache planning accomplishes. - [Adaptive Hints](/configure-plugins/adaptive/adaptive-hints) explains request hint injection and how downstream model paths can consume the hints. +- [Response Cache](/configure-plugins/adaptive/response-cache) explains the opt-in LLM response cache: + turning it on, what gets cached, and how savings are reported. State, telemetry, tool parallelism, and policy are whole-plugin configuration areas. They are documented on [Adaptive Configuration](/configure-plugins/adaptive/configuration) rather diff --git a/docs/configure-plugins/adaptive/configuration.mdx b/docs/configure-plugins/adaptive/configuration.mdx index 570729b34..abd8c843f 100644 --- a/docs/configure-plugins/adaptive/configuration.mdx +++ b/docs/configure-plugins/adaptive/configuration.mdx @@ -32,10 +32,12 @@ The top-level adaptive object contains: | `adaptive_hints` | Request hint-injection behavior. | | `tool_parallelism` | Tool scheduling observation or scheduling behavior. | | `acg` | Adaptive Cache Governor prompt-cache planning. | +| `response_cache` | Opt-in LLM response cache for repeated managed calls. | | `policy` | Adaptive-local handling for unknown fields and unsupported values. | -The requested area pages cover [Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg) and -[Adaptive Hints](/configure-plugins/adaptive/adaptive-hints). State, telemetry, tool parallelism, and +The requested area pages cover [Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg), +[Adaptive Hints](/configure-plugins/adaptive/adaptive-hints), and +[Response Cache](/configure-plugins/adaptive/response-cache). State, telemetry, tool parallelism, and policy remain whole-plugin settings: - Use `state.backend.kind = "in_memory"` for local experiments. diff --git a/docs/configure-plugins/adaptive/response-cache.mdx b/docs/configure-plugins/adaptive/response-cache.mdx new file mode 100644 index 000000000..fbce1f852 --- /dev/null +++ b/docs/configure-plugins/adaptive/response-cache.mdx @@ -0,0 +1,321 @@ +--- +title: "Response Cache" +description: "" +position: 5 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +Use the response cache when the same LLM request is made more than once and the +repeat should be served from a store instead of paying for the call again. A +repeat of an identical request returns the earlier answer instantly, at no +cost, and unchanged — usage fields intact, so a cached reuse is +shape-identical to a live call. + +The cache is an optional `response_cache` section of the +[Adaptive plugin](/configure-plugins/adaptive/configuration), not a standalone plugin +kind. It is off until the section is present, applies to +[managed LLM calls](/instrument-applications/instrument-llm-call) without any +call-site changes, and fails open: any cache error falls back to a normal live +call. + +## When To Use It + +- Development and test loops that replay the same prompts — repeats are + instant, free, and reproducible. +- CI suites that exercise real models — only the first run of each distinct + request pays. +- Demos and workshops — stable answers and no cost spikes. +- A shared team cache — point the `redis` backend at one store and identical + requests hit across processes and machines. +- Gateway deployments — enable caching for an agent without touching its code. + +Reuse requires the request to match exactly after normalization, so prompts +that embed volatile content (timestamps, random IDs) will not repeat. A stored +answer is also frozen for its TTL — size `ttl_seconds` to how fresh answers +must be, and use `bypass_rate` to re-run a sample of cacheable calls live. + +## `plugins.toml` Example + +```toml +version = 1 + +[[components]] +kind = "adaptive" +enabled = true + +[components.config] +version = 1 + +[components.config.response_cache] +ttl_seconds = 3600 # how long a stored answer stays reusable +namespace = "dev-harness" # separates environments, tenants, and experiments +bypass_rate = 0.0 # 0.1 would re-run 10% of cacheable calls live + +[components.config.response_cache.backend] +kind = "in_memory" # or "redis" for a shared cache +``` + +With this configuration the first occurrence of a request runs the provider +and stores the answer; an identical repeat within the TTL is served from the +store and skips the provider entirely. The request's provider surface is +auto-detected from its shape, so there is nothing else to configure. For file +discovery and precedence rules, see +[Plugin Configuration Files](/configure-plugins/plugin-configuration-files). + +## Plugin Configuration + +Use plugin configuration when the application should let NeMo Relay own the +cache lifecycle. + + + +```python +import nemo_relay + +adaptive_config = nemo_relay.adaptive.AdaptiveConfig( + response_cache=nemo_relay.adaptive.ResponseCacheConfig( + ttl_seconds=3600, + namespace="dev-harness", + ), +) + +plugin_config = nemo_relay.plugin.PluginConfig( + components=[nemo_relay.adaptive.ComponentSpec(adaptive_config)] +) + +report = nemo_relay.plugin.validate(plugin_config) +if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): + raise RuntimeError(report["diagnostics"]) + +await nemo_relay.plugin.initialize(plugin_config) + +# Managed calls need no changes. The first call runs the provider; +# an identical repeat is served from the cache. +request = nemo_relay.LLMRequest( + {}, {"model": "gpt-4o", "messages": [{"role": "user", "content": "What is NeMo Relay?"}]} +) +await nemo_relay.llm.execute("openai", request, call_model) # miss: runs call_model +await nemo_relay.llm.execute("openai", request, call_model) # hit: call_model is skipped +``` + + + +```js +const adaptive = require("nemo-relay-node/adaptive"); +const plugin = require("nemo-relay-node/plugin"); + +const adaptiveConfig = adaptive.defaultConfig(); +adaptiveConfig.response_cache = adaptive.responseCacheConfig({ + ttl_seconds: 3600, + namespace: "dev-harness", +}); + +const pluginConfig = plugin.defaultConfig(); +pluginConfig.components = [adaptive.ComponentSpec(adaptiveConfig)]; + +const report = plugin.validate(pluginConfig); +if (report.diagnostics.some((diagnostic) => diagnostic.level === "error")) { + throw new Error(JSON.stringify(report.diagnostics)); +} + +await plugin.initialize(pluginConfig); +// Managed LLM calls now serve identical repeats from the cache. +``` + + + +```rust +use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig}; +use nemo_relay_adaptive::plugin_component::ComponentSpec; +use nemo_relay_adaptive::{AdaptiveConfig, ResponseCacheConfig}; + +let mut adaptive = AdaptiveConfig::default(); +adaptive.response_cache = Some(ResponseCacheConfig { + ttl_seconds: 3600, + namespace: "dev-harness".into(), + ..ResponseCacheConfig::default() +}); + +let mut plugin_config = PluginConfig::default(); +plugin_config.components.push(ComponentSpec::new(adaptive).into()); + +let report = validate_plugin_config(&plugin_config); +assert!(!report.has_errors()); + +let active = initialize_plugins(plugin_config).await?; +// Managed LLM calls now serve identical repeats from the cache. +``` + + + + + +NeMo Relay plugin configuration keys use `snake_case` regardless of language or +file format. The `backend` helpers differ slightly by binding: Python uses +`nemo_relay.adaptive.BackendSpec.in_memory()` / `BackendSpec.redis(url)`, and +Node.js uses `adaptive.inMemoryBackend()` / `adaptive.redisBackend(url)`. Both +redis helpers set `key_prefix` to `"nemo_relay:"` explicitly, overriding the +default in the fields table; pass a `key_prefix` argument to choose your own. + + +## Manual API + +Use the manual runtime API when an integration needs to own the adaptive +lifecycle directly instead of activating the top-level plugin component. + + + +```python +import nemo_relay + +adaptive_config = nemo_relay.adaptive.AdaptiveConfig( + response_cache=nemo_relay.adaptive.ResponseCacheConfig(namespace="dev-harness"), +) + +runtime = nemo_relay.adaptive.AdaptiveRuntime(adaptive_config.to_dict()) +await runtime.register() +try: + # Run instrumented application work here. + runtime.wait_for_idle() +finally: + await runtime.shutdown() +``` + + + +The Node.js binding exposes the response cache through the adaptive plugin +component helpers. Use the Plugin Configuration example above when activating +the cache from Node.js. + + + +```rust +use nemo_relay_adaptive::{AdaptiveConfig, AdaptiveRuntime, ResponseCacheConfig}; + +let mut adaptive = AdaptiveConfig::default(); +adaptive.response_cache = Some(ResponseCacheConfig { + namespace: "dev-harness".into(), + ..ResponseCacheConfig::default() +}); + +let mut runtime = AdaptiveRuntime::new(adaptive).await?; +runtime.register().await?; + +// Run instrumented application work here. + +runtime.wait_for_idle(); +runtime.shutdown().await?; +``` + + + + +## What Gets Cached + +Only complete, replayable answers are stored: + +- A response with a non-null `error`, or a non-final `status` such as + `failed`, `cancelled`, `incomplete`, or `in_progress`, is never stored. A + failed upstream reply passes through to the caller untouched. +- Stateful requests bypass the cache entirely: a request that opts into + server-side persistence (a non-`false` `store` flag, or an OpenAI Responses + call without an explicit `store = false`), continues a stored interaction + (`previous_response_id`), or references server-side state (`conversation`, + `container`). Their answers depend on state the cache key cannot see. +- Streaming calls are cached too. On a miss the live chunks are forwarded to + the consumer while being assembled into one aggregate response — the same + shape a buffered call stores, so buffered and streaming calls share one + keyspace. On a hit the stored answer is replayed as provider-native chunks + (OpenAI Chat deltas, OpenAI Responses lifecycle events, or Anthropic + Messages events), so strict streaming clients parse it like a live stream. + Truncated or incomplete streams, and streams whose content cannot be + replayed faithfully (for example thinking blocks), are never stored; a + request whose surface cannot be inferred runs live, uncached. + +A hit returns the stored response unchanged. What the hit avoided — tokens and +estimated cost — is reported on the `response_cache` mark, never by editing +the response body. + +## Cache Keys + +Two requests hit the same entry when they are the same request after +normalization, under the default `key_strategy = "exact_request"`: + +- The request is decoded to its normalized form and fingerprinted with + SHA-256, so provider-shaped differences that mean the same thing collapse to + one key. When a request cannot be decoded faithfully, the raw body is + fingerprinted instead — that fallback can only cost a miss, never a wrong + hit. +- Field order and whitespace never matter (RFC 8785 canonicalization). +- Volatile request fields are always dropped from the key: `stream`, `user`, + `metadata`, `service_tier`, and `store`. Add project-specific noise fields + with `skip_keys`. +- Tool-call IDs are normalized, so randomly generated per-call IDs do not + fragment the keyspace. +- Request headers stay out of the key unless explicitly named in + `header_allowlist`; auth headers are rejected outright. +- The `namespace` and the provider name are folded into every key, so + environments and providers never share entries. + +## Observability + +Every cache decision emits a `response_cache` mark with +`data.status` set to one of: + +| Status | Meaning | +|---|---| +| `hit` | Served the stored answer; the provider was skipped. | +| `miss` | No stored answer; the call ran live and, on a clean result, the answer was stored. | +| `bypass` | The request is not cacheable, or the `bypass_rate` sampler chose to run live. | + +Mark metadata rides under `nemo_relay.response_cache.*`: `backend`, +`surface`, `key_hash` (the `sha256:…` fingerprint), `ttl_ms`, `age_ms` and +`saved_tokens` / `saved_cost_usd` on hits, and a `reason` on bypasses and +store-error misses (for example `sampled`, `stateful_store`, `store_error`, +`stream_no_codec`). Marks carry only fingerprints — never prompts, answers, or +credentials. + +`nemo-relay doctor` reports the cache state: `not configured` when the section +is absent, `configured but disabled (adaptive plugin disabled)` when the +adaptive component is off, `on; backend '' reachable` when healthy, and +a failure when the config is invalid or the backend is unreachable. + +## Fields + +| Field | Default | Notes | +|---|---|---| +| `ttl_seconds` | `3600` | How long a stored answer stays reusable, in seconds. | +| `namespace` | `""` | Folded into every key; separates environments, tenants, and experiments. | +| `priority` | `50` | LLM execution intercept priority. Lower values run earlier. | +| `bypass_rate` | `0.0` | Probability in `[0.0, 1.0]` of running a cacheable call live. A sampled call refreshes the stored answer, so this doubles as drift detection. | +| `cache_nondeterministic` | `true` | A repeat of a nondeterministic request serves one stored sample. Set `false` to cache only requests pinned deterministic (`temperature = 0`). | +| `key_strategy` | `"exact_request"` | The only supported strategy: reuse requires the same normalized request. | +| `header_allowlist` | `[]` | Headers folded into the key (case-insensitive). Auth headers are rejected. | +| `skip_keys` | `[]` | Extra top-level body keys dropped from the key, beyond the built-in volatile set. | +| `backend.kind` | `"in_memory"` | `"in_memory"`, or `"redis"` (requires building with the `redis-backend` feature). | +| `backend.config.max_bytes` | 256 MiB | In-memory size budget; the oldest entries are evicted first. | +| `backend.config.url` | — | Redis connection URL. Required for the `redis` backend. | +| `backend.config.key_prefix` | `"nemo-relay:llm-cache:"` | Prefix for keys in Redis. | + + +Cached entries are stored unredacted. PII sanitize guardrails rewrite emitted +telemetry, never payloads, so the store holds full request and response +bodies. A shared Redis backend must be trusted, access-controlled, and +namespaced per environment and tenant. + + +## Common Validation Failures + +- `ttl_seconds` is `0`, or `bypass_rate` is outside `[0.0, 1.0]`. +- `key_strategy` is not `"exact_request"`. +- `skip_keys` names an answer-determining field such as `messages`, `model`, + `tools`, or `tool_choice`. +- `header_allowlist` names an auth header such as `authorization` or + `x-api-key`. +- in_memory `max_bytes` set to zero or a non-integer. +- `backend.kind = "redis"` without `backend.config.url`, or in a build without + the `redis-backend` feature. +- A `redis` backend with an empty `namespace` warns: every caller of the + shared store would exchange cached answers. From 00ded9bb195b3b9104330e55fa869c3891f6efad Mon Sep 17 00:00:00 2001 From: Zhongxuan Wang Date: Wed, 22 Jul 2026 17:41:22 -0700 Subject: [PATCH 2/7] docs(node): use response-cache camelCase API Signed-off-by: Zhongxuan Wang --- docs/configure-plugins/adaptive/response-cache.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/configure-plugins/adaptive/response-cache.mdx b/docs/configure-plugins/adaptive/response-cache.mdx index fbce1f852..4130c1007 100644 --- a/docs/configure-plugins/adaptive/response-cache.mdx +++ b/docs/configure-plugins/adaptive/response-cache.mdx @@ -107,8 +107,8 @@ const adaptive = require("nemo-relay-node/adaptive"); const plugin = require("nemo-relay-node/plugin"); const adaptiveConfig = adaptive.defaultConfig(); -adaptiveConfig.response_cache = adaptive.responseCacheConfig({ - ttl_seconds: 3600, +adaptiveConfig.responseCache = adaptive.responseCacheConfig({ + ttlSeconds: 3600, namespace: "dev-harness", }); From d0fae8eba0902b8beba51d277a5422506d21cf3e Mon Sep 17 00:00:00 2001 From: Zhongxuan Wang Date: Wed, 22 Jul 2026 18:22:54 -0700 Subject: [PATCH 3/7] docs(response-cache): qualify reuse claims Signed-off-by: Zhongxuan Wang --- .../adaptive/response-cache.mdx | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/configure-plugins/adaptive/response-cache.mdx b/docs/configure-plugins/adaptive/response-cache.mdx index 4130c1007..da46146d2 100644 --- a/docs/configure-plugins/adaptive/response-cache.mdx +++ b/docs/configure-plugins/adaptive/response-cache.mdx @@ -8,10 +8,10 @@ SPDX-License-Identifier: Apache-2.0 */} Use the response cache when the same LLM request is made more than once and the -repeat should be served from a store instead of paying for the call again. A -repeat of an identical request returns the earlier answer instantly, at no -cost, and unchanged — usage fields intact, so a cached reuse is -shape-identical to a live call. +repeat should be served from a store instead of calling the provider again. An +eligible request that matches an unexpired cache entry can be served from the +store without calling the provider. Cached responses preserve the stored +response shape and usage fields. The cache is an optional `response_cache` section of the [Adaptive plugin](/configure-plugins/adaptive/configuration), not a standalone plugin @@ -22,13 +22,14 @@ call. ## When To Use It -- Development and test loops that replay the same prompts — repeats are - instant, free, and reproducible. -- CI suites that exercise real models — only the first run of each distinct - request pays. -- Demos and workshops — stable answers and no cost spikes. +- Development and test loops that replay the same prompts — eligible repeats + can reuse stored responses. +- CI suites that exercise real models — cached repeats can avoid additional + provider calls. +- Demos and workshops — reuse stable stored answers while reducing provider + calls. - A shared team cache — point the `redis` backend at one store and identical - requests hit across processes and machines. + requests can reuse entries across processes and machines. - Gateway deployments — enable caching for an agent without touching its code. Reuse requires the request to match exactly after normalization, so prompts From 0a3872073629470a3d6b457d1c4fba71de1790c3 Mon Sep 17 00:00:00 2001 From: Zhongxuan Wang Date: Mon, 27 Jul 2026 12:49:31 -0700 Subject: [PATCH 4/7] docs(response-cache): correct configuration guidance Signed-off-by: Zhongxuan Wang --- .../adaptive/configuration.mdx | 4 +- .../adaptive/response-cache.mdx | 187 ++++++++++++------ 2 files changed, 127 insertions(+), 64 deletions(-) diff --git a/docs/configure-plugins/adaptive/configuration.mdx b/docs/configure-plugins/adaptive/configuration.mdx index abd8c843f..21719d06f 100644 --- a/docs/configure-plugins/adaptive/configuration.mdx +++ b/docs/configure-plugins/adaptive/configuration.mdx @@ -1,7 +1,7 @@ --- title: "Adaptive Configuration" sidebar-title: "Configuration" -description: "Configure the built-in Adaptive component, including state, telemetry, hints, and ACG." +description: "Configure the Adaptive component, including state, telemetry, hints, ACG, and response caching." position: 2 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -35,7 +35,7 @@ The top-level adaptive object contains: | `response_cache` | Opt-in LLM response cache for repeated managed calls. | | `policy` | Adaptive-local handling for unknown fields and unsupported values. | -The requested area pages cover [Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg), +Dedicated pages cover [Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg), [Adaptive Hints](/configure-plugins/adaptive/adaptive-hints), and [Response Cache](/configure-plugins/adaptive/response-cache). State, telemetry, tool parallelism, and policy remain whole-plugin settings: diff --git a/docs/configure-plugins/adaptive/response-cache.mdx b/docs/configure-plugins/adaptive/response-cache.mdx index da46146d2..333b842b3 100644 --- a/docs/configure-plugins/adaptive/response-cache.mdx +++ b/docs/configure-plugins/adaptive/response-cache.mdx @@ -1,6 +1,6 @@ --- title: "Response Cache" -description: "" +description: "Configure exact-match response caching for managed LLM calls." position: 5 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -10,15 +10,18 @@ SPDX-License-Identifier: Apache-2.0 */} Use the response cache when the same LLM request is made more than once and the repeat should be served from a store instead of calling the provider again. An eligible request that matches an unexpired cache entry can be served from the -store without calling the provider. Cached responses preserve the stored -response shape and usage fields. +store without calling the provider. Buffered hits preserve the stored response +shape and usage fields; streaming hits replay an equivalent provider-native +stream. The cache is an optional `response_cache` section of the [Adaptive plugin](/configure-plugins/adaptive/configuration), not a standalone plugin kind. It is off until the section is present, applies to -[managed LLM calls](/instrument-applications/instrument-llm-call) without any -call-site changes, and fails open: any cache error falls back to a normal live -call. +[managed LLM calls](/instrument-applications/instrument-llm-call) without +changing the execution API. By default, only requests with an explicit numeric +`temperature = 0` are eligible; set `cache_nondeterministic = true` to opt +sampled requests into caching. Runtime backend errors fail open to a normal +live call, while invalid configuration is rejected during validation. ## When To Use It @@ -34,8 +37,9 @@ call. Reuse requires the request to match exactly after normalization, so prompts that embed volatile content (timestamps, random IDs) will not repeat. A stored -answer is also frozen for its TTL — size `ttl_seconds` to how fresh answers -must be, and use `bypass_rate` to re-run a sample of cacheable calls live. +answer remains reusable until its TTL expires unless a sampled bypass +successfully refreshes it. Size `ttl_seconds` to how fresh answers must be, and +use `bypass_rate` to re-run a sample of cacheable calls live. ## `plugins.toml` Example @@ -53,16 +57,17 @@ version = 1 ttl_seconds = 3600 # how long a stored answer stays reusable namespace = "dev-harness" # separates environments, tenants, and experiments bypass_rate = 0.0 # 0.1 would re-run 10% of cacheable calls live +cache_nondeterministic = false # set true to cache nondeterministic requests [components.config.response_cache.backend] kind = "in_memory" # or "redis" for a shared cache ``` -With this configuration the first occurrence of a request runs the provider -and stores the answer; an identical repeat within the TTL is served from the -store and skips the provider entirely. The request's provider surface is -auto-detected from its shape, so there is nothing else to configure. For file -discovery and precedence rules, see +With this configuration the first eligible occurrence of a request runs the +provider and attempts to store a complete answer. A later eligible, identical +request within the TTL is served from the store when that write succeeds. The +request's provider surface is auto-detected from its shape, so there is nothing +else to configure. For file discovery and precedence rules, see [Plugin Configuration Files](/configure-plugins/plugin-configuration-files). ## Plugin Configuration @@ -92,10 +97,24 @@ if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): await nemo_relay.plugin.initialize(plugin_config) +def call_model(_request): + return { + "model": "gpt-4o", + "choices": [{ + "message": {"role": "assistant", "content": "NeMo Relay instruments agent calls."}, + "finish_reason": "stop", + }], + } + # Managed calls need no changes. The first call runs the provider; # an identical repeat is served from the cache. request = nemo_relay.LLMRequest( - {}, {"model": "gpt-4o", "messages": [{"role": "user", "content": "What is NeMo Relay?"}]} + {}, + { + "model": "gpt-4o", + "messages": [{"role": "user", "content": "What is NeMo Relay?"}], + "temperature": 0, + }, ) await nemo_relay.llm.execute("openai", request, call_model) # miss: runs call_model await nemo_relay.llm.execute("openai", request, call_model) # hit: call_model is skipped @@ -121,44 +140,62 @@ if (report.diagnostics.some((diagnostic) => diagnostic.level === "error")) { throw new Error(JSON.stringify(report.diagnostics)); } -await plugin.initialize(pluginConfig); -// Managed LLM calls now serve identical repeats from the cache. +void (async () => { + await plugin.initialize(pluginConfig); + try { + // Eligible deterministic repeats are served from the cache. + } finally { + plugin.clear(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; +}); ``` ```rust -use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig}; -use nemo_relay_adaptive::plugin_component::ComponentSpec; +use nemo_relay::plugin::{ + clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig, +}; +use nemo_relay_adaptive::plugin_component::{register_adaptive_component, ComponentSpec}; use nemo_relay_adaptive::{AdaptiveConfig, ResponseCacheConfig}; -let mut adaptive = AdaptiveConfig::default(); -adaptive.response_cache = Some(ResponseCacheConfig { - ttl_seconds: 3600, - namespace: "dev-harness".into(), - ..ResponseCacheConfig::default() -}); +#[tokio::main] +async fn main() -> Result<(), Box> { + let mut adaptive = AdaptiveConfig::default(); + adaptive.response_cache = Some(ResponseCacheConfig { + ttl_seconds: 3600, + namespace: "dev-harness".into(), + ..ResponseCacheConfig::default() + }); -let mut plugin_config = PluginConfig::default(); -plugin_config.components.push(ComponentSpec::new(adaptive).into()); + let mut plugin_config = PluginConfig::default(); + plugin_config.components.push(ComponentSpec::new(adaptive).into()); -let report = validate_plugin_config(&plugin_config); -assert!(!report.has_errors()); + register_adaptive_component()?; + let report = validate_plugin_config(&plugin_config); + assert!(!report.has_errors()); -let active = initialize_plugins(plugin_config).await?; -// Managed LLM calls now serve identical repeats from the cache. + let _active = initialize_plugins(plugin_config).await?; + // Eligible deterministic repeats are now served from the cache. + + clear_plugin_configuration()?; + Ok(()) +} ``` -NeMo Relay plugin configuration keys use `snake_case` regardless of language or -file format. The `backend` helpers differ slightly by binding: Python uses -`nemo_relay.adaptive.BackendSpec.in_memory()` / `BackendSpec.redis(url)`, and -Node.js uses `adaptive.inMemoryBackend()` / `adaptive.redisBackend(url)`. Both -redis helpers set `key_prefix` to `"nemo_relay:"` explicitly, overriding the -default in the fields table; pass a `key_prefix` argument to choose your own. +Canonical plugin documents and `plugins.toml` use `snake_case`. Python helpers +follow that convention, while Node.js convenience helpers use `camelCase` and +serialize it to the canonical shape. Python uses +`BackendSpec.redis(url, key_prefix=...)`; Node.js uses +`adaptive.redisBackend(url, keyPrefix)`. Both Redis helpers default the prefix +to `"nemo_relay:"`, overriding the value in the fields table. ## Manual API @@ -186,28 +223,51 @@ finally: -The Node.js binding exposes the response cache through the adaptive plugin -component helpers. Use the Plugin Configuration example above when activating -the cache from Node.js. +```js +const adaptive = require("nemo-relay-node/adaptive"); + +const adaptiveConfig = adaptive.defaultConfig(); +adaptiveConfig.responseCache = adaptive.responseCacheConfig({ + namespace: "dev-harness", +}); + +const runtime = new adaptive.AdaptiveRuntime(adaptiveConfig); +void (async () => { + await runtime.register(); + try { + // Run instrumented application work here. + runtime.waitForIdle(); + } finally { + await runtime.shutdown(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; +}); +``` ```rust use nemo_relay_adaptive::{AdaptiveConfig, AdaptiveRuntime, ResponseCacheConfig}; -let mut adaptive = AdaptiveConfig::default(); -adaptive.response_cache = Some(ResponseCacheConfig { - namespace: "dev-harness".into(), - ..ResponseCacheConfig::default() -}); +#[tokio::main] +async fn main() -> Result<(), Box> { + let mut adaptive = AdaptiveConfig::default(); + adaptive.response_cache = Some(ResponseCacheConfig { + namespace: "dev-harness".into(), + ..ResponseCacheConfig::default() + }); -let mut runtime = AdaptiveRuntime::new(adaptive).await?; -runtime.register().await?; + let mut runtime = AdaptiveRuntime::new(adaptive).await?; + runtime.register().await?; -// Run instrumented application work here. + // Run instrumented application work here. -runtime.wait_for_idle(); -runtime.shutdown().await?; + runtime.wait_for_idle(); + runtime.shutdown().await?; + Ok(()) +} ``` @@ -235,9 +295,10 @@ Only complete, replayable answers are stored: replayed faithfully (for example thinking blocks), are never stored; a request whose surface cannot be inferred runs live, uncached. -A hit returns the stored response unchanged. What the hit avoided — tokens and -estimated cost — is reported on the `response_cache` mark, never by editing -the response body. +A buffered hit returns the stored response unchanged. A streaming hit replays +an equivalent provider-native stream. What the hit avoided — tokens and +estimated cost — is reported on the `response_cache` mark, never by editing a +buffered response body. ## Cache Keys @@ -257,8 +318,10 @@ normalization, under the default `key_strategy = "exact_request"`: fragment the keyspace. - Request headers stay out of the key unless explicitly named in `header_allowlist`; auth headers are rejected outright. -- The `namespace` and the provider name are folded into every key, so - environments and providers never share entries. +- The provider name is folded into every key, so different providers do not + share entries. The `namespace` is also folded into the key; assign distinct + values to isolate environments or tenants. Identical or empty namespaces can + share entries. ## Observability @@ -268,7 +331,7 @@ Every cache decision emits a `response_cache` mark with | Status | Meaning | |---|---| | `hit` | Served the stored answer; the provider was skipped. | -| `miss` | No stored answer; the call ran live and, on a clean result, the answer was stored. | +| `miss` | No stored answer; the call ran live, and a cacheable result is stored when the write succeeds. | | `bypass` | The request is not cacheable, or the `bypass_rate` sampler chose to run live. | Mark metadata rides under `nemo_relay.response_cache.*`: `backend`, @@ -288,10 +351,10 @@ a failure when the config is invalid or the backend is unreachable. | Field | Default | Notes | |---|---|---| | `ttl_seconds` | `3600` | How long a stored answer stays reusable, in seconds. | -| `namespace` | `""` | Folded into every key; separates environments, tenants, and experiments. | +| `namespace` | `""` | Folded into every key. Set distinct values to separate environments, tenants, and experiments. | | `priority` | `50` | LLM execution intercept priority. Lower values run earlier. | -| `bypass_rate` | `0.0` | Probability in `[0.0, 1.0]` of running a cacheable call live. A sampled call refreshes the stored answer, so this doubles as drift detection. | -| `cache_nondeterministic` | `true` | A repeat of a nondeterministic request serves one stored sample. Set `false` to cache only requests pinned deterministic (`temperature = 0`). | +| `bypass_rate` | `0.0` | Probability in `[0.0, 1.0]` of running a cacheable call live. A sampled call attempts to refresh the entry when its result is cacheable and the write succeeds. | +| `cache_nondeterministic` | `false` | Nondeterministic requests bypass the cache. Set `true` to cache and reuse sampled responses. | | `key_strategy` | `"exact_request"` | The only supported strategy: reuse requires the same normalized request. | | `header_allowlist` | `[]` | Headers folded into the key (case-insensitive). Auth headers are rejected. | | `skip_keys` | `[]` | Extra top-level body keys dropped from the key, beyond the built-in volatile set. | @@ -301,10 +364,10 @@ a failure when the config is invalid or the backend is unreachable. | `backend.config.key_prefix` | `"nemo-relay:llm-cache:"` | Prefix for keys in Redis. | -Cached entries are stored unredacted. PII sanitize guardrails rewrite emitted -telemetry, never payloads, so the store holds full request and response -bodies. A shared Redis backend must be trusted, access-controlled, and -namespaced per environment and tenant. +Cached responses are stored unredacted. PII sanitize guardrails rewrite emitted +telemetry, never payloads, so the store holds full response bodies. Requests +contribute only a SHA-256 fingerprint to the cache key. A shared Redis backend +must be trusted, access-controlled, and namespaced per environment and tenant. ## Common Validation Failures From 192b6247b3be7b9172820c6f2215ea5612f44e9d Mon Sep 17 00:00:00 2001 From: Zhongxuan Wang Date: Thu, 30 Jul 2026 17:15:30 -0700 Subject: [PATCH 5/7] docs(adaptive): align response cache guide Signed-off-by: Zhongxuan Wang --- crates/adaptive/README.md | 14 +- .../adaptive/configuration.mdx | 15 +- .../adaptive/response-cache.mdx | 210 +++++++++++------- .../switchyard/configuration.mdx | 2 +- docs/nemo-relay-cli/basic-usage.mdx | 22 +- docs/reference/migration-guides.mdx | 4 + 6 files changed, 166 insertions(+), 101 deletions(-) diff --git a/crates/adaptive/README.md b/crates/adaptive/README.md index 7dd1a1ced..540f35727 100644 --- a/crates/adaptive/README.md +++ b/crates/adaptive/README.md @@ -18,7 +18,8 @@ SPDX-License-Identifier: Apache-2.0 `nemo-relay-adaptive` is the Rust companion crate for adaptive NeMo Relay runtime behavior. Use it with `nemo-relay` when an agent runtime should learn -from observed executions, inject runtime hints, or persist adaptive state. +from observed executions, inject runtime hints, persist adaptive state, or +cache repeated LLM responses. Adaptive behavior is installed through the same plugin system used by the core runtime, so applications can enable it without changing their orchestration @@ -41,9 +42,11 @@ framework. - **`AdaptiveConfig`**: A canonical config contract for the top-level `adaptive` plugin component. - **Built-in component settings**: Typed config helpers for telemetry, - adaptive hints, tool parallelism, and the Adaptive Cache Governor. -- **State backends**: In-memory state by default and Redis-backed state behind - the `redis-backend` feature. + adaptive hints, tool parallelism, the Adaptive Cache Governor, and response + caching. +- **Storage backends**: In-memory backends for adaptive state and opt-in + response caching, with Redis-backed storage behind the `redis-backend` + feature. - **Learning primitives**: Runtime helpers and learners built on NeMo Relay events. - **Adaptive Cache Governor (ACG) module surface**: The canonical @@ -58,7 +61,8 @@ Install the published crate alongside the core runtime: cargo add nemo-relay nemo-relay-adaptive ``` -Enable Redis-backed state only when the application needs shared persistence: +Enable Redis-backed storage when adaptive state or the response cache needs +shared persistence: ```bash cargo add nemo-relay-adaptive --features redis-backend diff --git a/docs/configure-plugins/adaptive/configuration.mdx b/docs/configure-plugins/adaptive/configuration.mdx index 21719d06f..d85391b2e 100644 --- a/docs/configure-plugins/adaptive/configuration.mdx +++ b/docs/configure-plugins/adaptive/configuration.mdx @@ -11,9 +11,10 @@ SPDX-License-Identifier: Apache-2.0 */} Use this page when you want to configure the built-in Adaptive plugin component as a whole. The component kind is `adaptive`. -Adaptive plugin configuration uses the generic NeMo Relay plugin document shape. -Field names stay `snake_case` in every binding and in `plugins.toml`, even when -language helper functions use language-native naming conventions. +Adaptive plugin configuration uses the generic NeMo Relay plugin document +shape. Canonical plugin documents and `plugins.toml` use `snake_case`. +Binding helpers can use language-native names and serialize them to canonical +keys. For plugin file discovery, precedence, merge behavior, editor controls, and gateway conflict rules, refer to @@ -32,7 +33,7 @@ The top-level adaptive object contains: | `adaptive_hints` | Request hint-injection behavior. | | `tool_parallelism` | Tool scheduling observation or scheduling behavior. | | `acg` | Adaptive Cache Governor prompt-cache planning. | -| `response_cache` | Opt-in LLM response cache for repeated managed calls. | +| `response_cache` | Opt-in LLM response cache for repeated managed calls. Requires a non-empty trust-domain `namespace`. | | `policy` | Adaptive-local handling for unknown fields and unsupported values. | Dedicated pages cover [Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg), @@ -273,9 +274,9 @@ finally: -The Node.js binding exposes the built-in adaptive runtime through the adaptive -plugin component helpers. Use the Plugin Configuration example above when -activating adaptive behavior from Node.js. +The Node.js binding exposes `adaptive.AdaptiveRuntime` for manual lifecycle +ownership. Construct it from the same `adaptiveConfig`, then call `register()`, +`waitForIdle()`, and `shutdown()`. diff --git a/docs/configure-plugins/adaptive/response-cache.mdx b/docs/configure-plugins/adaptive/response-cache.mdx index 333b842b3..b9299bc0f 100644 --- a/docs/configure-plugins/adaptive/response-cache.mdx +++ b/docs/configure-plugins/adaptive/response-cache.mdx @@ -23,7 +23,11 @@ changing the execution API. By default, only requests with an explicit numeric sampled requests into caching. Runtime backend errors fail open to a normal live call, while invalid configuration is rejected during validation. -## When To Use It +`namespace` is required and defines one trusted cache-sharing domain. Do not +use one namespace across mutually untrusted tenants or upstreams. +`header_allowlist` does not replace this trust boundary. + +## When to Use It - Development and test loops that replay the same prompts — eligible repeats can reuse stored responses. @@ -31,15 +35,15 @@ live call, while invalid configuration is rejected during validation. provider calls. - Demos and workshops — reuse stable stored answers while reducing provider calls. -- A shared team cache — point the `redis` backend at one store and identical - requests can reuse entries across processes and machines. +- A shared team cache — callers in one trusted sharing domain can use the same + Redis store, namespace, and key prefix across processes and machines. - Gateway deployments — enable caching for an agent without touching its code. Reuse requires the request to match exactly after normalization, so prompts -that embed volatile content (timestamps, random IDs) will not repeat. A stored -answer remains reusable until its TTL expires unless a sampled bypass -successfully refreshes it. Size `ttl_seconds` to how fresh answers must be, and -use `bypass_rate` to re-run a sample of cacheable calls live. +that embed volatile content (timestamps, random IDs) will not repeat. The TTL +is the maximum entry age, not guaranteed residency; a backend can evict or +delete an entry sooner. Use `bypass_rate` to re-run a sample of cacheable calls +live and refresh an entry when the new response is stored. ## `plugins.toml` Example @@ -54,8 +58,8 @@ enabled = true version = 1 [components.config.response_cache] -ttl_seconds = 3600 # how long a stored answer stays reusable -namespace = "dev-harness" # separates environments, tenants, and experiments +ttl_seconds = 3600 # maximum reuse age +namespace = "dev-harness" # identifies one trusted cache-sharing domain bypass_rate = 0.0 # 0.1 would re-run 10% of cacheable calls live cache_nondeterministic = false # set true to cache nondeterministic requests @@ -65,9 +69,10 @@ kind = "in_memory" # or "redis" for a shared cache With this configuration the first eligible occurrence of a request runs the provider and attempts to store a complete answer. A later eligible, identical -request within the TTL is served from the store when that write succeeds. The -request's provider surface is auto-detected from its shape, so there is nothing -else to configure. For file discovery and precedence rules, see +request within the TTL can be served from the store when that write succeeds +and the entry remains present. The request's provider surface is auto-detected +from its shape, so there is nothing else to configure. For file discovery and +precedence rules, refer to [Plugin Configuration Files](/configure-plugins/plugin-configuration-files). ## Plugin Configuration @@ -78,6 +83,8 @@ cache lifecycle. ```python +import asyncio + import nemo_relay adaptive_config = nemo_relay.adaptive.AdaptiveConfig( @@ -95,8 +102,6 @@ report = nemo_relay.plugin.validate(plugin_config) if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): raise RuntimeError(report["diagnostics"]) -await nemo_relay.plugin.initialize(plugin_config) - def call_model(_request): return { "model": "gpt-4o", @@ -106,18 +111,25 @@ def call_model(_request): }], } -# Managed calls need no changes. The first call runs the provider; -# an identical repeat is served from the cache. -request = nemo_relay.LLMRequest( - {}, - { - "model": "gpt-4o", - "messages": [{"role": "user", "content": "What is NeMo Relay?"}], - "temperature": 0, - }, -) -await nemo_relay.llm.execute("openai", request, call_model) # miss: runs call_model -await nemo_relay.llm.execute("openai", request, call_model) # hit: call_model is skipped +async def main(): + await nemo_relay.plugin.initialize(plugin_config) + try: + # Managed calls need no changes. The first call runs the provider; + # an identical repeat can be served from the cache. + request = nemo_relay.LLMRequest( + {}, + { + "model": "gpt-4o", + "messages": [{"role": "user", "content": "What is NeMo Relay?"}], + "temperature": 0, + }, + ) + await nemo_relay.llm.execute("openai", request, call_model) + await nemo_relay.llm.execute("openai", request, call_model) + finally: + nemo_relay.plugin.clear() + +asyncio.run(main()) ``` @@ -143,7 +155,7 @@ if (report.diagnostics.some((diagnostic) => diagnostic.level === "error")) { void (async () => { await plugin.initialize(pluginConfig); try { - // Eligible deterministic repeats are served from the cache. + // Eligible deterministic repeats can be served from the cache. } finally { plugin.clear(); } @@ -179,7 +191,7 @@ async fn main() -> Result<(), Box> { assert!(!report.has_errors()); let _active = initialize_plugins(plugin_config).await?; - // Eligible deterministic repeats are now served from the cache. + // Eligible deterministic repeats can now be served from the cache. clear_plugin_configuration()?; Ok(()) @@ -190,12 +202,14 @@ async fn main() -> Result<(), Box> { -Canonical plugin documents and `plugins.toml` use `snake_case`. Python helpers -follow that convention, while Node.js convenience helpers use `camelCase` and -serialize it to the canonical shape. Python uses +Canonical plugin documents and `plugins.toml` use `snake_case`. Python +response-cache fields follow that convention, while Node.js uses `camelCase` at +its response-cache helper surface. `ComponentSpec` and `AdaptiveRuntime` +serialize those fields to canonical keys. Python uses `BackendSpec.redis(url, key_prefix=...)`; Node.js uses `adaptive.redisBackend(url, keyPrefix)`. Both Redis helpers default the prefix -to `"nemo_relay:"`, overriding the value in the fields table. +to `"nemo_relay:"`, overriding the value in the fields table. Configure the +same `key_prefix` in every process and binding that should share entries. ## Manual API @@ -206,19 +220,24 @@ lifecycle directly instead of activating the top-level plugin component. ```python +import asyncio + import nemo_relay adaptive_config = nemo_relay.adaptive.AdaptiveConfig( response_cache=nemo_relay.adaptive.ResponseCacheConfig(namespace="dev-harness"), ) -runtime = nemo_relay.adaptive.AdaptiveRuntime(adaptive_config.to_dict()) -await runtime.register() -try: - # Run instrumented application work here. - runtime.wait_for_idle() -finally: - await runtime.shutdown() +async def main(): + runtime = nemo_relay.adaptive.AdaptiveRuntime(adaptive_config.to_dict()) + await runtime.register() + try: + # Run instrumented application work here. + runtime.wait_for_idle() + finally: + await runtime.shutdown() + +asyncio.run(main()) ``` @@ -277,28 +296,36 @@ async fn main() -> Result<(), Box> { Only complete, replayable answers are stored: -- A response with a non-null `error`, or a non-final `status` such as - `failed`, `cancelled`, `incomplete`, or `in_progress`, is never stored. A - failed upstream reply passes through to the caller untouched. +- A response with a non-null `error` or a `status` such as `failed`, + `cancelled`, `incomplete`, or `in_progress` is never stored. - Stateful requests bypass the cache entirely: a request that opts into - server-side persistence (a non-`false` `store` flag, or an OpenAI Responses - call without an explicit `store = false`), continues a stored interaction - (`previous_response_id`), or references server-side state (`conversation`, - `container`). Their answers depend on state the cache key cannot see. + server-side persistence (a non-null `store` value other than `false`, or an + OpenAI Responses call without an explicit `store = false`), continues a + stored interaction (`previous_response_id`), or references server-side state + (`conversation`, `container`). Their answers depend on state the cache key + cannot see. - Streaming calls are cached too. On a miss the live chunks are forwarded to the consumer while being assembled into one aggregate response — the same shape a buffered call stores, so buffered and streaming calls share one keyspace. On a hit the stored answer is replayed as provider-native chunks (OpenAI Chat deltas, OpenAI Responses lifecycle events, or Anthropic Messages events), so strict streaming clients parse it like a live stream. - Truncated or incomplete streams, and streams whose content cannot be - replayed faithfully (for example thinking blocks), are never stored; a - request whose surface cannot be inferred runs live, uncached. + Provider-terminal token-limited Chat (`finish_reason = "length"`) and + Anthropic (`stop_reason = "max_tokens"`) answers can be stored. OpenAI + Responses answers with `status = "incomplete"`, streams without a terminal + event, and streams whose content cannot be replayed faithfully (for example + thinking blocks) are never stored. A streaming request whose surface cannot + be inferred runs live, uncached. + +Streaming publication is write-behind: Relay can report end-of-stream before +the backend write finishes. `wait_for_idle()` does not wait for cache writes. +An immediate repeat can therefore run live, and concurrent identical misses +can each call the provider because the cache does not coalesce them. A buffered hit returns the stored response unchanged. A streaming hit replays -an equivalent provider-native stream. What the hit avoided — tokens and -estimated cost — is reported on the `response_cache` mark, never by editing a -buffered response body. +an equivalent provider-native stream. Available saved-token and estimated-cost +values are reported on the `response_cache` mark, never by editing a buffered +response body. ## Cache Keys @@ -311,17 +338,26 @@ normalization, under the default `key_strategy = "exact_request"`: fingerprinted instead — that fallback can only cost a miss, never a wrong hit. - Field order and whitespace never matter (RFC 8785 canonicalization). -- Volatile request fields are always dropped from the key: `stream`, `user`, - `metadata`, `service_tier`, and `store`. Add project-specific noise fields - with `skip_keys`. +- Only the built-in noise fields `stream`, `user`, `metadata`, and `store` are + dropped. All other normalized or raw provider controls remain in the key, + including `service_tier`; there is no configurable skip list. +- OpenAI Chat requests preserve whether the caller used `max_tokens` or + `max_completion_tokens`, because providers can treat the two fields + differently. - Tool-call IDs are normalized, so randomly generated per-call IDs do not fragment the keyspace. -- Request headers stay out of the key unless explicitly named in - `header_allowlist`; auth headers are rejected outright. -- The provider name is folded into every key, so different providers do not - share entries. The `namespace` is also folded into the key; assign distinct - values to isolate environments or tenants. Identical or empty namespaces can - share entries. +- Request headers stay out of the key unless named in `header_allowlist`. + Allowlist every trusted, non-secret response-affecting header; an omitted + header does not partition the key. Known auth headers are rejected, but + validation cannot recognize every custom credential name, so never allowlist + credentials. +- The provider name and required namespace partition every key. A + Switchyard-selected backend ID is also partitioned automatically. Without + Switchyard, a provider name cannot distinguish upstreams that reuse that + name, so use separate configurations and namespaces for different trusted + upstream domains. +- Requests containing integers outside the exactly representable RFC 8785 + range (less than `-2^53` or greater than `2^53`) bypass the cache. ## Observability @@ -331,15 +367,15 @@ Every cache decision emits a `response_cache` mark with | Status | Meaning | |---|---| | `hit` | Served the stored answer; the provider was skipped. | -| `miss` | No stored answer; the call ran live, and a cacheable result is stored when the write succeeds. | +| `miss` | No entry was served; the call ran live. After an ordinary lookup miss, Relay attempts to store a cacheable result. | | `bypass` | The request is not cacheable, or the `bypass_rate` sampler chose to run live. | -Mark metadata rides under `nemo_relay.response_cache.*`: `backend`, -`surface`, `key_hash` (the `sha256:…` fingerprint), `ttl_ms`, `age_ms` and -`saved_tokens` / `saved_cost_usd` on hits, and a `reason` on bypasses and -store-error misses (for example `sampled`, `stateful_store`, `store_error`, -`stream_no_codec`). Marks carry only fingerprints — never prompts, answers, or -credentials. +Mark attributes use `nemo_relay.response_cache.*`: `backend`, `surface`, +`key_hash` (the `sha256:…` fingerprint), `ttl_ms`, and `age_ms` as applicable; +`saved_tokens` and `saved_cost_usd` appear on hits when they can be derived. A +`reason` appears on bypasses and store-error misses (for example `sampled`, +`stateful_store`, `store_error`, or `stream_no_codec`). Cache marks never +include prompts, answers, or credentials. `nemo-relay doctor` reports the cache state: `not configured` when the section is absent, `configured but disabled (adaptive plugin disabled)` when the @@ -350,36 +386,42 @@ a failure when the config is invalid or the backend is unreachable. | Field | Default | Notes | |---|---|---| -| `ttl_seconds` | `3600` | How long a stored answer stays reusable, in seconds. | -| `namespace` | `""` | Folded into every key. Set distinct values to separate environments, tenants, and experiments. | +| `ttl_seconds` | `3600` | Maximum entry age in seconds; a backend can evict an entry sooner. | +| `namespace` | `""` (unconfigured) | Required non-empty trust-domain partition folded into every key. Empty or whitespace-only values are rejected. | | `priority` | `50` | LLM execution intercept priority. Lower values run earlier. | | `bypass_rate` | `0.0` | Probability in `[0.0, 1.0]` of running a cacheable call live. A sampled call attempts to refresh the entry when its result is cacheable and the write succeeds. | -| `cache_nondeterministic` | `false` | Nondeterministic requests bypass the cache. Set `true` to cache and reuse sampled responses. | +| `cache_nondeterministic` | `false` | Only requests with an explicit numeric `temperature = 0` are eligible. Set `true` to cache and reuse sampled responses. | | `key_strategy` | `"exact_request"` | The only supported strategy: reuse requires the same normalized request. | -| `header_allowlist` | `[]` | Headers folded into the key (case-insensitive). Auth headers are rejected. | -| `skip_keys` | `[]` | Extra top-level body keys dropped from the key, beyond the built-in volatile set. | +| `header_allowlist` | `[]` | Trusted, non-secret response-affecting headers folded into the key (case-insensitive). Known auth headers are rejected. | | `backend.kind` | `"in_memory"` | `"in_memory"`, or `"redis"` (requires building with the `redis-backend` feature). | | `backend.config.max_bytes` | 256 MiB | In-memory size budget; the oldest entries are evicted first. | | `backend.config.url` | — | Redis connection URL. Required for the `redis` backend. | | `backend.config.key_prefix` | `"nemo-relay:llm-cache:"` | Prefix for keys in Redis. | +For a gateway that uses Switchyard, `switchyard.priority` must be lower than +`response_cache.priority`. To derive keys before ACG rewrites requests, set +`response_cache.priority` lower than `acg.priority`. With all three components, +priorities of `0`, `40`, and `50`, respectively, satisfy both orderings. + Cached responses are stored unredacted. PII sanitize guardrails rewrite emitted -telemetry, never payloads, so the store holds full response bodies. Requests -contribute only a SHA-256 fingerprint to the cache key. A shared Redis backend -must be trusted, access-controlled, and namespaced per environment and tenant. +telemetry, never payloads, so the store holds full response bodies. Cache +entries can also store provider and model diagnostics plus the key fingerprint; +they do not store full request bodies or headers. A shared Redis backend must be +trusted and access-controlled. Use a separate configuration and namespace for +each mutually untrusted tenant or upstream domain. ## Common Validation Failures -- `ttl_seconds` is `0`, or `bypass_rate` is outside `[0.0, 1.0]`. +- `namespace` is empty or whitespace-only, `ttl_seconds` is `0`, or + `bypass_rate` is outside `[0.0, 1.0]`. - `key_strategy` is not `"exact_request"`. -- `skip_keys` names an answer-determining field such as `messages`, `model`, - `tools`, or `tool_choice`. - `header_allowlist` names an auth header such as `authorization` or `x-api-key`. -- in_memory `max_bytes` set to zero or a non-integer. -- `backend.kind = "redis"` without `backend.config.url`, or in a build without - the `redis-backend` feature. -- A `redis` backend with an empty `namespace` warns: every caller of the - shared store would exchange cached answers. +- `in_memory` `max_bytes` is zero or is not an integer. +- `backend.kind` is unknown; or `redis` has a missing, non-string, or + whitespace-only `backend.config.url`, uses a non-string `key_prefix`, or is + unavailable because Relay was built without the `redis-backend` feature. +- Gateway Switchyard priority is equal to or greater than + `response_cache.priority`. diff --git a/docs/configure-plugins/switchyard/configuration.mdx b/docs/configure-plugins/switchyard/configuration.mdx index 2b09b555e..20ae161f9 100644 --- a/docs/configure-plugins/switchyard/configuration.mdx +++ b/docs/configure-plugins/switchyard/configuration.mdx @@ -138,7 +138,7 @@ The following table describes the Decision API and rollout settings: | --- | --- | --- | | `version` | `1` | Component configuration version. | | `mode` | `"enforce"` | `enforce` applies valid decisions. `observe_only` records the hypothetical decision but dispatches the trusted same-protocol default once. | -| `priority` | `0` | Component execution priority. | +| `priority` | `0` | Component execution priority. When response caching is enabled in the gateway, this value must be lower than `response_cache.priority` so Switchyard selects the backend before Relay derives the cache key. | | `decision_api_url` | Required | HTTP or HTTPS URL for the Switchyard Decision API. | | `decision_profile_id` | Required | Non-empty Switchyard profile identifier sent with each routing request. | | `decision_timeout_millis` | `25` | Decision API timeout in milliseconds; must be greater than zero. | diff --git a/docs/nemo-relay-cli/basic-usage.mdx b/docs/nemo-relay-cli/basic-usage.mdx index 13417c964..53dca3adb 100644 --- a/docs/nemo-relay-cli/basic-usage.mdx +++ b/docs/nemo-relay-cli/basic-usage.mdx @@ -44,10 +44,11 @@ observability is required. - `POST /v1/messages/count_tokens` - `GET /v1/models` -The gateway forwards raw provider JSON without rewriting OpenAI or Anthropic -payload schemas. It removes only hop-by-hop transport headers, forwards -streaming responses as streams, and emits NeMo Relay LLM start and end events -under the active session scope. +The gateway routes supported provider payloads through managed middleware. It +does not implicitly convert between OpenAI and Anthropic schemas, but configured +middleware such as Switchyard can translate them. Managed calls emit LLM start +and end events under the active session scope, and successful streams remain +streams. Model-list requests and startup probes use unmanaged passthrough. ## Transparent Run @@ -242,6 +243,19 @@ follows: | `/models`, `/v1/models` | `openai_base_url` | | `/v1/messages`, `/v1/messages/count_tokens` | `anthropic_base_url` | +For managed LLM routes, the middleware-selected result owns the client +response. Relay serializes a successful buffered result as HTTP `200` +`application/json`, or re-encodes a successful stream as HTTP `200` +`text/event-stream`. It does not preserve successful upstream status, headers, +or original JSON formatting. This keeps the response coherent when retry or +hedging middleware selects among concurrent attempts. + +For ordinary managed requests, an upstream HTTP error or undecodable buffered +response preserves that attempt's status, bytes, and forwardable response +headers. Retry-aware routing receives structured failures so it can select +another target. A transport failure before Relay starts the downstream response +uses Relay's JSON error envelope with HTTP `502`. + For ordinary provider requests, Relay resolves credentials in this order: 1. Relay preserves an inbound `Authorization`, `x-api-key`, `api-key`, or diff --git a/docs/reference/migration-guides.mdx b/docs/reference/migration-guides.mdx index 8e11c37ea..91d95bdf2 100644 --- a/docs/reference/migration-guides.mdx +++ b/docs/reference/migration-guides.mdx @@ -27,6 +27,10 @@ for 0.7 before using those surfaces. NeMo Relay does not adapt synchronous Rust middleware callbacks or one-argument LLM sanitizer callbacks. +Rust code that constructs `AdaptiveConfig` with an exhaustive struct literal +must add `response_cache: None`. Prefer `..AdaptiveConfig::default()` when the +literal should remain compatible with new optional fields. + ### Migrate Middleware Callbacks The following callback families are now asynchronous: conditional execution From 86dc2e60984ac21771e1ddf9204347f512fedfbb Mon Sep 17 00:00:00 2001 From: Zhongxuan Wang Date: Thu, 30 Jul 2026 17:26:39 -0700 Subject: [PATCH 6/7] docs(cli): clarify managed gateway responses Signed-off-by: Zhongxuan Wang --- docs/nemo-relay-cli/basic-usage.mdx | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/docs/nemo-relay-cli/basic-usage.mdx b/docs/nemo-relay-cli/basic-usage.mdx index 53dca3adb..c60d61de6 100644 --- a/docs/nemo-relay-cli/basic-usage.mdx +++ b/docs/nemo-relay-cli/basic-usage.mdx @@ -243,18 +243,21 @@ follows: | `/models`, `/v1/models` | `openai_base_url` | | `/v1/messages`, `/v1/messages/count_tokens` | `anthropic_base_url` | -For managed LLM routes, the middleware-selected result owns the client -response. Relay serializes a successful buffered result as HTTP `200` -`application/json`, or re-encodes a successful stream as HTTP `200` -`text/event-stream`. It does not preserve successful upstream status, headers, -or original JSON formatting. This keeps the response coherent when retry or -hedging middleware selects among concurrent attempts. - -For ordinary managed requests, an upstream HTTP error or undecodable buffered -response preserves that attempt's status, bytes, and forwardable response -headers. Retry-aware routing receives structured failures so it can select -another target. A transport failure before Relay starts the downstream response -uses Relay's JSON error envelope with HTTP `502`. +For managed LLM routes, the middleware-selected result determines the client +response: + +- A successful buffered result becomes HTTP `200` `application/json`. +- A successful stream becomes HTTP `200` `text/event-stream` and Relay + re-encodes its parsed events as SSE. +- For an ordinary managed request, an upstream HTTP error or undecodable + buffered response preserves the selected attempt's status, bytes, and + forwardable response headers. +- A transport failure before Relay starts the downstream response uses Relay's + JSON error envelope with HTTP `502`. + +Successful responses do not preserve the upstream status, headers, or original +JSON formatting. Retry-aware routing receives structured failures so it can +select another target. For ordinary provider requests, Relay resolves credentials in this order: From 8c7c1ab96226566e2a18d03fdd7469acd4105805 Mon Sep 17 00:00:00 2001 From: Zhongxuan Wang Date: Thu, 30 Jul 2026 18:00:55 -0700 Subject: [PATCH 7/7] docs: keep response cache guidance scoped Signed-off-by: Zhongxuan Wang --- .../switchyard/configuration.mdx | 2 +- docs/nemo-relay-cli/basic-usage.mdx | 25 +++---------------- 2 files changed, 5 insertions(+), 22 deletions(-) diff --git a/docs/configure-plugins/switchyard/configuration.mdx b/docs/configure-plugins/switchyard/configuration.mdx index 20ae161f9..2b09b555e 100644 --- a/docs/configure-plugins/switchyard/configuration.mdx +++ b/docs/configure-plugins/switchyard/configuration.mdx @@ -138,7 +138,7 @@ The following table describes the Decision API and rollout settings: | --- | --- | --- | | `version` | `1` | Component configuration version. | | `mode` | `"enforce"` | `enforce` applies valid decisions. `observe_only` records the hypothetical decision but dispatches the trusted same-protocol default once. | -| `priority` | `0` | Component execution priority. When response caching is enabled in the gateway, this value must be lower than `response_cache.priority` so Switchyard selects the backend before Relay derives the cache key. | +| `priority` | `0` | Component execution priority. | | `decision_api_url` | Required | HTTP or HTTPS URL for the Switchyard Decision API. | | `decision_profile_id` | Required | Non-empty Switchyard profile identifier sent with each routing request. | | `decision_timeout_millis` | `25` | Decision API timeout in milliseconds; must be greater than zero. | diff --git a/docs/nemo-relay-cli/basic-usage.mdx b/docs/nemo-relay-cli/basic-usage.mdx index c60d61de6..13417c964 100644 --- a/docs/nemo-relay-cli/basic-usage.mdx +++ b/docs/nemo-relay-cli/basic-usage.mdx @@ -44,11 +44,10 @@ observability is required. - `POST /v1/messages/count_tokens` - `GET /v1/models` -The gateway routes supported provider payloads through managed middleware. It -does not implicitly convert between OpenAI and Anthropic schemas, but configured -middleware such as Switchyard can translate them. Managed calls emit LLM start -and end events under the active session scope, and successful streams remain -streams. Model-list requests and startup probes use unmanaged passthrough. +The gateway forwards raw provider JSON without rewriting OpenAI or Anthropic +payload schemas. It removes only hop-by-hop transport headers, forwards +streaming responses as streams, and emits NeMo Relay LLM start and end events +under the active session scope. ## Transparent Run @@ -243,22 +242,6 @@ follows: | `/models`, `/v1/models` | `openai_base_url` | | `/v1/messages`, `/v1/messages/count_tokens` | `anthropic_base_url` | -For managed LLM routes, the middleware-selected result determines the client -response: - -- A successful buffered result becomes HTTP `200` `application/json`. -- A successful stream becomes HTTP `200` `text/event-stream` and Relay - re-encodes its parsed events as SSE. -- For an ordinary managed request, an upstream HTTP error or undecodable - buffered response preserves the selected attempt's status, bytes, and - forwardable response headers. -- A transport failure before Relay starts the downstream response uses Relay's - JSON error envelope with HTTP `502`. - -Successful responses do not preserve the upstream status, headers, or original -JSON formatting. Retry-aware routing receives structured failures so it can -select another target. - For ordinary provider requests, Relay resolves credentials in this order: 1. Relay preserves an inbound `Authorization`, `x-api-key`, `api-key`, or