Skip to content

cache.md

Latest commit

 

History

History
132 lines (99 loc) · 4.46 KB

cache.md

File metadata and controls

132 lines (99 loc) · 4.46 KB
Raw

rag-cache — tiered semantic cache reference

Step 4.1. Production implementations behind the RetrievalCache and AnswerCache SPIs (defined in rag-core — see caching.md).

Overview

rag-cache provides three tiers and a composition:

Tier Class Where Notes
L0 InMemoryRetrievalCache / InMemoryAnswerCache / InMemoryEmbeddingCache rag-cache LRU + TTL, per-process
L1 RedisRetrievalCache / RedisAnswerCache rag-backends exact, shared, injected as l1=
L2 SemanticRetrievalCache / SemanticAnswerCache rag-cache token-set Jaccard over recent queries
— TieredRetrievalCache / TieredAnswerCache rag-cache compose L0→L1→L2 + events + stats

TtlLruCache is the generic L0 engine.

Usage

Default (in-memory) tiered cache

from rag_cache import TieredRetrievalCache

cache = TieredRetrievalCache()  # L0 + L2, no L1

await cache.put_with_query(
    ctx, query_text="how does bm25 work",
    plan_hash=h, corpus_version=v, value=chunk_refs, scope=params_hash,
)

# Exact (L0/L1) → semantic (L2) on the query text:
refs = await cache.get_best(
    ctx, query_text="how does bm25 scoring work",
    plan_hash=h2, corpus_version=v, scope=params_hash,
)

get / put are the exact-only SPI methods (no L2); get_best / put_with_query add the L2 similarity tier and take a query_text + a scope that pins the non-text parameters (top_k, filters, corpus, ACL).

Production: inject the Redis L1 tier

from rag_cache import TieredRetrievalCache
from rag_backends import RedisRetrievalCache

cache = TieredRetrievalCache(
    l1=RedisRetrievalCache(url="redis://localhost:6379/0"),
    semantic_threshold=0.5,
)

Hit-rate stats

stats = cache.stats()          # TierStats snapshot
stats.l0_hits, stats.l1_hits, stats.l2_hits, stats.misses
stats.hits, stats.lookups, stats.hit_rate

The generic L0 engine

from rag_cache import TtlLruCache

lru: TtlLruCache[str, bytes] = TtlLruCache(max_entries=1024, default_ttl_seconds=3600)
lru.set("k", b"v")
lru.get("k")            # b"v" | None (None on miss or TTL expiry)
lru.evict_matching(lambda key: key.startswith("t1:"))

Configuration

rag.yaml backends.cache:

backends:
  cache:
    provider: redis          # none/memory → L0+L2 only; redis/valkey → +L1
    tiered: true
    l0_max_entries: 4096
    retrieval_ttl_seconds: 3600
    answer_ttl_seconds: 900
    semantic:
      enabled: true
      threshold: 0.5          # token-set Jaccard a paraphrase must clear
      max_entries_per_tenant: 1024

Events

The tiered cache emits (via rag-observability):

  • cache.hit — cache (retrieval/answer), tier (l0/l1/l2), latency_ms.
  • cache.miss — tier=none.
  • cache.invalidated — tier=all, entries_removed.

It also opens a cache.lookup OTel span with rag.cache.{name,tier,hit,elapsed_ms}.

Internals

  • Read-through + promotion — a lookup tries L0, then L1 (warming L0 on a hit), then L2 similarity (warming L0 under the current plan hash).
  • Write-through — put populates L0 + L1; put_with_query also registers the query text in the L2 index.
  • Reliability — every L1 (network) call is wrapped: a failure degrades to a miss (read) or skip (write), logged, never raised. L0 + L2 are in-process and cannot fail. health() reflects L0 (the serving floor); an unhealthy L1 does not fail the cache.
  • Tenant isolation — every key bakes in ctx.tenant_id; the L2 index is per-tenant. Asserted by tests/redteam/test_cross_tenant_cache.py.

Extension points

  • Swap L1 — inject any RetrievalCache / AnswerCache as l1= (Valkey, Memcached, a custom backend).
  • Swap L2 similarity — SemanticRetrievalCache / the tiered cache's internal QuerySimilarityIndex use token-set Jaccard; a vector-indexed variant can replace the linear scan behind the same get_best API.
  • Redis tag invalidation — RedisRetrievalCache.invalidate_corpus and RedisAnswerCache.invalidate_policy use per-corpus / per-policy tag sets for precise eviction.

Related