officialCodeWork · officialCodeWork · May 25, 2026 · May 25, 2026
diff --git a/TRACKER.md b/TRACKER.md
@@ -9,8 +9,10 @@
 > CI matrix always includes `ubuntu-22.04`, `macos-14`, and `windows-latest`.
 
 **Last updated:** 2026-05-25
-**Current phase:** Phase 1 — Ingestion + Knowledge Store
-**Next action:** Phase 1 Step 1.10 — Write path & ingest API: wire the full pipeline end-to-end (connector → parse → chunk → enrich → PII → embed → store) and expose `POST /v1/ingest`. This is the **CLI demo milestone** — end-to-end ingest works from a folder of documents.
+**Current phase:** Phase 2 — Retrieval Engine (Phase 1 complete 🎉)
+**Next action:** Phase 2 Step 2.1 — Knowledge store read layer: read interfaces for VectorStore, KeywordStore, GraphStore backends; filter push-down via `FilterExpr` from `rag-policy`. First step of the retrieval engine; closes the loop on the ingest pipeline (Step 1.10) so queries can hit indexed chunks.
+
+**Phase 1 complete — CLI demo milestone delivered.** `ragctl ingest <folder>` runs a folder of mixed-format documents through the full pipeline (connector → parse → chunk → enrich → PII → embed → store) and `POST /v1/ingest/document` exposes the same pipeline over HTTP. Next milestone on the demo ladder is **🥈 Curl-able RAG** at Step 3.1 (gateway query surface), which requires all of Phase 2 first.
 
 > **Refactor window (Steps 1.1a–1.1f):** Before resuming the connectors framework (1.2), we insert a six-step refactor that locks in architecture + optimization decisions which are very expensive to retrofit later (PolicyEngine PDP, RequestContext-threaded SPIs, split Retrieval/Index backends, bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split, hot-path discipline). See [docs/adr/ADR-0005…0009] and [docs/architecture/policy-engine.md], [request-context.md], [caching.md], [performance.md].
 
@@ -32,14 +34,14 @@
 | Phase | Title | Steps | ✅ Done | Remaining |
 |-------|-------|------:|-------:|----------:|
 | 0 | Foundation | 13 | **13** | 0 |
-| 1 | Ingestion + Knowledge Store | 16 | **15** | 1 |
+| 1 | Ingestion + Knowledge Store | 16 | **16** | 0 |
 | 2 | Retrieval Engine | 11 | 0 | 11 |
 | 3 | Gateway & Agent Runtime | 11 | 0 | 11 |
 | 4 | Reliability | 6 | 0 | 6 |
 | 5 | Eval & Observability | 7 | 0 | 7 |
 | 6 | Governance & Tenancy | 10 | 0 | 10 |
 | 7 | Pilot, Harden, GA | 10 | 0 | 10 |
-| **Total** | | **84** | **28** | **56** |
+| **Total** | | **84** | **29** | **55** |
 
 ---
 
@@ -82,7 +84,7 @@
 | 1.7 | PII detection | ✅ | `build/phase-1/step-1.7-pii-detection` | [#66](https://github.com/officialCodeWork/AgentContextOS/pull/66) | New `rag-pii` package (`packages/pii/`, v0.1.0): `RegexPIIDetector` (pure-Python, always available — EMAIL/PHONE/SSN/CREDIT_CARD/IP_ADDRESS with type-specific confidence scores and Luhn-validated credit cards; regex patterns mirror `rag_observability.events.check_pii` so the Step 0.7b log-leak CI gate and ingest detection share one definition of PII) + `PresidioPIIDetector` (optional `[presidio]` extra; lazy `AnalyzerEngine` init; overlap resolution matches the regex path). `PiiProcessor` applies `ctx.pii_policy.action` across the chunk list — **redact** (default, `<TYPE>` markers), **mask** (`*` of equal length, preserves layout), **block** (chunk dropped) — pre-filters spans by `policy.entities` + `policy.min_score`, and returns `(filtered_chunks, outcomes)` so callers see exactly what happened per chunk without re-running the detector. Pure helpers `redact_spans()` / `mask_spans()` exported for custom flows. Event protocol: one `pii.detected` `PiiEvent` per entity type per affected chunk, carrying only metadata (`field_name=chunk:<id>`, `pii_type`, `action_taken`, plus `chunk_id`/`document_id`/`principal_id`/`span_count` in `metadata`) — never raw matched text, so the existing log-leak CI gate keeps passing. SPI evolution: `PIISpan` promoted from dataclass to Pydantic v2 frozen model in `rag_core.types` (validates `end >= start` + 0–1 score); schema exported (`PIISpan.json`). `ragctl pii <path>` smoke command (`--detector regex\|presidio`, `--action redact\|mask\|block`, `--chunks N`) — 5 CLI tests. 35 unit tests under `tests/pii/` (rewriters, regex incl. parametrised Luhn, processor, Presidio stubbed at `sys.modules`). 8 contract tests cover SPI + PIISpan validation. `rag-core` 0.11.0 → 0.12.0. Cross-platform: core pure-Python; `[presidio]` extra needs `spacy download en_core_web_sm` separately. New docs: [reference/pii.md](docs/reference/pii.md), [architecture/pii.md](docs/architecture/pii.md). |
 | 1.8 | Embedder pipeline | ✅ | `build/phase-1/step-1.8-embedder-pipeline` | [#68](https://github.com/officialCodeWork/AgentContextOS/pull/68) | New `rag-embedders` package (`packages/embedders/`, v0.1.0): **3 plugin classes covering 4 model families** behind the existing `Embedder` SPI. `OpenAIEmbedder` (`[openai]` extra) covers `text-embedding-3-{small,large}` + `ada-002` with server-side dim reduction for v3 models via the `dimensions` request parameter; `CohereEmbedder` (`[cohere]` extra) covers `embed-{english,multilingual}-v3.0` with SDK v4/v5 response-shape probing and `search_document` default `input_type`; `SentenceTransformersEmbedder` (`[sentence-transformers]` extra) covers any HuggingFace model, with `bge_large_en()` and `e5_large_v2()` factory helpers — E5 passage prefix auto-applied via `_PASSAGE_PREFIXES`; encoding runs in `asyncio.to_thread` so the sync API doesn't block the loop. All three subclass `BatchingEmbedder` for shared sub-batch splitting (OpenAI 2048 / Cohere 96 / local 32), exponential-back-off retry via `RetryPolicy` (`retry_on` tuple declares which provider exceptions are transient), and dimension normalization (truncate + L2 renorm). `normalize_dimension()` exposed as a pure helper. `ragctl embed <path>` smoke command — default `noop` backend exercises the full parse → chunk → enrich → PII → embed pipeline without provider credentials; `--backend openai\|cohere\|bge\|e5` selects real backends. 41 unit tests under `tests/embedders/` (dim + retry + base + 3 backend test files with each provider stubbed at `sys.modules`); 5 CLI tests. PolicyEngine coverage linter allowlist extended to cover `ragctl/main.py` (operator CLI, not production policy boundary). Cross-platform: core pure-Python; `[sentence-transformers]` extra pulls in PyTorch (~600 MB) but ships wheels for ubuntu/macos/windows. New docs: [reference/embedders.md](docs/reference/embedders.md), [architecture/embedders.md](docs/architecture/embedders.md). |
 | 1.9 | CDC connectors (incremental sync) | ✅ | `build/phase-1/step-1.9-cdc-connectors` | [#70](https://github.com/officialCodeWork/AgentContextOS/pull/70) | Three change-data-capture connectors + a fingerprint-based dedup wrapper, all satisfying the existing `Connector` SPI so the Step 1.10 ingest pipeline drives them uniformly with bulk crawlers. `PostgresCDCConnector` (`[postgres-cdc]` extra — `psycopg[binary]`) uses the built-in `pgoutput` logical-replication plugin; WAL LSN as cursor; INSERT/UPDATE for the same primary key collapse to `Document.id="pg:<table>:<pk>"` so the latest value wins; DELETE shares the id with `metadata["cdc_op"]=="D"` so the ingest pipeline can treat it as a tombstone. `S3EventsConnector` (uses existing aioboto3) long-polls SQS for S3 event notifications, parses both direct-to-SQS and SNS-wrapped envelopes, filters cross-bucket records; tombstones (`ObjectRemoved`) get a synthesised content hash since eTag is absent for deletes; SQS messages are deleted only after their records are emitted (at-least-once delivery, dedup helper covers the exactly-once gap). `WebhookReceiverConnector` (pure stdlib) is push-style — wraps an `asyncio.Queue` that the gateway's webhook route fills via `publish(document, content)`; three backpressure modes (`wait`/`drop`/`raise`) for different operator preferences. `DedupFilter` wraps any `Connector` + `Cache` and drops emissions whose `Document.fingerprint` (== `Document.content_hash`) is already cached; tenant-namespaced cache key; TTL bounds memory + defines dedup horizon; itself satisfies the `Connector` SPI so it composes anywhere; exposes `DedupStats(seen, deduped)` for ingest-pipeline reporting. `Document.fingerprint` property added to `rag-core` (returns `content_hash`, exposed so storage layout is hideable). 35 unit tests under `tests/backends/` covering dedup/webhook/S3-events/postgres-cdc — all SDKs stubbed at `sys.modules` so CI runs without real Postgres / AWS / event sources. Cross-platform: webhook is pure-stdlib; S3 events reuses default aioboto3; `[postgres-cdc]` ships pre-built psycopg wheels for ubuntu/macos/windows (no system libpq). New docs sections in [reference/connectors.md](docs/reference/connectors.md) (CDC section + DedupFilter section + extended error table) and [architecture/connectors.md](docs/architecture/connectors.md) (pgoutput choice rationale, S3-event transport shapes, push-style webhook design, fingerprint dedup semantics). |
-| 1.10 | Write path & ingest API | ⏳ | — | — | Full ingest pipeline wired end-to-end: connector → parse → chunk → enrich → PII → embed → store; `POST /v1/ingest` |
+| 1.10 | Write path & ingest API | ✅ | `build/phase-1/step-1.10-write-path-ingest-api` | [#72](https://github.com/officialCodeWork/AgentContextOS/pull/72) | **CLI demo milestone — Phase 1 complete.** New `rag-ingest` package (`packages/ingest/`, v0.1.0): `IngestPipeline` orchestrator stitches every Phase 1 stage into one async call — `ingest_document(ctx, document, content, *, mime_type=None) → IngestResult` and `ingest_connector(ctx, connector, *, on_result=None) → IngestRunSummary`. Stage order parse → chunk → enrich → PII → embed → index (PII runs before embed so vectors only ever encode policy-compliant text — vector similarity attacks can't reconstruct PII). Optional `PolicyEngine.evaluate(ingest_doc, …)` short-circuits before parse (deny → `status=blocked_by_policy` without paying for parse/chunk/embed). Per-document errors caught and surfaced as `status=failed` so one bad doc never aborts a run; `KeyboardInterrupt`/`SystemExit` deliberately propagate. Package depends only on `rag-core`; `rag_policy.types` imported lazily inside `_policy_check` so rag-ingest itself stays optional-policy. SPI evolution: `IngestStatus` enum + `IngestResult` + `IngestRunSummary` Pydantic frozen models in `rag_core.types`; schema exported (`IngestResult.json`, `IngestRunSummary.json`); `rag-core` 0.12.0 → 0.13.0. FastAPI gateway 0.2.0 → 0.3.0: `POST /v1/ingest/document` multipart upload (tenant_id/corpus_id/principal_id/source_uri form fields + file part) returning `IngestResult` JSON; `GET /healthz`; `GET /v1/info`; `build_app(pipeline=None)` factory for DI; module-level `app` for `uvicorn rag_gateway.app:app`; `python-multipart` dep added. `ragctl ingest <path>` wired for real (replaces Step 0.10 scaffold) — directory crawl with summary, `--per-doc` for streaming, single-file mode, `--tenant`/`--corpus` overrides; uses `FilesystemConnector` + noop embedder/vector store so it runs anywhere. 38 new tests: 16 IngestPipeline unit (happy path / unsupported MIME / MIME override / PII metadata / embeddings indexed / blocked_by_policy short-circuit / skipped_empty / PII block drops chunks / error wrapping / MIME fallback) + 6 gateway FastAPI TestClient + 6 ragctl ingest CLI + scaffold-test cleanup. Cross-platform: pure-Python throughout (python-multipart is a pure wheel); no infra required for the CLI demo. Policy boundary: IngestPipeline IS the canonical `ingest_doc` PolicyEngine consumer (call site, not a bypasser); coverage linter still passes because the consultation lives in `_policy_check` with the PolicyDecision marker. Deliberately deferred: rag.yaml → backend wiring (Step 3.5), per-stage observability spans (Step 5.1), AuditWriter emissions for ingest events (Step 6.6), outbound webhooks (Step 3.9). New docs: [reference/ingest.md](docs/reference/ingest.md), [architecture/ingest.md](docs/architecture/ingest.md). |
 
 ---
 
@@ -238,6 +240,8 @@
 | [#68](https://github.com/officialCodeWork/AgentContextOS/pull/68) | feat(embedders): embedder pipeline — OpenAI / Cohere / BGE / E5 (Step 1.8) | `build/phase-1/step-1.8-embedder-pipeline` | ✅ Merged | 2026-05-25 |
 | [#69](https://github.com/officialCodeWork/AgentContextOS/pull/69) | chore(tracker): sync Step 1.8 → ✅ and log PRs #67, #68; advance to 1.10 | `chore/tracker-sync-pr68` | ✅ Merged | 2026-05-25 |
 | [#70](https://github.com/officialCodeWork/AgentContextOS/pull/70) | feat(backends): CDC connectors + Document.fingerprint dedup (Step 1.9) | `build/phase-1/step-1.9-cdc-connectors` | ✅ Merged | 2026-05-25 |
+| [#71](https://github.com/officialCodeWork/AgentContextOS/pull/71) | chore(tracker): sync Step 1.9 → ✅ and log PRs #69, #70 | `chore/tracker-sync-pr70` | ✅ Merged | 2026-05-25 |
+| [#72](https://github.com/officialCodeWork/AgentContextOS/pull/72) | feat(ingest,gateway): write path + POST /v1/ingest (Step 1.10 — CLI demo) | `build/phase-1/step-1.10-write-path-ingest-api` | ✅ Merged | 2026-05-25 |
 
 ---