Skip to content

chore(tracker): sync Step 1.4 → ✅ and log PRs #58, #59, #60 #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
officialCodeWork merged 2 commits into from
May 25, 2026
Merged

chore(tracker): sync Step 1.4 → ✅ and log PRs #58, #59, #60 #61

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
69a416f
chore(tracker): sync Step 1.4 → ✅ and log PRs #58, #59, #60
May 25, 2026
daf5453
chore(tracker): reconcile Phase 1 milestone arithmetic
May 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions TRACKER.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
> Use `Taskfile.yml` for task targets (cross-platform), `Makefile` for Unix convenience.
> CI matrix always includes `ubuntu-22.04`, `macos-14`, and `windows-latest`.

**Last updated:** 2026-05-24
**Last updated:** 2026-05-25
**Current phase:** Phase 1 — Ingestion + Knowledge Store
**Next action:** Phase 1 Step 1.4 — OCR pipeline: Tesseract + PaddleOCR plugins, image region extraction, confidence scoring.
**Next action:** Phase 1 Step 1.5 — Structure-aware chunker: heading-based chunking, parent-child `Chunk.parent_id` links, sentence-boundary, overlap, size normalization. Consumes `ParsedDocument.blocks` from Step 1.3 and OCR regions from Step 1.4 as natural chunk boundaries.

> **Refactor window (Steps 1.1a–1.1f):** Before resuming the connectors framework (1.2), we insert a six-step refactor that locks in architecture + optimization decisions which are very expensive to retrofit later (PolicyEngine PDP, RequestContext-threaded SPIs, split Retrieval/Index backends, bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split, hot-path discipline). See [docs/adr/ADR-0005…0009] and [docs/architecture/policy-engine.md], [request-context.md], [caching.md], [performance.md].

Expand All @@ -32,14 +32,14 @@
| Phase | Title | Steps | ✅ Done | Remaining |
|-------|-------|------:|-------:|----------:|
| 0 | Foundation | 13 | **13** | 0 |
| 1 | Ingestion + Knowledge Store | 16 | **11** | 5 |
| 1 | Ingestion + Knowledge Store | 16 | **10** | 6 |
| 2 | Retrieval Engine | 11 | 0 | 11 |
| 3 | Gateway & Agent Runtime | 11 | 0 | 11 |
| 4 | Reliability | 6 | 0 | 6 |
| 5 | Eval & Observability | 7 | 0 | 7 |
| 6 | Governance & Tenancy | 10 | 0 | 10 |
| 7 | Pilot, Harden, GA | 10 | 0 | 10 |
| **Total** | | **84** | **24** | **60** |
| **Total** | | **84** | **23** | **61** |

---

Expand Down Expand Up @@ -76,7 +76,7 @@
| 1.1f | ADRs 0005–0009 + reviewer checklist | ✅ | `build/phase-1/step-1.1f-adrs-reviewer-checklist` | [#52](https://github.com/officialCodeWork/AgentContextOS/pull/52) | ADR-0005 (PolicyEngine PDP), ADR-0006 (two-stage reranker default), ADR-0007 (tiered storage with BlobRef), ADR-0008 (cost-aware planner replacing reactive fallback), ADR-0009 (vector index strategy + quantization) all promoted to Accepted with implementation-path backlinks. Reviewer checklist in `docs/architecture/performance.md` extended with explicit items for plan/budget awareness, IndexHint+dtype handling, two-stage rerank shape, and BlobRef-safe consumers. ADR backlinks added to `rag_core.spi.reranker` and `rag_core.spi.storage` (the two missing module-level pointers). |
| 1.2 | Connectors framework | ✅ | `build/phase-1/step-1.2-connectors-framework` | [#55](https://github.com/officialCodeWork/AgentContextOS/pull/55) | `Connector` SPI evolved to `(Document, ConnectorState)` async-iterator with resumable watermark (`ConnectorState` frozen model: connector_id, tenant_id, cursor, last_seen_at, extra). Internal `Crawler[Raw]` base class in `rag_backends.connectors._base` (iteration scaffold, error policy, monotonic watermark). Built-in connectors: `FilesystemConnector` (cross-platform pathlib, sha256 hashing, sorted resume cursor), `S3Connector` (aioboto3, ListObjectsV2 continuation token, MinIO-compatible), `GCSConnector` (gcloud-aio-storage, REST nextPageToken, `[gcs]` extra). Policy boundary documented — connectors are upstream of `PolicyEngine`; the ingest pipeline (Step 1.10) is the enforcement point. 9 contract tests (NoopConnector resume + watermark + tenant mismatch), 9 unit tests (FilesystemConnector), MinIO + GCS integration tests skip-if-no-service. `ConnectorState` schema added to `dist/schemas/`. New docs: [reference/connectors.md](docs/reference/connectors.md), [architecture/connectors.md](docs/architecture/connectors.md). |
| 1.3 | Document parsers | ✅ | `build/phase-1/step-1.3-document-parsers` | [#57](https://github.com/officialCodeWork/AgentContextOS/pull/57) | New `rag-parsers` package (`packages/parsers/`, v0.1.0) shipping `PlainTextParser`, `MarkdownParser` (markdown-it-py), `HtmlParser` (BeautifulSoup + stdlib `html.parser`), `JsonParser`, `CsvParser`, `YamlParser` (PyYAML `safe_load`), `PdfParser` (pypdf), `DocxParser` (python-docx), `PptxParser` (python-pptx), `XlsxParser` (openpyxl). `Parser` SPI evolved: `parse(...)` now returns `ParsedDocument` (`Document` + extracted text + structural `Block` list with offsets + heading levels + parser-specific `attributes`). New types in `rag_core`: `Block`, `BlockType` (8-value enum), `ParsedDocument`. `rag_parsers.mime.detect_mime()` (puremagic + OOXML ZIP central-directory disambiguation + extension shortcut + hint fallback). `ParserRegistry` + `default_registry()` (first-match-wins; specific MIMEs before generics). `ragctl parse <path>` smoke-test command (MIME detect → parser select → print summary + first-N blocks). 35 parser unit tests + 6 ragctl tests + extended contract tests (148 contract tests total pass). Schemas regenerated (`Block.json`, `ParsedDocument.json`). `rag-core` 0.7.0 → 0.8.0. Cross-platform: every parser library is pure-Python or wheel-distributed; no `libmagic` / `lxml` system deps. New docs: [reference/parsers.md](docs/reference/parsers.md), [architecture/parsers.md](docs/architecture/parsers.md). |
| 1.4 | OCR pipeline | ⏳ | — | — | Tesseract + PaddleOCR plugins; image region extraction; confidence scoring |
| 1.4 | OCR pipeline | ✅ | `build/phase-1/step-1.4-ocr-pipeline` | [#60](https://github.com/officialCodeWork/AgentContextOS/pull/60) | New `rag-ocr` package (`packages/ocr/`, v0.1.0): `TesseractOCR` (per-word regions, ISO-639-3 lang codes, `pytesseract` wrapper) + `PaddleOCRBackend` (per-line regions, quadrilateral → axis-aligned bbox, lazy engine init); both behind optional extras `[tesseract]`/`[paddle]` so default install stays pure-Python. OCR SPI evolved: `BoundingBox(x0,y0,x1,y1)` + `OCRRegion(text,confidence,bbox,page_number)` + `OCRResult(text,confidence,regions,page_number,engine)` Pydantic v2 frozen models in `rag_core.types`; aggregate confidence is text-length-weighted mean (1.0 on empty regions so "no text" doesn't read as failure); `NoopOCR` migrated; 5 contract tests updated. `ragctl ocr <path>` smoke command (`--engine tesseract\|paddleocr`, `--lang`, `--regions N`) — 5 CLI tests. 22 unit tests under `tests/ocr/` stub both engines at the `sys.modules` boundary so CI runs without Tesseract binary or PaddlePaddle wheels. Schemas regenerated (`BoundingBox.json`, `OCRRegion.json`, `OCRResult.json`). `rag-core` 0.8.0 → 0.9.0. New docs: [reference/ocr.md](docs/reference/ocr.md), [architecture/ocr.md](docs/architecture/ocr.md). |
| 1.5 | Structure-aware chunker | ⏳ | — | — | Heading-based chunking, parent-child `Chunk.parent_id` links, sentence-boundary, overlap, size normalization |
| 1.6 | Metadata enricher | ⏳ | — | — | Auto-tag: language, doc type, created/modified dates, author, section path, reading level |
| 1.7 | PII detection | ⏳ | — | — | Presidio integration; per-tenant policy (block/redact/allow); `pii.detected` event; CI gate |
Expand Down Expand Up @@ -225,6 +225,9 @@
| [#55](https://github.com/officialCodeWork/AgentContextOS/pull/55) | feat(core,backends): connectors framework — SPI evolution + filesystem/S3/GCS (Step 1.2) | `build/phase-1/step-1.2-connectors-framework` | ✅ Merged | 2026-05-24 |
| [#56](https://github.com/officialCodeWork/AgentContextOS/pull/56) | chore(tracker): sync Step 1.2 → ✅ and log PR #55 | `chore/tracker-sync-pr55` | ✅ Merged | 2026-05-24 |
| [#57](https://github.com/officialCodeWork/AgentContextOS/pull/57) | feat(core,parsers): document parsers — PDF/DOCX/PPTX/XLSX/HTML/MD/text/JSON/CSV/YAML (Step 1.3) | `build/phase-1/step-1.3-document-parsers` | ✅ Merged | 2026-05-24 |
| [#58](https://github.com/officialCodeWork/AgentContextOS/pull/58) | chore(tracker): sync Step 1.3 → ✅ and log PRs #56, #57 | `chore/tracker-sync-pr57` | ✅ Merged | 2026-05-25 |
| [#59](https://github.com/officialCodeWork/AgentContextOS/pull/59) | fix(parsers): normalize CRLF/CR to LF in text & markdown parsers | `fix/parsers-windows-crlf` | ✅ Merged | 2026-05-25 |
| [#60](https://github.com/officialCodeWork/AgentContextOS/pull/60) | feat(core,ocr): OCR pipeline — region-aware SPI + Tesseract + PaddleOCR (Step 1.4) | `build/phase-1/step-1.4-ocr-pipeline` | ✅ Merged | 2026-05-25 |

---

Expand Down
Loading