officialCodeWork · officialCodeWork · May 24, 2026 · May 24, 2026 · May 24, 2026
diff --git a/TRACKER.md b/TRACKER.md
@@ -10,7 +10,7 @@
 
 **Last updated:** 2026-05-24
 **Current phase:** Phase 1 — Ingestion + Knowledge Store
-**Next action:** Phase 1 Step 1.3 — Document parsers: PDF, DOCX, PPTX, XLSX, HTML, Markdown, plain text, JSON, CSV, YAML parsers with MIME detection. The Connectors framework (1.2) lands the discovery/fetch side; 1.3 fills in the parse step that turns raw bytes into structured chunks.
+**Next action:** Phase 1 Step 1.4 — OCR pipeline: Tesseract + PaddleOCR plugins, image region extraction, confidence scoring. (Step 1.3 — Document parsers — is in progress: see PR for `build/phase-1/step-1.3-document-parsers`.)
 
 > **Refactor window (Steps 1.1a–1.1f):** Before resuming the connectors framework (1.2), we insert a six-step refactor that locks in architecture + optimization decisions which are very expensive to retrofit later (PolicyEngine PDP, RequestContext-threaded SPIs, split Retrieval/Index backends, bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split, hot-path discipline). See [docs/adr/ADR-0005…0009] and [docs/architecture/policy-engine.md], [request-context.md], [caching.md], [performance.md].
 
@@ -75,7 +75,7 @@
 | 1.1e | Cache SPI split + perf discipline + async telemetry | ✅ | `build/phase-1/step-1.1e-cache-split-perf-telemetry` | [#51](https://github.com/officialCodeWork/AgentContextOS/pull/51) | `EmbeddingCache` (`rag_core.spi.embedding_cache`, key tenant+model_id+model_version+text_hash, `invalidate_model`), `RetrievalCache` (`rag_core.spi.retrieval_cache`, key tenant+plan_hash+corpus_version, `invalidate_corpus`), `AnswerCache` (`rag_core.spi.answer_cache`, key tenant+plan_hash+corpus_version+policy_version, `invalidate_policy`). All ctx-first; signature linter extended. Noop in-memory impls + 18 conformance tests covering hit/miss/version-partition/invalidate/tenant-isolation. `AsyncTelemetrySink` in `rag-observability` — bounded buffer (default 10_000), non-blocking `submit()`, per-kind drop + exporter-error counters, idempotent start/stop, drain-timeout-cancel; 9 unit tests. Hot-path discipline doc in `performance.md` expanded with concrete hot/cold call-site table. New `docs/reference/rag-observability.md`. `caching.md` SPI shapes updated with ctx-first signatures. `rag-core` 0.6.0 → 0.7.0. |
 | 1.1f | ADRs 0005–0009 + reviewer checklist | ✅ | `build/phase-1/step-1.1f-adrs-reviewer-checklist` | [#52](https://github.com/officialCodeWork/AgentContextOS/pull/52) | ADR-0005 (PolicyEngine PDP), ADR-0006 (two-stage reranker default), ADR-0007 (tiered storage with BlobRef), ADR-0008 (cost-aware planner replacing reactive fallback), ADR-0009 (vector index strategy + quantization) all promoted to Accepted with implementation-path backlinks. Reviewer checklist in `docs/architecture/performance.md` extended with explicit items for plan/budget awareness, IndexHint+dtype handling, two-stage rerank shape, and BlobRef-safe consumers. ADR backlinks added to `rag_core.spi.reranker` and `rag_core.spi.storage` (the two missing module-level pointers). |
 | 1.2 | Connectors framework | ✅ | `build/phase-1/step-1.2-connectors-framework` | [#55](https://github.com/officialCodeWork/AgentContextOS/pull/55) | `Connector` SPI evolved to `(Document, ConnectorState)` async-iterator with resumable watermark (`ConnectorState` frozen model: connector_id, tenant_id, cursor, last_seen_at, extra). Internal `Crawler[Raw]` base class in `rag_backends.connectors._base` (iteration scaffold, error policy, monotonic watermark). Built-in connectors: `FilesystemConnector` (cross-platform pathlib, sha256 hashing, sorted resume cursor), `S3Connector` (aioboto3, ListObjectsV2 continuation token, MinIO-compatible), `GCSConnector` (gcloud-aio-storage, REST nextPageToken, `[gcs]` extra). Policy boundary documented — connectors are upstream of `PolicyEngine`; the ingest pipeline (Step 1.10) is the enforcement point. 9 contract tests (NoopConnector resume + watermark + tenant mismatch), 9 unit tests (FilesystemConnector), MinIO + GCS integration tests skip-if-no-service. `ConnectorState` schema added to `dist/schemas/`. New docs: [reference/connectors.md](docs/reference/connectors.md), [architecture/connectors.md](docs/architecture/connectors.md). |
-| 1.3 | Document parsers | ⏳ | — | — | PDF, DOCX, PPTX, XLSX, HTML, Markdown, plain text, JSON, CSV, YAML parsers; MIME detection |
+| 1.3 | Document parsers | 🚧 | `build/phase-1/step-1.3-document-parsers` | [#57](https://github.com/officialCodeWork/AgentContextOS/pull/57) | New `rag-parsers` package (`packages/parsers/`, v0.1.0) shipping `PlainTextParser`, `MarkdownParser` (markdown-it-py), `HtmlParser` (BeautifulSoup + stdlib `html.parser`), `JsonParser`, `CsvParser`, `YamlParser` (PyYAML `safe_load`), `PdfParser` (pypdf), `DocxParser` (python-docx), `PptxParser` (python-pptx), `XlsxParser` (openpyxl). `Parser` SPI evolved: `parse(...)` now returns `ParsedDocument` (`Document` + extracted text + structural `Block` list with offsets + heading levels + parser-specific `attributes`). New types in `rag_core`: `Block`, `BlockType` (8-value enum), `ParsedDocument`. `rag_parsers.mime.detect_mime()` (puremagic + OOXML ZIP central-directory disambiguation + extension shortcut + hint fallback). `ParserRegistry` + `default_registry()` (first-match-wins; specific MIMEs before generics). `ragctl parse <path>` smoke-test command (MIME detect → parser select → print summary + first-N blocks). 35 parser unit tests + 6 ragctl tests + extended contract tests (148 contract tests total pass). Schemas regenerated (`Block.json`, `ParsedDocument.json`). `rag-core` 0.7.0 → 0.8.0. Cross-platform: every parser library is pure-Python or wheel-distributed; no `libmagic` / `lxml` system deps. New docs: [reference/parsers.md](docs/reference/parsers.md), [architecture/parsers.md](docs/architecture/parsers.md). |
 | 1.4 | OCR pipeline | ⏳ | — | — | Tesseract + PaddleOCR plugins; image region extraction; confidence scoring |
 | 1.5 | Structure-aware chunker | ⏳ | — | — | Heading-based chunking, parent-child `Chunk.parent_id` links, sentence-boundary, overlap, size normalization |
 | 1.6 | Metadata enricher | ⏳ | — | — | Auto-tag: language, doc type, created/modified dates, author, section path, reading level |

diff --git a/dist/schemas/Block.json b/dist/schemas/Block.json
@@ -0,0 +1,57 @@
+{
+  "$defs": {
+    "BlockType": {
+      "description": "Coarse structural type of a parsed block.\n\nEmitted by `Parser` implementations (Step 1.3) so the structure-aware\nchunker (Step 1.5) can find natural boundaries without re-parsing.  The\ntaxonomy is deliberately small \u2014 parsers map library-native node types\nonto one of these labels.",
+      "enum": [
+        "heading",
+        "paragraph",
+        "list_item",
+        "table_row",
+        "code",
+        "quote",
+        "caption",
+        "other"
+      ],
+      "title": "BlockType",
+      "type": "string"
+    }
+  },
+  "description": "One structural unit produced by a Parser (Step 1.3).\n\nBlocks preserve enough document structure for the heading-aware chunker\n(Step 1.5) to find natural boundaries without re-parsing the source\nbytes.  Each block carries the extracted text, its 0-based start offset\ninto ``ParsedDocument.text`` (the end is ``start + len(text)``), an\noptional 1-6 heading level (only for ``BlockType.heading``), and a\nfree-form attributes dict for parser-specific extras such as\n``{\"page\": 3}`` for PDFs or ``{\"sheet\": \"Q1\", \"row\": 5}`` for XLSX.",
+  "properties": {
+    "type": {
+      "$ref": "#/$defs/BlockType"
+    },
+    "text": {
+      "title": "Text",
+      "type": "string"
+    },
+    "start": {
+      "title": "Start",
+      "type": "integer"
+    },
+    "level": {
+      "anyOf": [
+        {
+          "type": "integer"
+        },
+        {
+          "type": "null"
+        }
+      ],
+      "default": null,
+      "title": "Level"
+    },
+    "attributes": {
+      "additionalProperties": true,
+      "title": "Attributes",
+      "type": "object"
+    }
+  },
+  "required": [
+    "type",
+    "text",
+    "start"
+  ],
+  "title": "Block",
+  "type": "object"
+}
diff --git a/dist/schemas/ParsedDocument.json b/dist/schemas/ParsedDocument.json
@@ -0,0 +1,220 @@
+{
+  "$defs": {
+    "ACL": {
+      "description": "Single access-control entry: who can do what on which resource.",
+      "properties": {
+        "principal_id": {
+          "title": "Principal Id",
+          "type": "string"
+        },
+        "action": {
+          "$ref": "#/$defs/ACLAction"
+        },
+        "resource": {
+          "title": "Resource",
+          "type": "string"
+        }
+      },
+      "required": [
+        "principal_id",
+        "action",
+        "resource"
+      ],
+      "title": "ACL",
+      "type": "object"
+    },
+    "ACLAction": {
+      "enum": [
+        "read",
+        "write",
+        "delete",
+        "admin"
+      ],
+      "title": "ACLAction",
+      "type": "string"
+    },
+    "Block": {
+      "description": "One structural unit produced by a Parser (Step 1.3).\n\nBlocks preserve enough document structure for the heading-aware chunker\n(Step 1.5) to find natural boundaries without re-parsing the source\nbytes.  Each block carries the extracted text, its 0-based start offset\ninto ``ParsedDocument.text`` (the end is ``start + len(text)``), an\noptional 1-6 heading level (only for ``BlockType.heading``), and a\nfree-form attributes dict for parser-specific extras such as\n``{\"page\": 3}`` for PDFs or ``{\"sheet\": \"Q1\", \"row\": 5}`` for XLSX.",
+      "properties": {
+        "type": {
+          "$ref": "#/$defs/BlockType"
+        },
+        "text": {
+          "title": "Text",
+          "type": "string"
+        },
+        "start": {
+          "title": "Start",
+          "type": "integer"
+        },
+        "level": {
+          "anyOf": [
+            {
+              "type": "integer"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Level"
+        },
+        "attributes": {
+          "additionalProperties": true,
+          "title": "Attributes",
+          "type": "object"
+        }
+      },
+      "required": [
+        "type",
+        "text",
+        "start"
+      ],
+      "title": "Block",
+      "type": "object"
+    },
+    "BlockType": {
+      "description": "Coarse structural type of a parsed block.\n\nEmitted by `Parser` implementations (Step 1.3) so the structure-aware\nchunker (Step 1.5) can find natural boundaries without re-parsing.  The\ntaxonomy is deliberately small \u2014 parsers map library-native node types\nonto one of these labels.",
+      "enum": [
+        "heading",
+        "paragraph",
+        "list_item",
+        "table_row",
+        "code",
+        "quote",
+        "caption",
+        "other"
+      ],
+      "title": "BlockType",
+      "type": "string"
+    },
+    "Document": {
+      "description": "A single ingested source document, before chunking.",
+      "properties": {
+        "id": {
+          "title": "Id",
+          "type": "string"
+        },
+        "tenant_id": {
+          "title": "Tenant Id",
+          "type": "string"
+        },
+        "corpus_id": {
+          "title": "Corpus Id",
+          "type": "string"
+        },
+        "source_uri": {
+          "title": "Source Uri",
+          "type": "string"
+        },
+        "content_hash": {
+          "title": "Content Hash",
+          "type": "string"
+        },
+        "title": {
+          "anyOf": [
+            {
+              "type": "string"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Title"
+        },
+        "mime_type": {
+          "anyOf": [
+            {
+              "type": "string"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Mime Type"
+        },
+        "status": {
+          "$ref": "#/$defs/DocumentStatus",
+          "default": "pending"
+        },
+        "acls": {
+          "items": {
+            "$ref": "#/$defs/ACL"
+          },
+          "title": "Acls",
+          "type": "array"
+        },
+        "metadata": {
+          "additionalProperties": true,
+          "title": "Metadata",
+          "type": "object"
+        },
+        "created_at": {
+          "format": "date-time",
+          "title": "Created At",
+          "type": "string"
+        },
+        "updated_at": {
+          "format": "date-time",
+          "title": "Updated At",
+          "type": "string"
+        }
+      },
+      "required": [
+        "tenant_id",
+        "corpus_id",
+        "source_uri",
+        "content_hash"
+      ],
+      "title": "Document",
+      "type": "object"
+    },
+    "DocumentStatus": {
+      "enum": [
+        "pending",
+        "processing",
+        "ready",
+        "failed"
+      ],
+      "title": "DocumentStatus",
+      "type": "string"
+    }
+  },
+  "description": "A ``Document`` plus its extracted text and structural blocks.\n\nReturned by ``Parser.parse``.  The chunker (Step 1.5) consumes\n``blocks`` for structure-aware boundary detection; parsers that cannot\nrecover structure (plain text, opaque blobs) emit a single\n``BlockType.paragraph`` block spanning the full text.\n\n``detected_mime`` carries the parser's normalized MIME type, which may\ndiffer from the caller-supplied hint when sniffing reveals a more\nspecific content type (e.g. a ``.txt`` file that is actually JSON).",
+  "properties": {
+    "document": {
+      "$ref": "#/$defs/Document"
+    },
+    "text": {
+      "title": "Text",
+      "type": "string"
+    },
+    "blocks": {
+      "items": {
+        "$ref": "#/$defs/Block"
+      },
+      "title": "Blocks",
+      "type": "array"
+    },
+    "detected_mime": {
+      "anyOf": [
+        {
+          "type": "string"
+        },
+        {
+          "type": "null"
+        }
+      ],
+      "default": null,
+      "title": "Detected Mime"
+    }
+  },
+  "required": [
+    "document",
+    "text"
+  ],
+  "title": "ParsedDocument",
+  "type": "object"
+}
diff --git a/docs/README.md b/docs/README.md
@@ -15,6 +15,7 @@
 | [iac.md](architecture/iac.md) | IaC overview: Terraform module design, Helm chart structure, dev/prod environments, extension points |
 | [storage-backends.md](architecture/storage-backends.md) | Storage backend architecture: PgVector, Qdrant, Redis, S3/MinIO, tenant isolation, integration test strategy |
 | [connectors.md](architecture/connectors.md) | Connectors framework: SPI shape, `ConnectorState` watermark, `Crawler` base class, policy boundary, CDC interaction |
+| [parsers.md](architecture/parsers.md) | Document parsers: `Block` taxonomy, `ParsedDocument` return shape, MIME detection, policy boundary, library choices |
 
 ## reference/
 
@@ -26,6 +27,7 @@
 | [rag-observability.md](reference/rag-observability.md) | `rag-observability` — structured logger, event registry, `AsyncTelemetrySink` (bounded, drop-on-overflow) |
 | [rag-policy.md](reference/rag-policy.md) | `rag-policy` reference — `PolicyEngine`, `PolicyWriter`, `PolicyDecision`, `PolicyResult`, `FilterExpr` |
 | [connectors.md](reference/connectors.md) | `rag-backends` connectors — `FilesystemConnector`, `S3Connector`, `GCSConnector`, resumable crawls via `ConnectorState` |
+| [parsers.md](reference/parsers.md) | `rag-parsers` reference — `Parser` SPI return shape, `default_registry()`, `detect_mime()`, format coverage, `ragctl parse` |
 
 ## guides/