feat(core,parsers): document parsers — PDF/DOCX/PPTX/XLSX/HTML/MD/text/JSON/CSV/YAML (Step 1.3)

[officialCodeWork]

Summary

Step 1.3 — ten built-in document parsers + MIME detection. The new rag-parsers package (packages/parsers/, v0.1.0) plugs into the ingest pipeline between connectors (Step 1.2) and the chunker (Step 1.5).

The Parser SPI is evolved to return ParsedDocument (the embedded Document plus extracted text and a list of structural Block records) so Step 1.5 can recover structure without re-parsing source bytes. This is a load-bearing decision; reviewer attention on Block taxonomy welcome.

Highlights

• Text-family parsers — PlainTextParser, MarkdownParser (markdown-it-py), HtmlParser (BeautifulSoup + stdlib html.parser), JsonParser, CsvParser, YamlParser (PyYAML safe_load).
• Binary parsers — PdfParser (pypdf), DocxParser (python-docx, Heading 1-6 style detection + table rows), PptxParser (python-pptx, slide title as level-2 heading + body + speaker notes), XlsxParser (openpyxl read-only).
• MIME detection — rag_parsers.mime.detect_mime(): extension shortcut → puremagic content sniff → OOXML ZIP central-directory disambiguation → caller-hint fallback. No libmagic system dep.
• Registry — ParserRegistry + default_registry() with first-match-wins ordering (specific MIMEs before fallbacks).
• ragctl parse <path> — smoke-test command: MIME-detect, parse, print summary + first-N blocks. Useful for design-partner demos before Step 1.10's full ingest API.

SPI evolution

New types in rag_core:

• Block — type / text / start (offset into ParsedDocument.text) / level (1-6 for headings) / attributes (e.g. {\"page\": 3}, {\"sheet\": \"Q1\", \"row\": 5}).
• BlockType — 8-value enum: heading, paragraph, list_item, table_row, code, quote, caption, other.
• ParsedDocument — document + text + blocks + detected_mime.

Parser.parse(...) return type changes from Document to ParsedDocument. NoopParser migrated. Contract tests extended for the new shape, empty-input handling, and block-offset invariants. JSON schemas regenerated.

rag-core 0.7.0 → 0.8.0.

Cross-platform

Per the standing Windows/macOS/Linux-without-WSL constraint, every parser library is pure-Python or wheel-distributed: puremagic, markdown-it-py, beautifulsoup4 + stdlib html.parser, PyYAML, pypdf, python-docx, python-pptx, openpyxl. No libmagic / lxml / system OCR dependencies in this PR (OCR lands in Step 1.4).

Tests

• 35 unit tests under tests/parsers/ (MIME detection, text-family, binary parsers via in-memory fixture generation, registry resolution).
• 6 new ragctl tests for the parse command (typer CliRunner).
• Contract tests extended; 148 contract tests total pass.
• Full task lint clean: ruff, ruff format, mypy --strict, RAG001 logging gate.

Policy boundary

Like connectors, parsers are upstream of PolicyEngine. The PDP can't enforce ACL/PII on bytes that haven't become Chunks yet — enforcement stays in the Step 1.10 pipeline (per-document quota check before parse; per-chunk ACL + PII after chunk). The parsers package depends only on rag-core, not on rag-policy.

Documentation

• docs/reference/parsers.md — public + technical reference (Parser SPI, supported MIMEs, usage, error handling, internals, extension points).
• docs/architecture/parsers.md — block-taxonomy rationale, MIME resolution strategy, policy boundary, hot-path discipline, related ADRs.
• docs/README.md updated with both new entries.

Known pre-existing issue

`uv run ragctl parse ` currently fails to import workspace packages outside pytest, because uv-generated editable-install .pth files in `.venv/lib/python3.12/site-packages/editable_impl_rag.pth` lack trailing newlines, so Python's `site.py` silently skips them. Same failure on `main`: `uv run ragctl version` reproduces it. Tests work because pytest's `pythonpath` config adds each `packages//src` explicitly. This isn't introduced by Step 1.3 — flagged as a separate spawned task to investigate.

Tracker

TRACKER.md Step 1.3 flipped ⏳ → 🚧; Next action advanced to Step 1.4 (OCR). The Phase 1 milestone counts will be reconciled when #56 (the chore tracker-sync for Step 1.2) merges and this PR rebases / merges on top.

Test plan

• `uv run pytest tests/ packages/ -x -q` — full suite passes
• `uv run pytest tests/contract/test_parser.py -x -q` — extended contract tests pass
• `uv run pytest tests/parsers/ -x -q` — 35 parser tests pass
• `uv run pytest packages/ragctl/tests/ -x -q` — 24 ragctl tests pass (incl. new `parse`)
• `uv run ruff check .` — clean
• `uv run ruff format --check .` — clean
• `uv run mypy packages/ apps/gateway/` — Success: no issues found in 113 source files
• `uv run python scripts/check_logging.py` — RAG001 gate clean
• `PYTHONPATH=packages/core/src uv run python -m rag_core.gen_schemas dist/schemas/` — `Block.json` + `ParsedDocument.json` regenerated, no other drift
• CI matrix will exercise ubuntu-22.04 + macos-14 + windows-latest

🤖 Generated with Claude Code