[lia-by-librechat]

Summary

• Require service PDF/DOCX extraction for /text, /embed, /embed-upload and /local/embed. No enable/format opt-out or local migrated-parser fallback. Caller policy, identity, zero-based PDF pages, chunking and storage remain RAG-owned.
• Replace the Unix socket with bounded authenticated HTTP. Both capabilities and extraction require a service-specific token. Preserve strict framing, JSON/UTF-8/hash/result limits, admission, cancellation and the single 10-second budget. Abort and drain the actual transport, including TLS.
• Service-auth failures become safe 503 UNAVAILABLE, never end-user authentication failures. No reflected remote messages, URLs or credentials; no proxy, redirect, retry or insecure-TLS fallback.

Default Deployment

Both normal Compose recipes share a processor build pinned to Code Interpreter c7bfe421243c66d4d5d9c6e3d08d276586ac47bf. Nonroot extraction-init uses that build to atomically provision a persistent 64-lowercase-hex token: directory 0755, file 0444, no newline. Its healthcheck is disabled. The worker and callers mount credentials read-only; only init has a writable mount.

The worker inherits the shipped native gateway ENTRYPOINT and gateway-protected authenticated healthcheck. It uses a private internal: true network, no host port or singleton container name, default Docker seccomp, nonroot/read-only execution, dropped capabilities and fixed CPU/memory/PID/tmpfs bounds. No app credentials, uploads, database configuration or storage mounts are supplied. Callers retain their default network plus the private extraction network. Atomic !override covers both shared services and both recipe merge orders. Replicas share credentials without singleton socket ownership.

Configuration:

• RAG_EXTRACTION_URL: defaults to http://extraction:8000. Remote endpoints require verified HTTPS using standard trust. Plain HTTP is restricted to that Compose endpoint and loopback. Origin only: no userinfo, query, fragment or non-root path.
• RAG_EXTRACTION_TOKEN_FILE: defaults to /run/extraction/token; bounded, nonblocking regular-file read, exactly 64 lowercase hex bytes. Read once per job, so client rotation takes effect on the next job; coordinate service-side rotation.
• RAG_EXTRACTION_DEADLINE_SECONDS: positive, at most 10. Obsolete RAG_EXTRACTION_SOCKET, RAG_EXTRACTION_ENABLED and RAG_EXTRACTION_FORMATS are rejected.

Boundaries

• Shared-host containers still share a kernel. These controls are not a kernel-escape guarantee. The worker can instead run without app credentials on a separate VM behind verified HTTPS; no alternate runtime or VM is required by default.
• Coordinate authenticated caller images and worker rollout. Legacy Unix-socket callers are incompatible; do not update the worker alone. This PR does not merge, roll out or deploy anything.
• PDF remains text-only without OCR; PDF_EXTRACT_IMAGES=true rejects PDFs. DOCX supports body content, not full Docx2txt parity. Other formats remain unmigrated. Plaintext/raw-text policy is unchanged.
• Before storage, disconnect drains preparation and prevents mutation. Batched AsyncPg drains inserts before current-attempt rollback; older rows survive. Started single-shot/async or synchronous writes may remain committed, and sync partial failures may retain rows. Known-disconnected callers receive no success. No unsafe file-wide undo.

Verification

• 505 focused tests passed; 6 existing Pandoc-dependent cases skipped. Includes real TLS trust/hostname verification, credential failures, mandatory routes, page/policy/auth/storage behavior, HTTP/TLS upload backpressure, deadlines and repeated cancellation.
• Real current Python client/default-Compose worker qualification passed against the final pinned source: automatic nonroot provisioning, permissions/read-only access, safe missing/wrong credentials, PDF/DOCX, upload cancellation, two replicas and restart. Checks the shipped gateway, inherited healthcheck and actual AF_UNIX/socketpair denial, not a test-only policy.
• Publication-time exact-head reruns passed: 505 focused tests, 6 skips, and 1 real hardened-worker integration. Both recipes and both merge orders also passed on Compose 2.38.2 (5 checks, official checksum-verified binary) and 5.6.0. Black, compilation of all 13 Python files in the PR diff, pip check and full base-to-head whitespace checks passed. Dependency files unchanged. Exact-head CI build-and-test passed, including unit and integration steps: https://github.com/LibreChat-AI/rag-api/actions/runs/39962758266. Parent owns consumer review; no prior-head review is claimed for this head.
• Whole local unit suites, database integration and the full embedding-provider/application stack were not run. Upload cancellation qualification does not claim observation of a running parser being cancelled.

Retained Finding Ledger

Finding	Disposition
R1 Extraction-stage HTTP disconnect	Fixed d155c6807
R2 Binary fixture attributes	Fixed d155c6807; bytes unchanged
R3 Post-extraction request lifetime	Fixed 0c19824b3
R4 P2 Second cancellation interrupting rollback	Fixed c9d5bc8a5
R5 Required-default correction	Fixed 434382d0e
R6 Compose duplicate-security-option merging	Fixed 1eb1ccc17; atomic overrides retained for init and worker

No rejected or outstanding prior findings. Earlier clean reviews do not cover this HTTP head; parent coordinates its exact-head review.

[lia-by-librechat]

Head for review: 9f569dff9a015dfcb51ccd09c8b86863d313333f.
Base/merge-base: c4e5cbf79fcece1df9adfff5dfed983a5990ca9f (main).

Opt-in Unix-socket extraction now serves selected PDF/DOCX through /text, /embed, /embed-upload and /local/embed, without local-parser fallback. Caller policy/storage remain in RAG; service pages map to zero-based metadata. Other formats remain local.

Checks: 254 focused socket/route/regression tests passed; real Python client → hardened processor PDF/DOCX test passed on this head; touched-file Black, compilation and diff checks passed. Local runtime is Python 3.12.3; processor/client qualification uses the shipped Python 3.13 image. CI is running.

Existing dependency blocker: test_requirements.txt pins httpx==0.27.0, incompatible with the runtime Google client. Local tests used resolved httpx==0.28.1; dependency files are unchanged. Whole suites and database integration were not run.

[lia-by-librechat]

Head for fresh review: d155c680776384ab401aa4c23e644c5729b81da0.
Base/merge-base: c4e5cbf79fcece1df9adfff5dfed983a5990ca9f (main, rechecked remote).

Follow-up combines the disconnect audit and binary-fixture fix:

• Before: all eight PDF/DOCX route cases left extraction open after http.disconnect. A Python breakpoint showed the disconnect queued while the live route still awaited the processor response.
• After: a watcher exists only for selected-format extraction, after FastAPI body parsing. Disconnect cancels and drains the processor request before persistence; uploads are cleaned and local source files retained. Eight disconnect tests plus eight disconnect/success-race tests pass without cancelling the ASGI application task. Auth/logging middleware are exercised.
• Scoped fixture .gitattributes marks PDF/DOCX binary. Both fixture hashes are unchanged; full base-to-head git diff --check passes.

Verification: 270 focused pytest tests passed; real Python client → hardened processor PDF/DOCX qualification passed; touched-file Black, Python compilation, pip check and diff checks passed. Whole suites and database integration were not run. Dependency pins, default enablement and page/raw-text semantics are unchanged.

[lia-by-librechat]

Head: d155c680776384ab401aa4c23e644c5729b81da0

Independent exact-head review complete: no findings. Reviewer read the full diff and checked protocol, route/identity/page mapping, disconnect handling, rollback and cleanup in source. It did not rerun application/native-container tests.

Local verification: 270 focused tests, including 8 realistic ASGI disconnect and 8 completion-race cases, plus real Python client → hardened PDF/DOCX processor qualification passed. Black, compilation, pip check and full base-to-head diff checks passed. CI build-and-test passed.

Parent-audit findings fixed in this head: actual disconnect cancellation and binary fixture attributes. No rejected or outstanding findings. Whole suites/database integration were not run locally; processor qualification remains opt-in. Extraction is disabled by default. No service enabled or deployment performed.

[lia-by-librechat]

Head for fresh independent review: 0c19824b3f58c6ddd68afd445584be1c1482874c.
Base/merge-base: c4e5cbf79fcece1df9adfff5dfed983a5990ca9f (main, rechecked remote).

The two-file follow-up extends selected PDF/DOCX request ownership through caller preparation/storage/cleanup. Before: six real ASGI post-extraction disconnect cases across all three embedding routes still committed both new PDF chunks and returned success. After: preparation drains without mutation; batched AsyncPg drains started inserts then rolls back only its attempt, preserving older rows. Cleanup drains before return.

Commit boundary is explicit: single-shot async or already-started sync storage finishes/drains and may retain committed data. Known-disconnected callers receive no success. Sync partial failure can retain committed rows; no unsafe file-wide undo. Disabled/unselected behavior is unchanged.

Checks: 311 focused pytest tests passed on the unchanged patch; final Black, compilation, pip check, and base-to-head diff checks passed. Transport/page/raw-text semantics are unchanged. Hardened processor qualification is reused from d155c6807, not rerun on this head. Whole suites/database integration not run. The earlier head's clean review does not cover this head; parent coordinates fresh review.

[lia-by-librechat]

Head for fresh independent review: c9d5bc8a57b184fb591f2bdef620f385a1ecd589.
Base/merge-base: c4e5cbf79fcece1df9adfff5dfed983a5990ca9f (main, rechecked remote).

R4 (P2) fixed. Before: six real ASGI/frozen-wrapper/busy-executor cases returned 499 after cancelling queued attempt rollback, retaining the new rows and removing the upload. A live Python breakpoint confirmed the operation was already awaiting rollback when the wrapper cancelled it.

After: selected cleanup becomes monotonic before producer shutdown, in-flight insert drain and rollback. Task-owned cleanup and worker/join/unlink drains defer repeated cancellation until completion. Added 22 controlled cases cover completed-batch/queued-rollback races, pending inserts, preparation and unlink; older rows survive and no caller return/upload removal occurs before required drain. Conservative single-shot/sync commit semantics and disabled/unselected behavior remain unchanged.

Checks: 333 focused tests passed; final Black, compilation, pip check and full base-to-head diff checks passed. Transport unchanged; hardened PDF/DOCX processor qualification reused from d155c6807, not rerun. Whole suites/database integration not run. CI is running.

Finding ledger:

• R1 extraction disconnect: fixed in d155c6807.
• R2 binary attributes: fixed in d155c6807.
• R3 post-extraction lifetime: fixed in 0c19824b3.
• R4 second-cancellation rollback race (P2): fixed in this head.
  No rejected findings. The 0c19824b3 review reported R4; it was not clean and does not cover this head. Parent coordinates fresh review.

[lia-by-librechat]

Head: c9d5bc8a57b184fb591f2bdef620f385a1ecd589

Independent exact-head review complete: no findings. Reviewer read the full diff and rechecked R1–R4. Isolated final-batch/queued-rollback and repeated-cancellation drain reproductions passed. It did not run pytest, real ASGI/AnyIO, the processor container or database integration.

CI build-and-test passes. Local verification: 333 focused tests, Black, compilation, pip check and full base-to-head diff check passed. Parent independently reran all 22 new cancellation-race regressions on this head: passed. Hardened PDF/DOCX interoperability qualification is reused from d155c6807; transport is unchanged, not rerun on this head. Whole local suites/database integration were not run.

Finding	Disposition
R1 Extraction-stage HTTP disconnect	Fixed d155c6807
R2 Binary fixture attributes	Fixed d155c6807; bytes unchanged
R3 Post-extraction request lifetime	Fixed 0c19824b3
R4 P2 Second cancellation interrupting rollback	Fixed c9d5bc8a5

No rejected or outstanding findings. Selected pre-storage cancellation prevents mutations; batched AsyncPg drains inserts before current-attempt rollback. Started single-shot/sync writes may remain committed, but known-disconnected callers receive no success. No unsafe file-wide undo.

Disabled by default. No merge, enablement or deployment performed.

[lia-by-librechat]

Maintainer direction changed: migrated PDF/DOCX parsing must always use the isolated extraction service by default. Revision in progress: remove the enablement/format-subset opt-outs that leave local migrated parsers active; add the hardened processor and read-only caller socket wiring to standard Compose recipes.

The current head and its clean review/CI cover the prior opt-in design. The new default will receive focused verification and a fresh exact-head review. Other formats remain a separate migration; no whole-parser-isolation claim or deployment performed.

[lia-by-librechat]

Head for fresh independent review: 434382d0e60825af7f5d661c25f543251b7d1435.
Base/merge-base: c4e5cbf79fcece1df9adfff5dfed983a5990ca9f (main, rechecked remote).

PDF/DOCX processing is now required, not opt-in. Enable/format-subset settings and the local PDF implementation are removed. Obsolete environment keys are rejected. All four routes use the service and full request lifecycle; absent socket fails closed without affecting plaintext. Other formats remain explicitly unmigrated.

Both normal RAG Compose recipes include one shared hardened processor definition pinned to Code Interpreter 77bcd2ec2e63be523bfab2dfa11159a2ceec0088, health dependency, read-only /socket caller mount and group 10001. Processor isolation/resource bounds are unchanged, with no app credentials/storage passed. RAG itself does not spawn a sidecar, so the LibreChat stack can share one processor.

Checks: 384 focused tests passed; 6 existing Pandoc-dependent cases skipped. Fresh current-head real client/default-Compose processor PDF/DOCX qualification passed, including engine security limits and nonroot/read-only socket access. Black, compilation, pip check, and full base-to-head diff checks passed. Whole suites/database integration/full embedding-provider stack not run. CI is running.

R1–R4 regression protections and PDF page/raw-text semantics are preserved. Conservative single-shot/started-sync writes may remain committed after disconnect; sync partial failures can retain rows rather than delete other attempts. Prior review/CI covered the old opt-in head, not this head. Parent coordinates fresh review.

[lia-by-librechat]

Head for fresh review: 1eb1ccc17cd3ec5bd36415fbfe84812e9911c682.
Base/merge-base: c4e5cbf79fcece1df9adfff5dfed983a5990ca9f (main, rechecked remote).

Fixes the known CI failure on 434382d0e: CI's Compose 2.38.2 expands the shared extends service twice and rejects duplicated security_opt entries. Reproduced in a disposable, network-disabled Compose 2.38.2 container. Atomic processor-service !override fixes both merge orders without changing any hardening value. Each recipe and both merged orders preserve one processor/socket and singular security/tmpfs/mount lists on 2.38.2 and local 5.6.0.

Render failures now expose sanitized stderr rather than opaque CalledProcessError or captured model stdout. Added reverse-order and diagnostic regressions. Audited legacy upload tests: valid requests are plaintext; PDF filenames elsewhere exercise path validation. Required PDF/DOCX service behavior and R1–R4 protections are unchanged.

Checks: 386 focused tests passed; 6 existing Pandoc-dependent cases skipped. Current-head actual default-Compose processor/client PDF/DOCX qualification passed, including engine security/resource settings and read-only socket access. Black, compilation, pip check and full diff checks passed. Whole unit suites, database integration and full embedding-provider startup were not run locally. Prior 434382d0e CI failed and its review was interrupted; neither is claimed clean. Parent coordinates fresh exact-head review.

[lia-by-librechat]

Head: 1eb1ccc17cd3ec5bd36415fbfe84812e9911c682

Independent exact-head review complete: no findings. Reviewer read the full required-default diff, R1–R6, affected callers and pinned processor contract. Both normal recipes and both merge orders rendered correctly on Compose 5.6.0. Reviewer did not rerun Python runtime tests, Compose 2.38.2, the processor container or database integration.

CI build-and-test passes. Local verification: 386 focused tests passed, 6 Pandoc-dependent cases skipped. Both Compose 2.38.2 and 5.6.0 passed individual/combined render checks; current-head default-Compose processor and Python-client qualification passed. Black, compilation, pip check and full diff checks passed. Parent separately ran 60 focused mandatory-default/blocked-loader/cancellation regressions: passed. Whole local suites, database integration and the full database/embedding stack were not run.

PDF/DOCX isolation is REQUIRED with omitted configuration. There is no enablement or format-subset opt-out, local PDF implementation, or synchronous migrated-loader bypass. Missing processor returns 503 UNAVAILABLE without local parsing/storage writes. Both normal Compose recipes supply one credential-free hardened companion and read-only caller socket mounts. Other formats remain unmigrated.

Prior findings R1–R4 remain fixed; requirement correction R5 removes the opt-in design; R6 fixes Compose 2.38.2 duplicate-security-option merging with atomic processor override. None rejected or outstanding. Earlier opt-in reviews do not cover this design; interrupted 434382d review is not counted as clean.

Conservative started single-shot/sync storage may remain committed after disconnect, but known-disconnected callers receive no success. No unsafe file-wide undo. No merge or deployment performed.

[lia-by-librechat]

Head for fresh independent review: 812264158c4c709670ffb6340926b1ed63215130.
Base/merge-base: c4e5cbf79fcece1df9adfff5dfed983a5990ca9f (main).
Final processor source: c7bfe421243c66d4d5d9c6e3d08d276586ac47bf (service PR #359).

Required PDF/DOCX extraction now uses service-token-authenticated bounded HTTP, with verified HTTPS required remotely. Service authentication remains separate from caller authentication: failures return safe 503 UNAVAILABLE. No local parser, proxy, redirect, retry or insecure-TLS fallback. Both Compose recipes provision credentials automatically as UID 10001, inherit the native gateway/healthcheck, use the private network and support replicas.

Local checks before publication: 505 focused tests passed, 6 existing Pandoc cases skipped; final-pinned real default-worker/client qualification passed provisioning/auth/PDF/DOCX/upload cancellation/two replicas/restart and the shipped gateway. Black, compilation, pip check and full diff checks passed. Publication-time exact-head reruns passed: 505 focused tests, 6 skips, and 1 real worker integration. All 5 Compose contract checks also passed with checksum-verified Compose 2.38.2; the standard run used 5.6.0. Qualification containers, volumes, networks and the temporary compatibility tool were removed. Exact-head CI build-and-test passed, including unit and integration steps: https://github.com/LibreChat-AI/rag-api/actions/runs/39962758266. Whole local suites/database integration/full provider stack were not run.

R1–R6 remain fixed; their ledger is retained in the PR body. Existing conservative started-write/partial-sync limitations remain. Shared-host kernel risk remains; a separate credential-free VM with verified TLS is supported. Authenticated caller images and worker must be coordinated, not rolled out independently. No merge or deployment performed. Parent owns this exact-head consumer review; prior-head reviews are not claimed to cover it.

[lia-by-librechat]

Head 812264158c4c709670ffb6340926b1ed63215130: independent review complete, no findings. Reviewer traced the complete diff, R1–R6, real routes/storage/rollback, and pinned processor contract; all four Compose 5.6.0 renders, changed-Python parsing and whitespace checks passed. Runtime Python/TLS/worker/database tests were not repeated by the reviewer. Implementation checks: 505 focused tests passed, 6 existing Pandoc cases skipped; final-pinned default-worker/client provisioning/auth/PDF/DOCX/cancellation/two-replica/restart qualification passed. Compose 2.38.2 and 5.6.0 compatibility, Black, compilation and pip check passed. Exact-head CI build-and-test passed, including unit/integration steps. R1–R6 remain fixed; none rejected or outstanding. Whole local suites and full provider/application stack were not run. Shared-kernel risk and coordinated authenticated caller/worker rollout remain explicit. No merge or deployment.

[danny-avila]

@codex review the latest head, final review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	🔄 Running since 2026-10-23T13:55:44.056928Z	8122641	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.