[OMEGA-326] chroma knowledge indexing fails under Landlock during compaction

[paul-v-snet]

Description

Although I was unable to reproduce the issue itself, both causes described in the issue were confirmed and verified in the code: policy.py did not include AccessFs.REFER (required by Chroma for reparent/rename operations during compaction under Landlock), while rag.py and lib_chromadb.py opened PersistentClient using different literal paths (/PeTTa/chroma_db vs ./chroma_db) for the same directory - ChromaDB 1.5.9 treats these as two different "systems".

Applied the fixes proposed in the issue.

How Has This Been Tested?

• Attempted to reproduce the issue before and after applying the fixes - everything worked successfully in both cases.
• Ran all mock tests - all passed.

Checklist

• The code generated by LLM is reviewed by the PR creator
• Self-review completed
• Test scenarios above are passed with the version of the code from PR

[surafelfikru]

This makes the DB path depend on cwd — before it was resolved relative to this file, so it worked from anywhere. Fine for the container and the README install since both cd to the PeTTa root, but anywhere else the os.makedirs below just creates an empty DB and the agent comes up with no memory, silently. Could we log the abspath when the client opens so that's at least visible?

Also, did you check whether the REFER change on its own fixes it? The reporter tested both together, so I can't tell if this part is doing any work.

[paul-v-snet]

Good point, agree. I'll add the abspath logging 👍

As for REFER - as I mentioned, I was unable to reproduce the issue, so I cannot check this unfortunately.
But yeah, I'm assuming that ChromaDB compaction could resolve that with REFER access only, so the changes related to DB_PATH may not be necessary at all here, and we can just leave it as it was before - I suggest we do it this way.

[surafelfikru]

lib_chromadb hardcodes ./chroma_db and has no env override, so setting CHROMA_DB_PATH to anything else puts the two clients back on separate systems — same bug as #295, since Chroma keys the system off the raw persist_directory string. Should we just drop the override here, or have it error when it isn't ./chroma_db?

[paul-v-snet]

I don't think so, I think it would be better to just leave it as it was before.

[TossSky]

Tested: images v0.1.15, v0.1.18, and v0.1.15 carrying only profile/policy.yaml from v0.1.18. Default Landlock policy, Local embeddings, volume wiped before every run. Two data sets: 33 files (~330 KB) and 300 files (3.6 MB).

What I checked

• The bug from issue #295 reproduces on the reporter's version, v0.1.15: indexing dies partway with chromadb.errors.InternalError: Error in compaction: Failed to apply logs to the metadata segment, twice out of two runs
• The same scenario on v0.1.18 completes: Knowledge: 33 files (0 unchanged, 33 re-indexed)
• 300 files / 3.6 MB also index cleanly on v0.1.18: Knowledge: 300 files (0 unchanged, 300 re-indexed), with compaction demonstrably running (chroma grows to 37 MB, the HNSW segment is rewritten mid-run, the write-ahead log is applied and pruned)
• Reparenting was forbidden throughout that run: cross-directory rename returns EXDEV under the policy and succeeds without it, same directories, same device

The cause is not the missing REFER

Control run: v0.1.15 image, python code untouched, with only profile/policy.yaml replaced by the one from v0.1.18. Indexing completes, and grep -c REFER profile/policy.py inside that image returns 0.

The difference that matters is in the policy file:

   read_only:
-  - /dev/shm
   read_write:
+  - /var/tmp
+  - /dev/shm

Chroma writes to shared memory during compaction, and /dev/shm was read-only. That was fixed in 89f5083, which is in v0.1.16, v0.1.17 and v0.1.18. The reporter was on v0.1.15, the last version without it, which is also why the failure does not reproduce on current code.

Verdict: the change does not address the cause of this issue. The bug is real and was fixed in v0.1.16 by 89f5083. Compaction completes with reparenting forbidden, so the claim that it requires REFER does not hold. Whether to grant the permission for other reasons is a separate call.