OpenConceptLab/ocl_online#249 | Expansion re-evaluation to clear stale ES tags, and view permission

[snyaggarwal]

Linked Issue

Closes OpenConceptLab/ocl_online#249

Summary

Part of #249. Re-evaluating an expansion now gives the from-scratch result, and ES tags for rows that leave the expansion are removed.

Changes

Re-evaluate is from scratch (Expansion.add_references, core/collections/models.py)

• When re-evaluating all references (force_reevaluate and is_adding_all), the expansion's concepts, mappings and version links are cleared first. The expansion id is kept.
• Non-versioned references re-resolve to the latest released version. The first one for a repo locks it for the rest of the run, so versions of a repo are never mixed.
• On the incremental path, a stored lock is reused only if that version is still released, active and not retired.

ES mirrors the expansion

• After dedupe, only the surviving members are append-indexed.
• Rows dropped by dedupe, or that left the expansion on re-evaluate, are fully re-indexed from the DB through the new Expansion.full_index_resources. This rebuilds all six collection fields, not only expansion_url.
• APPEND_COLLECTION_FIELDS_SCRIPT is unchanged.

Retired versions (core/common/models.py)

• get_latest_released_version skips retired versions.
• is_latest_released returns False instead of raising when no released version exists.

Permission (core/collections/views.py)

• Removed the permission_classes = (CanViewConceptDictionary,) line from CollectionVersionExpansionReEvaluateView. It had no effect, because the base view's get_permissions already requires staff, the owner or an org member for POST.
• Added a regression test that a user with only view access gets 403.

Tests

• Re-evaluate moves to a newer release, drops stale members and locks, and keeps the expansion id.
• Unreleased and retired versions are never picked implicitly.
• Only post-dedupe rows are append-indexed, and the dropped rows go to full_index_resources.
• Re-evaluate returns 403 for a user without edit access.
• Indexing is mocked in the new tests, so none of them need Redis or ES.
• Ran core.collections, core.common and core.sources in the API container: 642 tests passed.

Performance

Re-evaluating the PEPFAR MER_FY22 and MER_FY24 expansions took 52–58s, about the same as without the change. Evaluation dominates. ES writes were mocked, so the ES-side cost wasn't measured. On both, no stored row matches a re-evaluated row, so the old rows are fully re-indexed on top of the new ones. That roughly doubles ES writes for a re-evaluate, and it is what clears the stale tags.

Not in this PR

• Collection release copying HEAD's rows (seed_references, ocl_issues#2549).
• COVID-19-Starter-Set data remediation (ocl_online#267).
• Cleanup of tags already stale in ES. That goes through expansion_tag_fix.py.

Not covered by tests

The second non-versioned reference locking to the first's version, and re-evaluate matching a fresh expansion exactly, are not tested.

[snyaggarwal]

Review result: no blocking issues found.

I reviewed the expansion re-evaluation changes, permission adjustment, and latest-released filtering. The forced re-evaluation path now rebuilds membership in place and reindexes resources that leave the expansion, and the permission behavior is covered by the new non-member test.

Verification run:

• docker exec oclapi2-api-1 python manage.py test core.collections.tests.tests.ExpansionTest core.collections.tests.tests.CollectionViewsAPITest --keepdb -v2 — 65 tests passed.
• docker exec oclapi2-api-1 pylint -j2 core/collections/models.py core/collections/views.py core/common/models.py — 10.00/10.

Note: I attempted to submit this as an approving review, but GitHub does not allow approving your own PR. CI Pylint was still in progress when I checked the PR status.

(snyaggarwal • GPT 5.5)

[snyaggarwal]

CodeX review without any finding - #932 (comment)
PR is green
Manual review done