Skip to content

8669: Clean up removed sites and purge old detection results - #167

Merged
turegjorup merged 6 commits into
developfrom
fix/8669-dir-cleanup-and-purge
Oct 21, 2026
Merged

turegjorup merged 6 commits into
developfrom
fix/8669-dir-cleanup-and-purge

Conversation

@turegjorup

Copy link
Copy Markdown
Contributor

Link to ticket

https://leantime.itkdev.dk/#/tickets/showTicket/8669

Description

Make the server's latest dir result decide which installations and sites exist, and purge detection results not seen for 30 days.

Changes:

  • DirectoryHandler ignores dir results older than the server's latest, so a replay cannot apply an old list.
  • Sites belong to the listed dir they are in, or its closest parent dir. Sites and detection results outside every listed dir are deleted. Replaces the LIKE '<dir>%' match, where _ is a wildcard and there was no / boundary.
  • An unchanged dir result is re-applied on every harvest instead of only having its timestamp updated.
  • app:data:purge runs daily at 03:00 with up to 30 minutes jitter, --date="30 days ago". --date takes relative dates.
  • DrupalReleaseSchedule becomes the stateful DefaultSchedule that attribute tasks join. The workers consume scheduler_default instead of scheduler_drupal_releases.
  • --failure-limit=1 dropped from the workers.

Why: stg_ai-screening on dmzwebstgitk04 stayed listed after the dir was removed. Results for removed dirs were never deleted, because the handler compared the new installations with the same collection it had just refilled, and the delete query bound the server entity instead of its id. A docker replay on 2026-10-19 then rebuilt the site, and the dir list was never re-applied because its hash had not changed. The dir replay run on prod today applied each server's older dir lists too, which deleted 27 live sites.

The purge was never scheduled, so results back to 2024 remained. Some sites that stopped reporting are inside a listed parent dir, e.g. /data/www/offline/htdocs under /data/www/offline, and only an age limit removes them. Attribute tasks cannot be stateful on their own, and the worker restarts every 15 minutes, so the jittered purge would have skipped about three days in four.

--failure-limit=1 is no longer needed. The worker resets services after each message, which replaces a closed EntityManager, and failed messages are retried and then moved to the failed transport.

Verified on today's prod dump: replaying dir results, then docker results, restores the 27 sites, removes stg_ai-screening and workspace.itkdev.dk, and deletes only results for dirs that are no longer listed. A 30-day purge then removes 370 results, 4 dead sites and installations and 13 stale domains.

After deploy: wait for one harvest, then run bin/console app:replay:detection-results --type docker to restore the 27 sites. Do not replay on the current release.

Checklist

  • My code is covered by test cases.
  • My code passes our test (all our tests).
  • My code passes our static analysis suite.
  • My code passes our continuous integration process.

Ignore dir results older than the server's latest, so replays cannot apply an old list. Attach sites to the listed dir they are in instead of a LIKE prefix, where _ was a wildcard, and delete sites and results outside every listed dir.
An unchanged dir result only had its timestamp updated, so sites recreated after it was first processed, e.g. by a docker replay, were never removed.
Run app:data:purge on the default schedule at 03:00 with up to 30 minutes jitter. --date now takes relative dates such as "30 days ago". The worker consumes scheduler_default.
Attribute tasks such as the purge cannot be stateful on their own, and the worker restarts every 15 minutes, so the jittered purge would skip most days. Turn DrupalReleaseSchedule into the stateful default schedule that attribute tasks join, and consume only scheduler_default.
The worker resets services after each message, which replaces a closed EntityManager, and failed messages are retried and then sent to the failed transport. Stopping on every failure only restarted the worker.
@github-actions

Copy link
Copy Markdown

API Specification - Non-breaking changes

No changelog changes

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.27%. Comparing base (d139191) to head (8d03f65).
⚠️ Report is 13 commits behind head on develop.

Additional details and impacted files
@@              Coverage Diff              @@
##             develop     #167      +/-   ##
=============================================
+ Coverage      66.70%   69.27%   +2.56%     
- Complexity      1295     1300       +5     
=============================================
  Files            168      168              
  Lines           3998     3987      -11     
=============================================
+ Hits            2667     2762      +95     
+ Misses          1331     1225     -106     
Flag Coverage Δ
unittests 69.27% <100.00%> (+2.56%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@turegjorup turegjorup self-assigned this Oct 21, 2026
@turegjorup
turegjorup merged commit c308ed6 into develop Oct 21, 2026
19 checks passed
@turegjorup
turegjorup deleted the fix/8669-dir-cleanup-and-purge branch October 21, 2026 10:52
@turegjorup turegjorup mentioned this pull request Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants