Repository navigation
8669: Clean up removed sites and purge old detection results - #167
Merged
Merged
Conversation
Ignore dir results older than the server's latest, so replays cannot apply an old list. Attach sites to the listed dir they are in instead of a LIKE prefix, where _ was a wildcard, and delete sites and results outside every listed dir.
An unchanged dir result only had its timestamp updated, so sites recreated after it was first processed, e.g. by a docker replay, were never removed.
Run app:data:purge on the default schedule at 03:00 with up to 30 minutes jitter. --date now takes relative dates such as "30 days ago". The worker consumes scheduler_default.
Attribute tasks such as the purge cannot be stateful on their own, and the worker restarts every 15 minutes, so the jittered purge would skip most days. Turn DrupalReleaseSchedule into the stateful default schedule that attribute tasks join, and consume only scheduler_default.
The worker resets services after each message, which replaces a closed EntityManager, and failed messages are retried and then sent to the failed transport. Stopping on every failure only restarted the worker.
API Specification - Non-breaking changesNo changelog changes |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #167 +/- ##
=============================================
+ Coverage 66.70% 69.27% +2.56%
- Complexity 1295 1300 +5
=============================================
Files 168 168
Lines 3998 3987 -11
=============================================
+ Hits 2667 2762 +95
+ Misses 1331 1225 -106
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to ticket
https://leantime.itkdev.dk/#/tickets/showTicket/8669
Description
Make the server's latest dir result decide which installations and sites exist, and purge detection results not seen for 30 days.
Changes:
DirectoryHandlerignores dir results older than the server's latest, so a replay cannot apply an old list.LIKE '<dir>%'match, where_is a wildcard and there was no/boundary.app:data:purgeruns daily at 03:00 with up to 30 minutes jitter,--date="30 days ago".--datetakes relative dates.DrupalReleaseSchedulebecomes the statefulDefaultSchedulethat attribute tasks join. The workers consumescheduler_defaultinstead ofscheduler_drupal_releases.--failure-limit=1dropped from the workers.Why:
stg_ai-screeningon dmzwebstgitk04 stayed listed after the dir was removed. Results for removed dirs were never deleted, because the handler compared the new installations with the same collection it had just refilled, and the delete query bound the server entity instead of its id. A docker replay on 2026-10-19 then rebuilt the site, and the dir list was never re-applied because its hash had not changed. The dir replay run on prod today applied each server's older dir lists too, which deleted 27 live sites.The purge was never scheduled, so results back to 2024 remained. Some sites that stopped reporting are inside a listed parent dir, e.g.
/data/www/offline/htdocsunder/data/www/offline, and only an age limit removes them. Attribute tasks cannot be stateful on their own, and the worker restarts every 15 minutes, so the jittered purge would have skipped about three days in four.--failure-limit=1is no longer needed. The worker resets services after each message, which replaces a closed EntityManager, and failed messages are retried and then moved to the failed transport.Verified on today's prod dump: replaying dir results, then docker results, restores the 27 sites, removes
stg_ai-screeningandworkspace.itkdev.dk, and deletes only results for dirs that are no longer listed. A 30-day purge then removes 370 results, 4 dead sites and installations and 13 stale domains.After deploy: wait for one harvest, then run
bin/console app:replay:detection-results --type dockerto restore the 27 sites. Do not replay on the current release.Checklist