mcardle-features-runners - #42
Conversation
* Prevent double-progress-bars from swallowing query executions * Increased logging verbosity for tracking dropped/errored files * Introduce persistent error ledger into db * Wire error ledger into pipeline * Additional comments/concerns for discussion * Separated failure ledger schema and additional logging in crawlers * Check for active subjects and record genuine failures * Config update * Improved failure ledger schema * Introduce batch crawler * Config for data dictionary * Add OH timezone and timezone fail logging * Deletes stale records for audio_journals on insert (avoids collision when journal upload is delayed compared to timestamp) * DB connection optimizations
Introduces audio qc override to backend runners
* Introduce datetime_overrides table to db * Utilize datetime_overrides on file parse failure * Remove override_consumed check for subsequent crawler runs
|
Posting @rtmcardle 's comments here for the record on #41:
|
dheshanm
left a comment
dheshanm
left a comment
There was a problem hiding this comment.
This PR adds multiple fixes:
- Adds the missing timezone to OH
- Much improved error handling and persistence in DB via.
pipeline_ledger
And features:
- AudioQC overrides
These are all great additions to the pipeline. One observation for future consideration: the pipeline is becoming more tightly coupled with its companion repository, dptools/dpinterview-web
2 participants
Additional features developed in July 2026, recording pipeline failures to a ledger, respecting audio qc overrides, and respecting manual date selection (for runsheet linking malnamed files).