Fix/ingester retry deferred submissions - #2121
Open
nuclearcat wants to merge 5 commits into
Open
nuclearcat wants to merge 5 commits into
nuclearcat wants to merge 5 commits into
Conversation
upload_logexcerpt() returned the excerpt itself when the POST to storage failed, and the caller used it as the output_files url. The schema validation right after rejected the submission over that url, so the file was moved to failed/ and its results were never ingested. Return None instead and leave the item untouched, keeping the excerpt inline. Nothing is cached either, so the next submission carrying the same excerpt retries the upload. Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
Storage answers 409 "Upload already in progress" when another worker is uploading the same excerpt, and connection errors happen. A single attempt means the excerpt stays inline in the database for no good reason. Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
A failed flush means the database refused the write (deadlock, lock timeout, a missing grant), which says nothing about the submissions. They were moved to failed/ and never looked at again, so a transient database problem lost results permanently: a missing GRANT on the labs table dropped every lab test result for two days. Move them to pending_retry/ instead, as they were before 034a0ec, which is also what the log message has claimed all along. Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
Nothing ever read pending_retry/, so a deferred submission stayed there forever. Sweep it back into the spool, oldest first, once the spool is drained and at least five minutes since the last sweep, so retries never delay new submissions. A spool that never drains gets a sweep every thirty minutes anyway. Replaces the "TODO: retry failed files every x cycles" from the original monitoring command. Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
Retrying forever keeps the data safe but hides the cause, which is how a missing GRANT went unnoticed for two days. Sample the backlog on every sweep and exit non-zero when it has not shrunk for INGESTER_STALL_GRACE_MINUTES, so the container restart is visible. Any decrease restarts the clock, and the default hour leaves room for several sweeps. The backlog is watched rather than the success count: a partial breakage (one table unwritable) keeps most submissions flowing while a subset is deferred indefinitely. Also export it as kcidb_ingestion_pending_retry for alerting. Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
nuclearcat
force-pushed
the
fix/ingester-retry-deferred-submissions
branch
from
September 17, 2026 12:18
7974da6 to
aa355a4
Compare
| with os.scandir(directory) as it: | ||
| return sum(1 for e in it if e.is_file() and e.name.endswith(".json")) | ||
| except OSError: | ||
| return 0 |
Contributor
There was a problem hiding this comment.
It might be safer to raise an Exception here, otherwise a permission error would constantly return as no files to process. Or at least a warning if this is intended.
| # Requeue deferred submissions once the spool is drained, so | ||
| # they never delay new ones, and eventually regardless in case | ||
| # it never drains. | ||
| spool_drained = cache_pos >= len(cached_files) |
Contributor
There was a problem hiding this comment.
I might have a wrong understanding here, so correct me if wrong.
We are checking only the number of pending files, and if new batches come with their own failure files, we risk never reaching zero (or current floor) for enough time (altough might be unlikely) to trigger the grace time limit.
If that is the case, maybe getting the timestamp from pending files and checking how old are the older pending files.
| RETRY_SWEEP_INTERVAL_SEC = 300 | ||
| RETRY_SWEEP_MAX_INTERVAL_SEC = 1800 | ||
| RETRY_SWEEP_LIMIT = 5000 | ||
| STALL_GRACE_MINUTES = int(os.environ.get("INGESTER_STALL_GRACE_MINUTES", 60)) |
Contributor
There was a problem hiding this comment.
worth adding a try cache for malformed input
Comment on lines
+169
to
+181
| uploaded_url = upload_logexcerpt(log_excerpt, log_hash) | ||
| if uploaded_url is None: | ||
| # Keep the excerpt inline, and nothing cached, so the next | ||
| # submission carrying it retries the upload. | ||
| logger.error( | ||
| "Could not upload log_excerpt for %s %s (hash %s)", | ||
| item_type, | ||
| id, | ||
| log_hash, | ||
| ) | ||
| return | ||
| set_in_cache(log_hash, uploaded_url) | ||
| set_log_excerpt_ofile(item, uploaded_url) |
Member
There was a problem hiding this comment.
If storage is down (or throws any error) and the excerpt is over 16k, the flush fails and the whole buffer goes to pending_retry/, not just the one file.
set_log_excerpt_ofile would clear the log_excerpt and set the url, but it will not be called if upload_logexcerpt returns None.
log_excerpt = models.CharField(max_length=16384, blank=True, null=True)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.