Repository navigation
fix(client): report the error message of a failed async task - #776
RaphaelFakhri wants to merge 1 commit into
Conversation
Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
|
@all-contributors add @RaphaelFakhri for bug, code |
|
I've put up a pull request to add @RaphaelFakhri! 🎉 |
:java_duke: JaCoCo coverage report
|
|
||||||||||||||
|
HTML test reports are available as workflow artifacts (zipped HTML). • Download: Artifacts for this run |
edeandrea
left a comment
There was a problem hiding this comment.
Thanks for this, @RaphaelFakhri — a clear issue report and the fix goes in at the right layer. I appreciate that you went looking at what the poll response actually carries rather than just reworking the message text.
Worth recording something I checked while reviewing, because it makes the change look better than the description claims. Upstream TaskStatusResponse actually carries two failure fields, error_message and failure, and I went in half-expecting to ask why you'd picked the thinner one. It turns out to be the opposite:
| Orchestrator | error_message |
failure |
|---|---|---|
local (what DoclingServeContainer runs) |
yes | no |
| RQ | yes | no |
| Ray | yes | yes |
Only the Ray dispatcher populates failure, and it only exists from docling-serve v1.25.0, whereas error_message comes back from every tag in our compatibility table down to v1.18.0. So this picks the one field that works for everybody. I've opened #778 to track the other one separately.
Three requests below, all in tests and docs — the production change itself is good as it stands. Two further comments are just me confirming things are right, so you don't have to wonder whether they're veiled asks.
| @Test | ||
| void failedTaskReportsTheErrorMessageOfTheServer() { | ||
| getWireMock().stubFor( | ||
| post(urlPathEqualTo("/v1/convert/source/async")) | ||
| .willReturn(okJson(taskStatus("pending"))) | ||
| ); | ||
|
|
||
| getWireMock().stubFor( | ||
| get(urlPathEqualTo(POLL_PATH)) | ||
| .willReturn(okJson(""" | ||
| { | ||
| "task_id": "%s", | ||
| "task_status": "failure", | ||
| "task_meta": { | ||
| "num_docs": 1, | ||
| "num_processed": 1, | ||
| "num_succeeded": 0, | ||
| "num_failed": 1 | ||
| }, | ||
| "error_message": "Input document is not valid" | ||
| } | ||
| """.formatted(TASK_ID))) | ||
| ); | ||
|
|
||
| var future = client(null) | ||
| .convertSourceAsync(convertRequest()) | ||
| .toCompletableFuture(); | ||
|
|
||
| assertThatThrownBy(() -> future.get(10, TimeUnit.SECONDS)) | ||
| .hasRootCauseMessage("Async conversion failed for task %s: Task failed: Input document is not valid".formatted(TASK_ID)); | ||
| } |
There was a problem hiding this comment.
Good call adding coverage rather than trusting the change — one request though, on where it lives.
Could you move it into a new @Nested class AsyncTaskFailureTests in AbstractDoclingServeClientTests and drop it from here?
AbstractDoclingServeClientTests is where we exercise what the client does with an actual request/response pair over the wire, which is exactly what this test checks — that a failure poll body carrying error_message comes back out as a usable exception. It's also where the other two fixes in this release ended up: NonJsonResponseTests and UnprocessableEntityResponseTests were both added as focused @Nested classes for precisely this "server sent X, client should report Y" shape, so a third fits the groove.
This class is a narrower thing — it's all about the asyncExecutor builder option, and every other test here leans on this.executions or this.threadNames to prove work landed on the right executor. Yours passes client(null) and needs neither, which is a good sign it wants a different home.
NonJsonResponseTests (line 491) is the closest template: a @Nested class, an @AfterEach calling getWiremockServer().resetAll(), and getDoclingClient(false, true) for a WireMock-backed client. You'll want your own TASK_ID and stub helper in the new class.
Two nearby classes look like candidates but aren't, so you don't lose time on the detour: TaskTests runs against a real container via Testcontainers, and EmptyRequiredStringTests keeps its TASK_ID and taskStatus helpers private and is scoped just as tightly as this one.
| getWireMock().stubFor( | ||
| get(urlPathEqualTo(POLL_PATH)) | ||
| .willReturn(okJson(""" | ||
| { | ||
| "task_id": "%s", | ||
| "task_status": "failure", | ||
| "task_meta": { | ||
| "num_docs": 1, | ||
| "num_processed": 1, | ||
| "num_succeeded": 0, | ||
| "num_failed": 1 | ||
| }, | ||
| "error_message": "Input document is not valid" | ||
| } | ||
| """.formatted(TASK_ID))) | ||
| ); |
There was a problem hiding this comment.
This stub covers the case the issue describes, which is the right one to lead with. Could you add four more variations on it? Together they pin the whole decision tree in the FAILURE branch rather than one path through it.
To be clear on where these go: they belong in the new AsyncTaskFailureTests class from my other comment, next to the relocated test — not here. I'm only commenting on this line because it's the stub they're all variations of.
"error_message": ""→ expectTask failed: TaskStatusMetadata(...). This is the only thing that exercises the.filter(Utils::isNotNullOrBlank)guard onAsyncOperations:134. The server never emits a blank value, so if that guard ever breaks, no other test and no real deployment will catch it.error_messagepresent, notask_meta→ expectTask failed: Input document is not valid.task_metapresent, noerror_message→ expectTask failed: TaskStatusMetadata(...). This is the exact scenario Async task failure hides the error message returned by Docling Serve #775 opens with, and it's untested both before and after the PR —failedTaskRunsEveryPollOnCustomExecutorhas neither field, so it only ever proves the bareTask failedfallback.- Any of the above through
chunkSourceWithHybridChunkerAsync. The description says the fix covers the chunk methods and it does, since they shareAsyncOperations— one assertion turns that from reasoning into coverage, and Async task failure hides the error message returned by Docling Serve #775 names the chunk methods explicitly. Worth keeping it with the other failure cases rather than inChunkTests, since what it's checking is failure reporting, not chunking.
|
|
||
| ### {{ gradle.project_version }} | ||
|
|
||
| * **A failed async task reports the reason given by Docling Serve** — When an async conversion or chunking task failed, the exception only showed the task metadata, which holds document counts, so the cause of the failure was lost. `TaskStatusPollResponse` now has an `errorMessage` field, mapped from the `error_message` that Docling Serve returns, and the exception message includes it when it is present. |
There was a problem hiding this comment.
The entry is accurate, but I think it will generate a bug report in a few weeks. Could you append the last sentence here?
| * **A failed async task reports the reason given by Docling Serve** — When an async conversion or chunking task failed, the exception only showed the task metadata, which holds document counts, so the cause of the failure was lost. `TaskStatusPollResponse` now has an `errorMessage` field, mapped from the `error_message` that Docling Serve returns, and the exception message includes it when it is present. | |
| * **A failed async task reports the reason given by Docling Serve** — When an async conversion or chunking task failed, the exception only showed the task metadata, which holds document counts, so the cause of the failure was lost. `TaskStatusPollResponse` now has an `errorMessage` field, mapped from the `error_message` that Docling Serve returns, and the exception message includes it when it is present. How much detail that message carries depends on the server's error-sanitization setting: by default Docling Serve reports a generic `Internal processing error.` for most infrastructure failures, and the underlying exception text only when it is started with `DOCLING_SERVE_DEBUG_ERROR_DETAILS=true`. |
The reason: build_public_task_error in docling-jobkit returns the real exception text for only four exception types. Everything else — timeouts, OOM, internal errors — comes back as the constant "Internal processing error." unless debug detail is switched on. Someone will hit a timeout, see a message that says nothing, and reasonably assume this feature is broken.
| // Prefer the reason reported by Docling Serve, as the task metadata only holds document counts | ||
| var errorMessage = Optional.ofNullable(statusResponse.getErrorMessage()) | ||
| .filter(Utils::isNotNullOrBlank) | ||
| .or(() -> Optional.ofNullable(statusResponse.getTaskStatusMetadata()).map(Object::toString)) | ||
| .map(reason -> "Task failed: %s".formatted(reason)) | ||
| .orElse("Task failed"); |
There was a problem hiding this comment.
Flagging this so you don't wonder whether I skipped it: it's correct as written, including dropping the TaskStatusMetadata dump when there's a reason to show instead.
I'd actually drafted a comment asking you to keep the counts, on the theory that a failed batch would be reporting something like "3 of 10 documents failed." It isn't. FAILURE only ever gets set from an except Exception handler — per-document failures go into the result's errors list and the task still completes SUCCESS. So those counts are whatever partial progress had been recorded when the task died, not a failure tally. Swapping them for the actual reason is a straight improvement, and the dump was only ever there because nothing better was available.
| /** | ||
| * Error message reported by Docling Serve when the task failed. | ||
| * | ||
| * @param errorMessage the error message, or null if not available | ||
| * @return the error message, or null if not available | ||
| */ | ||
| @JsonProperty("error_message") | ||
| @Nullable | ||
| private String errorMessage; |
There was a problem hiding this comment.
Field shape is right, and I want to say so explicitly because the alternative is tempting. Nullable String plus @Nullable matches the four fields already on this type; returning Optional would follow our general "Optional over null" guidance but would make this one field behave unlike its neighbors inside a Lombok wire-format DTO. Consistency is the better call here.
And on what's deliberately not here, so it doesn't look like an oversight: upstream also carries failure: PublicFailureInfo (category / message / retryable / phase). Leaving it out is correct for this PR — it's Ray-only and v1.25.0+, while error_message works everywhere we support. I've opened #778 for it, which also captures an unknown-enum-value policy question we need to settle before adding FailureCategory as a Java enum.
What
When an async conversion or chunking task fails, the client builds the exception message from
task_meta, which only holds document counts:Docling Serve reports the cause of the failure in the
error_messagefield of the status poll response, butTaskStatusPollResponsedoesn't map it, so the cause is lost.How
TaskStatusPollResponsegains a nullableerrorMessagefield, mapped fromerror_message.FAILURE,AsyncOperationsuseserrorMessagewhen it is not blank. Otherwise it falls back to the task metadata, then toTask failed, as before.whats-new.mdhas an entry for the change.With the response above plus
"error_message": "Input document is not valid", the message is now:Tests
New
failedTaskReportsTheErrorMessageOfTheServerinAbstractDoclingServeClientAsyncExecutorTests, so both Jackson backends run it. It stubs a failed task witherror_messagein WireMock and checks the root cause message.TaskStatusMetadata(...))../gradlew --no-daemon :docling-serve-client:test --tests '*AsyncExecutorTests' :docling-serve-api:test spotlessCheckpasses (14 and 246 tests, 0 failures). The existingfailedTaskRunsEveryPollOnCustomExecutortest, which has neither field, still getsTask failed.Related
Closes #775