[DiadkoShmek]

Description

Cancelling one prediction in an adaptive batch can prevent a different, still-active prediction from receiving its response. _wait_response removes the cancelled request's Future; _predict_callback then indexes the missing ID, raises KeyError, and indexes it again while distributing that error. A cancelled first member can strand its peer, while a cancelled last member produces an unhandled event-loop callback error.

Deliver batch results and model errors only to present, unfinished Futures. Apply the same completed-Future guard when clearing a failed queue. Shared model prediction continues, and the surviving request receives its original result or model exception.

Changes Made

• Guard per-request delivery in _predict_callback and _clear_queue.
• Add public load_batching/model predict regressions using a gated synthetic SumModel, real merge/split and numeric codecs. Cover cancellation of either member, successful/error model outcomes, no-cancellation controls and recovery through the same batcher.
• Add a queue-cleanup regression for an already-cancelled registered Future.

Related Issues

Source-audited defect; no existing issue selected. Bounded all-state batching/cancellation searches found no matching fix.

Validation

• Exact original implementation with the final batching tests: 6 failed, 44 passed.
• Fixed batching plus codec suites: 249 passed.
• Independent AI review reran 50 batching tests and eight extra cancellation-timing/cleanup cases, with correct surviving results or original exception identity and no loop callback errors.
• Black, Flake8, git diff --check and full mypy ./mlserver pass (101 source files).

The public Python batching boundary is tested. HTTP/gRPC disconnect behavior, cancellation of the shared prediction task and the full integration/runtime matrix were not tested. No provider calls or model downloads were used.

Checklist

• Code follows the project's style guidelines.
• Related batching and codec tests pass as detailed above.
• No documentation or public signature changes are needed.
• Code is reviewed by at least one other team member. Independent AI review is complete; upstream team review remains pending.
• No breaking API change.

Additional Notes

AI assistance: OpenAI Codex performed source inspection, implementation, local verification and independent AI review. No human review or upstream-team membership is claimed.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[DiadkoShmek]

CLA Assistant currently displays Seldon CLA version 2024-04-12, section 3, stating that the Contribution "will infringe upon or misappropriate" third-party rights. Could you confirm whether "will not infringe" is intended and correct the displayed agreement? I can complete the CLA after that wording is clarified.