Skip to content

fix(agent): degrade tool-result images for text-only providers - #10779

Open
he-yufeng wants to merge 7 commits into
AstrBotDevs:masterfrom
he-yufeng:fix/tool-result-image-degradation
Open

he-yufeng wants to merge 7 commits into
AstrBotDevs:masterfrom
he-yufeng:fix/tool-result-image-degradation

Conversation

@he-yufeng

@he-yufeng he-yufeng commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #10702

Problem

When a plugin tool returns ImageContent and the current provider does not support multimodal input, the entire turn can fail with a 400 response such as "Multimodal data provided, but model does not support multimodal requests." The provider error does not identify the tool result that introduced the image.

Root cause

tool_loop_agent_runner.py caches tool-result images and appends them to the context as ImageURLPart user messages. Its supports_image check treated an empty or missing modalities value as image support for backward compatibility, so providers with unknown or text-only capabilities could receive image data.

Tool-result images differ from initial request context: they introduce new multimodal content that the provider did not supply or request.

Fix

  • Append cached tool-result images only when the provider config explicitly lists "image" in modalities.
  • Treat empty, null, and missing modalities values as unknown capabilities. Skip the image append and avoid reading the image back as base64.
  • Preserve the existing text description with the cached image path in the tool result, so the model still knows that the tool returned an image.
  • Keep the final diff limited to tool_loop_agent_runner.py and its runner regression test. The unrelated QQ Official commits have been reverted from this PR.

Testing

  • The new regression drives the real ToolLoopAgentRunner with a mixed ImageContent and TextContent tool result. Against the old upstream implementation, the empty-capabilities case appended an unwanted ImageURLPart; the fixed branch skips it. Explicit null, text-only with tool_use, and image-capable with tool_use configurations are covered as well.
  • python -I -m pytest tests/test_tool_loop_agent_runner.py -q: 64 passed.
  • Related modality and image tests in tests/unit/test_astr_main_agent.py: 12 passed.
  • Full tracked Python suite, including docs/tests: 4877 passed, 14 skipped, 2 failed. Both failures are the two test_config_update_revokes_only_affected_shell_sessions cases, and both reproduce unchanged on upstream base 832fc6d98.
  • Ruff check, Ruff format check, and git diff --check passed on the changed files.

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code
  • I have run tests
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective

AI Assistance Disclosure

  • AI tools were used (specify below)

Claude Code assisted with investigating the issue, implementing the fix, and running the regression checks.

Inbound quotes already work: the adapter reads message_reference off
incoming messages and builds a Reply component from it. The outbound
half was never wired up, so a plugin quoting a message with
Comp.Reply(id=...) produced plain text with no quote bubble.

_parse_to_qqofficial() now returns the first Reply component's message
id alongside the media fields, and both send paths (_post_send_one and
_send_by_session_common) attach it as message_reference. All four send
APIs used here (v2 groups, v2 C2C, guild channel, guild DM) accept that
field. The active-send fallback drops the reference together with
msg_id, so a stale quote id degrades to plain text instead of blocking
the message.
…nd fallback

Address review findings:

- The passive-to-active fallback stripped message_reference together with
  msg_id on any failure, so an unrelated send error (markdown rejection,
  transient server error) lost a perfectly valid quote context. The active
  retry now keeps the reference first and only drops it when the retry
  with the reference also fails, since a dead reference id must not block
  the message body either.
- The Reply extraction tests only covered Reply at the chain start. Added
  a Plain-before-Reply parser case and two fallback-behavior cases.
When the QQ official API refuses a native markdown payload, the
fallback used to drop msg_id and resend through the active-message
interface, which runs into the 304049 active-message rate limit and
loses the reply. Reorder the fallback so a rejected markdown degrades
to a plain content reply that keeps msg_id (passive reply, msg_type
reset from 2 to 0, stream trailing newline restored). The stream
newline repair retry also keeps msg_id. Only errors that are not
markdown rejections take the active-send fallback, which keeps the
message_reference two-stage retry from the previous fix. Fallback
raises are chained so the original rejection stays visible in logs.
When a plugin tool returns ImageContent and the provider does not
explicitly declare image support, the cached image was still appended
to the context as an ImageURLPart, causing a 400 that aborted the
entire turn. Now only append cached images when the provider config
explicitly lists 'image' in modalities. The text description in the
tool result already tells the LLM the image exists and where it is
cached.

Fixes AstrBotDevs#10702

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 5 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py" line_range="660" />
<code_context>
+                    fallback_content = cast(str, fallback_payload.get("content") or "")
+                    if fallback_content and not fallback_content.endswith("\n"):
+                        fallback_payload["content"] = fallback_content + "\n"
+                return await send_func(fallback_payload)

             # QQ 流式 markdown 分片校验:内容必须以换行结尾。
</code_context>
<issue_to_address>
**QQ replies are lost**

When QQ rejects a Markdown send and the plain-content retry also fails with an error recoverable by active sending, `_send_with_markdown_fallback` lets the plain-content retry’s exception escape, so it never reaches the active-send fallback and the reply is not delivered.

Catch the plain-content retry error and continue to the active-send fallback.

Also at `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:689`.
</issue_to_address>

### Comment 2
<location path="astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py" line_range="689" />
<code_context>
             return await send_func(payload)
         except _QQOFFICIAL_SEND_API_ERRORS as err:
-            logger.info("[QQOfficial] 回复消息失败: %s, 尝试使用主动发送接口。", err)
-            if payload.get("msg_id"):
+            # markdown 被服务端拒绝(50056)时,保留 msg_id 退化为纯 content
+            # 被动回复。先去掉 msg_id 会把被动回复变成主动消息,既烧主动
</code_context>
<issue_to_address>
**Quoted friend replies are lost**

When a friend-session send has no cached `msg_id` and QQ rejects its `message_reference`, `_send_by_session_common` omits `msg_id` for friend sends, while `_send_with_markdown_fallback` only retries without `message_reference` inside the `msg_id` branch; the QQ error escapes and the reply body is lost.

Allow the rejected-reference retry independently of whether the payload has a `msg_id`.

Also at `astrbot/core/platform/sources/qqofficial/qqofficial_platform_adapter.py:414`.
</issue_to_address>

### Comment 3
<location path="astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py" line_range="703" />
<code_context>
+                        raise fallback_err from err
+                    # 主动发送仍失败,引用 id 可能已失效;去掉引用最后重试一次,
+                    # 失效引用不应阻断正文送达
+                    retry_payload = fallback_payload.copy()
+                    retry_payload.pop("message_reference", None)
+                    try:
</code_context>
<issue_to_address>
**Valid reply quotes disappear**

When an active send with a valid reference gets an unrelated API error and the retry without the reference succeeds, `_send_with_markdown_fallback` removes `message_reference` after any active-send API error, so a successful retry delivers an unquoted reply; when QQ accepted the referenced send but returned a server error, the retry can also deliver a duplicate.

Retry without the reference only when the API error identifies a rejected reference, and do not retry ambiguous server errors.

Also at `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:704-709`.
</issue_to_address>

### Comment 4
<location path="astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py" line_range="458-459" />
<code_context>
             payload["msg_seq"] = random.randint(1, 10000)

+        # 插件通过 Reply 组件给出的引用 id,四个发送接口(群/C2C/频道/频道私信)
+        # 都以 message_reference 接受;拿不到 id 时保持原样,不影响正文发送
+        if reply_message_id:
+            payload["message_reference"] = {"message_id": reply_message_id}
+
</code_context>
<issue_to_address>
**New QQ comments are Chinese**

When the newly added QQ Official comments and fallback log are reviewed or emitted, the added comments and quote-fallback log remain in Chinese, so maintainers receive non-English implementation guidance and diagnostics contrary to the repository convention.

Translate the newly added comments and log message into English.

Also at `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:639-641`, `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:664`, `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:687`, `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:692-693`, `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:701-702`, `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:707`, `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:995-996`.
</issue_to_address>

### Comment 5
<location path="astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py" line_range="994-999" />
<code_context>
                     file_source = file_path
                 elif i.url:
                     file_source = i.url
+            elif isinstance(i, Reply) and not reply_message_id:
+                # 出站引用回复:取第一个带被引用消息 id 的 Reply 组件
+                quoted_id = str(i.id).strip()
+                if quoted_id:
+                    reply_message_id = quoted_id
             else:
                 logger.debug(f"qq_official 忽略 {i.type}")
         return (
</code_context>
<issue_to_address>
**Unrelated QQ quoting broadens scope**

When the PR's stated problem remains limited to tool-result images for text-only providers, the new QQ Official outbound quote handling adds behavior unrelated to the text-only provider image failure, broadening this PR beyond the problem it claims to fix.

Move the QQ outbound quote-support changes into a separate PR unless they are explicitly part of this issue's scope.

Also at `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:459-460`, `astrbot/core/platform/sources/qqofficial/qqofficial_platform_adapter.py:413-414`.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 3 findings to address first, and if QQ reports an error after accepting a send, the retries can deliver a duplicate; the final retry can also send without the requested quote. Delivered messages cannot be recalled by reverting this change.

Blocking findings: astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:660, astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:689, astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:703


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py Outdated
Comment thread astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py Outdated
Comment thread astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py Outdated
Comment on lines +458 to +459
# 都以 message_reference 接受;拿不到 id 时保持原样,不影响正文发送
if reply_message_id:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ Low · New QQ comments are Chinese

When the newly added QQ Official comments and fallback log are reviewed or emitted, the added comments and quote-fallback log remain in Chinese, so maintainers receive non-English implementation guidance and diagnostics contrary to the repository convention.

Translate the newly added comments and log message into English.

Also at astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:639-641, astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:664, astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:687, astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:692-693, astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:701-702, astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:707, astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:995-996.

Prompt for AI agents
In `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py` at lines 458-459:

**New QQ comments are Chinese**

When the newly added QQ Official comments and fallback log are reviewed or emitted, the added comments and quote-fallback log remain in Chinese, so maintainers receive non-English implementation guidance and diagnostics contrary to the repository convention.

Translate the newly added comments and log message into English.

Also at `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:639-641`, `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:664`, `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:687`, `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:692-693`, `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:701-702`, `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:707`, `astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py:995-996`.

From AGENTS.md L50:

5. Use **English** for all comments and logs.

Comment thread astrbot/core/platform/sources/qqofficial/qqofficial_message_event.py Outdated
Cover empty, null, text-only, and explicitly image-capable providers through the real tool-loop runner. Providers without explicit image support keep the cached-image text without loading or appending image data.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] 纯文本 provider 遇到工具返回的图片时整轮 400,且报错不指向来源

1 participant