Random 'No tool output found ' exception from the endpoint side

[hyeoungho]

While running the latest code from the repo, I get below error from the computer-use-preview endpoint:

  File "C:\Work\python-projects\ComputerUse\.venv\Lib\site-packages\openai\_base_client.py", line 1023, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': 'No tool output found for computer call call_4.', 'type': 'invalid_request_error', 'param': 'input', 'code': None}}

The issue here is that the case is quite random - sometimes it throws at the very beginning and the other times it does after a few actions. I cannot figure out which condition makes the problem. I am using the same CUA.py code of the latest version in this repo. I just added the print part to see what's going on.

(.venv) c:\Work\python-projects\ComputerUse\computer-use-model\computer-use>python main.py --instruction "Open Chrome browser and search the weather of Bellevue WA and summarize the information for this week"
User: Open Chrome browser and search the weather of Bellevue WA and summarize the information for this week
***computer_call_output***
call_id: call_1
screenshot {}
***calling client***
call_id: call_2

Action: Launching Chrome browser on desktop
***computer_call_output***
call_id: call_2
double_click {'x': 45, 'y': 140}
***calling client***
call_id: call_3

***computer_call_output***
call_id: call_3
wait {}
***calling client***
call_id: call_zvCUYd0VDqkNxYhJ1Kl7RUao

Action: Searching weather for Bellevue, WA
***computer_call_output***
call_id: call_4
type {'text': 'weather Bellevue WA'}
***calling client***
Traceback (most recent call last):
....

[RandulaJAriyawanse]

Same problem here. And it's the same if you follow the OpenAI computer use docs (swapping with the Azure client)
https://platform.openai.com/docs/guides/tools-computer-use

[hyeoungho]

Last night, I figured out what is happening (on the model endpoint side). Basically, the model somehow thinks that the current input belongs to older call_id, which is the reason of the exception since the action taken does not match with the old call_id.

In below snippet, the call_id changes when the second client call, which contains the action order from the model (call_2). However, the model expected the previous call_id (call_1) when the system reports the result to the model.

previous_response_id: resp_a
***computer_call_output***
call_id: call_1
action: wait
wait {}
***calling client***
call_id: call_2

Action: Searching current week weather Bellevue
previous_response_id: resp_a
***computer_call_output***
call_id: call_2
action: click
click {'button': 'left', 'x': 210, 'y': 61}
***calling client*** 
response exception
> c:\work\python-projects\computeruse\computer-use-model\computer-use\cua.py(247)continue_task()
-> import pdb;pdb.set_trace()
(Pdb) e
BadRequestError("Error code: 400 - {'error': {'message': 'No tool output found for computer call call_1.', 'type': 'invalid_request_error', 'param': 'input', 'code': None}}")