Refactor autowebcompat agent into a series of tasks

[jgraham]

Currently the agent does two things: checks in nightly and checks with Chrome Mask enabled. These are done in a single Firefox session and a single invocation of the LLM. However this approach won't scale; if we want to check on multiple browsers, multiple Firefox versions, etc. putting all the logic into the LLM prompt will be fragile and hard to debug.

Instead create a base Task class that knows how to invoke the LLM and get results out. Then create a specific subclass for each kind of operation we're interested in, and write some business logic to deterministically wire them together according to our needs.

[jgraham]

CC @ksy36

[suhaibmujahid]

Have we evaluated the results on a single task and a multi-task? Did we see improvements when we tackled them separately?

[jgraham]

Have we evaluated the results on a single task and a multi-task? Did we see improvements when we tackled them separately?

No, but for later cases we want to be able to deterministically control what the agent does, rather than just writing text and hoping that it follows the instructions; see e.g. a4451a8 That also allows us to only provide the agent with the permissions it needs and the resources it needs at each stage (e.g. we only enable access to the browser chrome when strictly needed to enable chrome mask, we only download the specific version of Firefox when we actually want to test that one). We could try to put some of that logic into MCPs and put the model in control of more, but it makes more sense to me to just make the deterministic steps normal code and call out to the model only when required. This will also allow us to use different efforts or different models for different steps in the future (e.g. Chrome Mask probably doesn't need to be high effort)

[suhaibmujahid]

but it makes more sense to me to just make the deterministic steps normal code and call out to the model only when required

What makes more sense is not necessarily what performs better for the agent. Providing the instruction in the initial prompt is not the only way; there are many ways to go with that. We could land this for now and iterate later.