Observability for agents

[evgenyrp]

There should be a way to track agentic steps in production for easier inspection and debugging.

We use Weights and Biases Weave for evaluation (see example of evaluation and traces). We should explore how suitable it is for production workflows.

Also, we need dashboards that show API request rates, processed queue messages, error counts, AI costs, and other useful metrics for the Hackbot services. Ideally, those should be customizable (think Grafana way). Maybe GCP dashboards can be sufficient.

[evgenyrp]

We'll start with integrating with W&B Weave initially and then adjust if we don't like it. GCP dashboards seem to be sufficient for service monitoring, as Cloud Run provides some out of the box.