Repository navigation
Commit 24ef3c9
fix(gooddata-eval): keep the simulated user from writing MAQL
In context mode an agent that hit a MAQL error and asked "Should I
retry with this corrected formula?" was classified as asking a
question, not as asking for confirmation. The question went to the
clarification judge, which graded it as lost context. The simulated
user was then asked to restate what had been said earlier, and the
only formula said earlier was the agent's own rejected proposal, so it
sent that back. The turn failed even though the metric the agent
finally created matched the expected one.
- Treat retry / try (it|this|that) again / resubmit after
"should/shall/may/can I", "do you want me to" and "would you like
me to" as a confirmation, so the user approves the revised proposal
with "Yes, go ahead."
- A retry that offers a choice stays a question: "Should I retry
with gross or net revenue?" and "Should I retry? Or adjust the
formula first?". An OR inside an inline formula or a {type/id}
reference is not a choice, and a dot inside a token ("2.5") does
not end the sentence.
- `use` and `run` stay out, because they would turn choice
questions such as "Which one should I use?" into confirmations.
- The verbs now end on a word boundary, so "Should I address ..." is
no longer read as "Should I add".
- Tell the simulated user in reply_restating and reply_from_facts
never to write MAQL, SQL, formulas or object definitions.
- Replace any generated reply that still contains MAQL or SQL with
NO_ANSWER_REPLY. That covers uppercase SELECT, an uppercase
aggregation call (SUM(...), COUNT(...), ...), "select" in any case
before an aggregation call, lowercase SQL shaped as select * /
a column list / a table followed by where, group by, order by or
join, and {metric/...}, {fact/...}, {label/...}, {attribute/...} or
{dataset/...} references in braces or angle brackets. A plain
"Select the first option from the list." is left alone. Both
callers answer a question, so the fallback has to answer one too,
not just approve. Fixture set answers are sent unchanged.
jira: QA-29876
risk: low
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>1 parent f04c9ce commit 24ef3c9
2 files changed
Lines changed: 142 additions & 10 deletions
File tree
- packages/gooddata-eval
- src/gooddata_eval/core/agentic
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
40 | | - | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
41 | 46 | | |
42 | | - | |
43 | | - | |
44 | | - | |
| 47 | + | |
| 48 | + | |
45 | 49 | | |
46 | 50 | | |
47 | 51 | | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
48 | 62 | | |
49 | 63 | | |
50 | 64 | | |
| |||
77 | 91 | | |
78 | 92 | | |
79 | 93 | | |
80 | | - | |
| 94 | + | |
81 | 95 | | |
82 | 96 | | |
83 | 97 | | |
| |||
238 | 252 | | |
239 | 253 | | |
240 | 254 | | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
241 | 280 | | |
242 | 281 | | |
243 | 282 | | |
| |||
271 | 310 | | |
272 | 311 | | |
273 | 312 | | |
274 | | - | |
275 | | - | |
| 313 | + | |
| 314 | + | |
276 | 315 | | |
277 | 316 | | |
278 | 317 | | |
279 | | - | |
| 318 | + | |
280 | 319 | | |
281 | 320 | | |
282 | 321 | | |
| |||
289 | 328 | | |
290 | 329 | | |
291 | 330 | | |
292 | | - | |
| 331 | + | |
293 | 332 | | |
294 | 333 | | |
295 | | - | |
| 334 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1450 | 1450 | | |
1451 | 1451 | | |
1452 | 1452 | | |
| 1453 | + | |
| 1454 | + | |
| 1455 | + | |
| 1456 | + | |
| 1457 | + | |
| 1458 | + | |
| 1459 | + | |
| 1460 | + | |
| 1461 | + | |
| 1462 | + | |
| 1463 | + | |
| 1464 | + | |
| 1465 | + | |
| 1466 | + | |
| 1467 | + | |
| 1468 | + | |
| 1469 | + | |
| 1470 | + | |
1453 | 1471 | | |
1454 | 1472 | | |
1455 | 1473 | | |
| |||
1567 | 1585 | | |
1568 | 1586 | | |
1569 | 1587 | | |
| 1588 | + | |
| 1589 | + | |
| 1590 | + | |
| 1591 | + | |
| 1592 | + | |
| 1593 | + | |
| 1594 | + | |
| 1595 | + | |
| 1596 | + | |
| 1597 | + | |
| 1598 | + | |
| 1599 | + | |
| 1600 | + | |
| 1601 | + | |
| 1602 | + | |
| 1603 | + | |
| 1604 | + | |
| 1605 | + | |
| 1606 | + | |
| 1607 | + | |
| 1608 | + | |
| 1609 | + | |
| 1610 | + | |
| 1611 | + | |
| 1612 | + | |
| 1613 | + | |
| 1614 | + | |
| 1615 | + | |
| 1616 | + | |
| 1617 | + | |
| 1618 | + | |
| 1619 | + | |
| 1620 | + | |
| 1621 | + | |
| 1622 | + | |
| 1623 | + | |
| 1624 | + | |
| 1625 | + | |
| 1626 | + | |
| 1627 | + | |
| 1628 | + | |
| 1629 | + | |
| 1630 | + | |
1570 | 1631 | | |
1571 | 1632 | | |
1572 | 1633 | | |
| |||
1960 | 2021 | | |
1961 | 2022 | | |
1962 | 2023 | | |
| 2024 | + | |
| 2025 | + | |
| 2026 | + | |
| 2027 | + | |
| 2028 | + | |
| 2029 | + | |
| 2030 | + | |
| 2031 | + | |
| 2032 | + | |
| 2033 | + | |
| 2034 | + | |
| 2035 | + | |
| 2036 | + | |
| 2037 | + | |
| 2038 | + | |
| 2039 | + | |
| 2040 | + | |
| 2041 | + | |
| 2042 | + | |
| 2043 | + | |
| 2044 | + | |
| 2045 | + | |
| 2046 | + | |
| 2047 | + | |
| 2048 | + | |
| 2049 | + | |
| 2050 | + | |
| 2051 | + | |
| 2052 | + | |
| 2053 | + | |
| 2054 | + | |
| 2055 | + | |
1963 | 2056 | | |
1964 | 2057 | | |
1965 | 2058 | | |
| |||
0 commit comments