Agents / sentry-triage / eval suite
Eval suite · mocked scenarios
sentry-triage
Runs tagged with a trajekt.scenario. Each scenario is compared with itself across two versions, so a handful of scenarios is enough to catch a regression before it ships.
FAIL3.2.0vs3.1.0· 12 scenarios compared
- Mean duration regressed across the suite
- Mean cost regressed across the suite
- Steps per run regressed across the suite
- Tokens per run regressed across the suite
- Repeated tool calls regressed across the suite
- refund-rounding: 0/3 → 2/3 failed
- refund-rounding: Score · correct 1.00 → 0.33
- tax-rate-eu: Score · correct 1.00 → 0.33
Across the suite
| Metric | A · 3.1.0 | B · 3.2.0 | Change | Verdict |
|---|---|---|---|---|
| Score · correct | 1.00 | 0.89 | −0.11 | stable |
| Failed runs | 2.8% | 5.6% | +2.8 pts | stable |
| Failed tool calls | 1.0% | 4.6% | +3.5 pts | stable |
| Mean duration | 11.3s | 14.6s | +26% | regression |
| Mean cost | $0.047 | $0.061 | +26% | regression |
| Steps per run | 9.06 | 10.61 | +16% | regression |
| Tokens per run | 11.3k | 14.6k | +26% | regression |
| Repeated tool calls | 0 | 0.64 | +0.64 | critical |
| Runs on a new tool path 6 of 12 scenarios now take a tool path never seen in the baseline — shown, never blocking | — | 50.0% | regression |
Values are means of the per-scenario averages. Each scenario is paired with itself (Wilcoxon signed-rank test, one-sided, p < 0.05); duration, cost, steps and tokens are compared as ratios, so a large scenario does not outweigh a small one. Scores (higher is better) come first. A metric is flagged only if it stays significant after correcting for the number of metrics tested (Holm) and the typical change is large enough (+15%, +5 pts on a rate, −0.05 on a score), with at least 8 scenarios.
Scenario by scenario
6 worse · 0 better · 0 changed · 6 same| Scenario | What changed | Runs A / B | ||
|---|---|---|---|---|
| refund-rounding | worse | Fails more often · 0/3 → 2/3 failedMean duration +87%Mean cost +83%Tokens per run +89%Repeated tool calls 0 → 1Score · correct 1.00 → 0.33Tool path changed | 3 / 3 | diff → |
| tax-rate-eu | worse | Mean duration +67%Mean cost +64%Tokens per run +62%Repeated tool calls 0 → 1.67Score · correct 1.00 → 0.33Tool path changed | 3 / 3 | diff → |
| oauth-callback-500 | worse | Mean duration +62%Mean cost +52%Tokens per run +52%Repeated tool calls 0 → 1.33Tool path changed | 3 / 3 | diff → |
| coupon-expired | worse | Mean cost +63%Tokens per run +64%Repeated tool calls 0 → 1.33Tool path changed | 3 / 3 | diff → |
| npe-checkout-discount | worse | Repeated tool calls 0 → 1Tool path changed | 3 / 3 | diff → |
| search-index-stale | worse | Repeated tool calls 0 → 1Tool path changed | 3 / 3 | diff → |
| cart-empty-state | same | — | 3 / 3 | diff → |
| invoice-pdf-blank | same | — | 3 / 3 | diff → |
| password-reset-mail | same | — | 3 / 3 | diff → |
| payment-timeout | same | — | 3 / 3 | diff → |
| search-typo-tolerance | same | — | 3 / 3 | diff → |
| session-expired-loop | same | — | 3 / 3 | diff → |
A scenario is flagged when its failure rate rises by 50 points or more (e.g. 0/3 → 2/3), when a score drops by 0.25 or more, when duration, cost, steps or tokens grow by 50% or more, when it repeats a tool call it did not repeat before, or when it takes a tool path never seen in A.
App versions evaluated
| App version | Scenarios | Runs | Failed | Last run |
|---|---|---|---|---|
| 3.2.0 | 12 | 36 | 2 | 2 d ago |
| 3.1.0 | 12 | 36 | 1 | 6 d ago |