Agents / sentry-triage
sentry-triage
1.5.0last run 10 min ago
Runs · 24 h
40
280 over 7 days
Failed · 24 h
10.0%
p50 duration · 24 h
15.0s
Mean cost · 24 h
$0.052
Is it getting worse?
Modelgpt-4.1-mini(145 runs)vsclaude-sonnet-4-5(415 runs)no regression
| Metric | A · claude-sonnet-4-5 | B · gpt-4.1-mini | Change | Verdict |
|---|---|---|---|---|
| Failed runs | 4.1% | 3.4% | −0.6 pts | stable |
| Failed tool calls | 2.8% | 1.4% | −1.4 pts | stable |
| Mean duration | 12.1s | 11.6s | −4% | stable |
| Mean cost | $0.049 | $0.0059 | −88% | better |
| Steps per run | 9.44 | 9.15 | −3% | stable |
| Tokens per run | 11.9k | 11.5k | −4% | stable |
| Repeated tool calls | 0.13 | 0.03 | −74% | better |
| Runs on a new tool path | 0.0% | 0.0% | 0.0 pts | stable |
| Score · correct | 0.91 | 0.91 | +0.00 | stable |
A metric is flagged only when the change is both statistically significant (one-sided test, p < 0.01, Holm-corrected for the number of metrics) and large enough to matter (+15% on a mean, +3 pts on a rate, −0.05 on a score), with at least 20 runs on each side.
Last 14 days · dashed lines mark the first run of each version
stepsrepeated tool calls
p50p95
Tool paths
Model gpt-4.1-mini· 145 runs
| sentry.getIssue → gitlab.searchCode → gitlab.readFile → jira.createIssue → slack.postMessage | 97% |
| sentry.getIssue → gitlab.searchCode ×2 → gitlab.readFile → jira.createIssue → slack.postMessage | 2% |
| sentry.getIssue → gitlab.searchCode ×2 → gitlab.readFile → jira.createIssue → gitlab.createMergeRequest → slack.postMessage | 1% |
Model claude-sonnet-4-5· 415 runs
| sentry.getIssue → gitlab.searchCode → gitlab.readFile → jira.createIssue → slack.postMessage | 90% |
| sentry.getIssue → gitlab.searchCode ×2 → gitlab.readFile → jira.createIssue → slack.postMessage | 5% |
| sentry.getIssue → gitlab.searchCode ×3 → gitlab.readFile → jira.createIssue → slack.postMessage | 3% |
| sentry.getIssue → gitlab.searchCode ×2 → gitlab.readFile → jira.createIssue → gitlab.createMergeRequest → slack.postMessage | 1% |
| sentry.getIssue → gitlab.searchCode ×3 → gitlab.readFile → jira.createIssue → gitlab.createMergeRequest → slack.postMessage | 0% |
Models
| Model | First run | Runs | Failed | p50 duration | Mean cost | Steps | Repeats | Score · correct |
|---|---|---|---|---|---|---|---|---|
| claude-sonnet-4-5 | 14 d ago | 415 | 4.1% | 11.4s | $0.049 | 9.4 | 0.13 | 0.91 |
| gpt-4.1-mini | 14 d ago | 145 | 3.4% | 11.4s | $0.0059 | 9.2 | 0.03 | 0.91 |
Recent runs
Runs with repeated tool calls →| Run | Version | Status | Started | Duration | Steps | Cost | Flags |
|---|---|---|---|---|---|---|---|
| run_b590779e | 1.5.0 app 3.2.0 · prompt v8 | ok | 10 min ago | 11.5s | 9 | ≈$0.043 | |
| run_ad4ce345 | 1.5.0 app 3.2.0 · prompt v8 | ok | 1 h ago | 18.9s | 13 | ≈$0.081 | repeat ×2 |
| run_ae257847 | 1.5.0 app 3.2.0 · prompt v8 | ok | 2 h ago | 14.9s | 11 | ≈$0.065 | repeat ×1 |
| run_6f21d623 | 1.5.0 app 3.2.0 · prompt v8 | ok | 2 h ago | 14.4s | 11 | ≈$0.0075 | repeat ×1correct 0.00 |
| run_34f63b8a | 1.5.0 app 3.2.0 · prompt v8 | ok | 3 h ago | 15.1s | 11 | ≈$0.0083 | repeat ×1correct 0.00 |
| run_4ca1c1c2 | 1.5.0 app 3.2.0 · prompt v8 | ok | 3 h ago | 15.7s | 11 | ≈$0.065 | repeat ×1correct 0.00 |
| run_1f44248d | 1.5.0 app 3.2.0 · prompt v8 | ok | 4 h ago | 17.4s | 11 | ≈$0.064 | repeat ×1 |
| run_604782f9 | 1.5.0 app 3.2.0 · prompt v8 | ok | 5 h ago | 11.5s | 9 | ≈$0.048 | correct 0.00 |
| run_2a751ac7 | 1.5.0 app 3.2.0 · prompt v8 | ok | 5 h ago | 12.6s | 9 | ≈$0.0057 | |
| run_6d9634c4 | 1.5.0 app 3.2.0 · prompt v8 | ok | 6 h ago | 16.4s | 11 | ≈$0.067 | repeat ×1correct 0.00 |
| run_9b3a9bed | 1.5.0 app 3.2.0 · prompt v8 | ok | 6 h ago | 14.5s | 11 | ≈$0.064 | repeat ×1 |
| run_ebad7bdf | 1.5.0 app 3.2.0 · prompt v8 | ok | 7 h ago | 13.3s | 10 | ≈$0.0056 | retry ×11 tool error |