botscoutRecord the configuration.
Use the same cases and policy context for each tool. Keep model version, prompt, date, and settings with every run.
BOTSCOUT BENCHMARK / 1.0.0
Same task.
Inspectable results.
A starter suite for routing support tickets, choosing when to escalate, and reviewing grounded reply drafts. Built from 24 original synthetic scenarios for a fictional product.
Keep failures in the sample.
Every case is required once. Capture decisions, drafts, and optional cost and latency. Record timeouts or execution errors explicitly; they count as failures instead of disappearing from the sample.
Make the judgment visible.
A human checks whether each draft is supported by policy and usable. Missing human reviews of produced drafts prevent an end-to-end success figure. Execution errors count as unsuccessful cases.
How scoring works
| Measure | Rule |
|---|---|
| Routing | Exact match against each case’s expected queue. |
| Escalation | Exact boolean match; missed required escalations are reported separately. |
| End-to-end acceptance | Correct routing and escalation plus a human review marking the draft both supported and usable. Only shown when every produced draft has human review; explicit execution errors count as failures. |
| Cost and latency | Operator-supplied measurements. Totals and mean latency are withheld when coverage is incomplete. |
A sample case
I cancelled six weeks ago. Give me a refund now.
Policy: Refunds after 30 days require a human billing review; do not promise eligibility.
Expected queue: billing · Escalation: required
The downloaded suite includes expected labels. This makes the starter methodology inspectable, but it is not a hidden or production-representative benchmark.
Run it with your workflow
Pass the cases through your chosen tool, complete the response template, and retain the original outputs. The BotScout owner uses the private scoring console to validate supplied outputs. It computes metrics without invoking a model or sending data to a provider. Agree on a scope and secure transfer method before sharing an evaluation file.
The owner can import response JSON through the private evidence console. The server recomputes the metrics before a separate publication review. Raw drafts and reviewer identifiers are not included in public evidence.
Ask about a scoped evaluation ↗What this can and cannot establish
- This small public synthetic suite is a development baseline, not a representative production sample or hidden test set.
- Execution, tool versions, costs, latency, and human review are supplied by the operator. Scoring does not independently verify execution.
- Draft quality requires an identified human reviewer; automated routing alone is not end-to-end success.
There are no published vendor scores until actual outputs are submitted and reviewed. A passing starter test is not a certification, safety guarantee, or claim of financial return.
Dataset provenance and version
Original synthetic scenarios created for BotScout with AI assistance; no customer tickets.
Dataset version 1.0.0 · scorer 1.0.0
2b6738903d1c5d528a69435181d1d9cbc8a10bd1168613b1b30cc567b235a988