botscout.Sponsor your tool ↗

BOTSCOUT / PILOT LAB

Compare your
AI test results.

Add your results. See which tasks passed, which failed, and how much time and money each run used.

Save and compare your test runs.Save measured runs, catch regressions, and plan the next retest.
Open my Pilot Notebook ↗
01 Record every attempt02 Inspect the differences03 Choose what to share
01

Add your test results.

Record results below, or import a completed worksheet. Start with one case and keep every attempt; small samples are diagnostic, not proof of reliability.

A / BASELINE CONFIGURATION

What did you observe?

Include failures and timeouts. Use one tool version and configuration revision per file.

B / OPTIONAL CHALLENGER

Did the next version improve?

Use the same anonymous case IDs, groups, and manual baselines. Compare another configuration of this project.

Worksheet instructions & failure categories

Use 1–200 cases. Complete accepted_true_or_false and critical_failure_true_or_false with true or false. A critical failure must be unaccepted. Use anonymous IDs, never customer names or case content.

Record manual baseline, usage cost, review/correction, and fallback time on every case. Enter 0 when measured zero; leave unknown values blank. Accepted cases have zero fallback. Unknown values prevent an economic projection.

The optional failure_reason column accepts accuracy, missed_escalation, execution_error, permission_boundary, other, or unspecified. Leave it empty on accepted cases. Keep tool version, configuration revision, and test date on every row.

Your planning context

These example assumptions affect extrapolated value and the review gate. Replace them with your own; they are not provider prices.

Analyze sends these recorded measurements or worksheets and assumptions to BotScout for a transient calculation. They are not saved on the server unless you separately contribute below. After analysis, you can explicitly save a copy in this browser’s notebook. Use only nonconfidential measurements; no customer content or credentials.

PILOT METHOD 1.0.0

A better number still needs an explanation.

Critical failures come first.

Any recorded critical failure triggers “Stop and investigate.” Otherwise, acceptance below your target calls for revision. Missing costs or times call for more measurement.

Compare like with like.

Paired comparisons require the same IDs, groups, and manual baselines. Improvements and regressions both stay visible. The engine cannot verify identical inputs or independent execution.

Keep the limits visible.

Calculations describe the submitted sample. They do not prove statistical significance or general reliability. Value prices time released, subtracts usage and fixed tools costs, and excludes setup and maintenance.

No agent is run or provider charged. No automatic training, product ranking, or production authorization takes place. Sponsorship does not affect this analysis.