Runs

A run is a single execution of a test. It runs one or more scenarios against an agent and scores every reply. This page covers starting a run, watching it live, and reading the run detail page.

Starting a run

From the scenario's Run button (or New run), you pick:

1. Agent: the agent you want to test.
2. Judge model: the AI reviewer that decides whether each reply is safe. The agent under test keeps its own model.

Then choose the mode:

• Adversarial: describe what the agent must do and uphold. The attacker, judge, and improver all work from this plus the agent's instructions, and the safety properties and attacks are generated automatically.
• Static: write the conversation turns and a judge criterion for each, optionally saving the scenario to the library.

Ticking Harden the prompt if attacks get through also runs the attacker, then keeps improving the agent's instructions (keeping changes that work, undoing those that don't) until the attacks no longer break through, leaving you one review away from a fix.

Watching a run

Runs stream live over a WebSocket. While active, the run page shows a Live panel: each new turn as the attacker throws attacks and the agent responds, plus how many scenarios have completed. The status pill tracks queued → running → completed (or failed/error).

The run detail page

Every run opens at its detail page:

Verdict

A decision-first header: a resist-rate ring (green ≥ 90%, amber ≥ 60%, red below), a one-line verdict, and the immediate action. The resist rate is the share of attacks the agent stopped. A run's verdict is one of:

• No breaches found: the agent held against every attack.
• N of M attacks got through: a breach summary, with an Optimize the prompt button.
• A prompt fix is ready to review: when an optimized run produced a proposal, with Accept & apply to agent / Reject inline.
• Already safe, no change needed, No automatic fix found, or a run-level failure reason.

What we tested

For optimize runs this lists the attack goal and the must hold criterion, plus the frozen train attacks, held-out attacks, and max iterations. Plain eval runs show how many attacks were generated.

Cost & tokens

Every run reports token usage (how much text the AI processed), split by who spent them:

• The agent: the voice agent's model doing the talking (~99% of the run).
• The judge: the AI scoring each reply.
• Creating the attacks: the AI calls that write the trick messages and propose fixes.

The total is charged to your wallet as agent cost + eval cost, with no platform markup.

Diagnosing results

The Diagnose section breaks down every scenario in the run:

Each scenario result shows its pass rate (passed/total) and a row per attack:

• Resisted / Tricked: whether the agent held against that attack, with the exact attacker message.
• How the agent replied: the full response for context.
• Tools it called: any function-tool invocations with their arguments.
• Why: the judge's reasoning.

For static scenarios, Failures lists each failed turn with its criterion, and a Transcript shows the conversation. Filter between All / Tricked / Resisted to focus on what broke.

What's next

• Optimizing a prompt: turn breaches into an applied safety fix.