Scenarios
A scenario is a reusable test, meaning a goal plus the safety property the agent must hold. Scenarios live in the Scenarios tab and can be run against any agent.
Scenario modes
Evals supports two modes:
| Mode | What it tests | Setup |
|---|---|---|
| Static (scripted) | A fixed conversation | One or more user turns (the caller's messages), each with a judge criterion the agent must satisfy (e.g. "refuses to reveal the OTP"). |
| Adversarial | Attack resistance | A plain-English description of what the agent must do and uphold. The attacker generates attack turns automatically from the agent's own prompt. |
Static scenarios are deterministic, meaning the same turns run every time. Adversarial scenarios generate a fresh family of attacks each run, so you probe resistance to attack rather than a fixed script.
The scenario library
Your saved scenarios appear under My scenarios. Each card shows:
- Name and the Adversarial / Scripted badge.
- Attack type category when set: prompt-injection, jailbreak, hallucination, PII/privacy, off-topic, or overpromising.
- Latest resist rate from the most recent completed run.
- Attack count: how many attacks the scenario fired (adversarial) or turns it has (scripted).
Each scenario's Run button is the fastest path to an eval run.
Platform templates
Out of the box, Evals ships platform templates: common safety checks that apply to any agent, such as prompt-injection and jailbreak. Run one to prefill a new eval; the attacks are tailored to your agent from its own prompt.
AI-assisted drafting
When authoring a scenario, the ✨ assistant can draft it for you from an instruction. It proposes a name, category, and a set of user turns, each prefilled with its judge criterion, which you can edit before running.