Evaluations in the console¶
The concept behind agents is explained in Agents.
In the Evaluations section you write down what an agent has to be able to do: a handful of conversations, each answer held to rules you wrote yourself. Running one replays those conversations against a pinned agent version and records what held and what did not. Use it to see, after a change to an agent, whether it still does its job.
An evaluation is always about agents. You need an administrator account of the organisation, and
the evaluations and agents capabilities have to be on in the organisation's ceiling. You do the
same with a key in Evaluations through the API; agents themselves are stored
in Agents in the console.
Adding an evaluation¶
Choose Add evaluation. The drawer opens with five fields:
| Field | What you fill in |
|---|---|
| Identifier | how you recognise this evaluation; can change later. The field shapes what you type into lowercase letters, digits, - and _, starting with a letter or digit, 64 characters at most |
| Description | what this evaluation measures. For readers only |
| Judge model | the model that decides the judged checks. Choose nothing and every agent uses its own model |
| Agents | the agents this evaluation measures, at least one. Saving later replaces the whole set with whatever is ticked then |
| Runs as | the key every run acts as |
Runs as is the field that decides whether you can start a run yourself. Every step of a run is an ordinary turn, and a turn is always a key's act: it is metered, audited and billed to that key and governed by its policy. You are a person and have no key of your own, so without this choice the Run buttons stay disabled, with that reason beside them. A key starting a run through the API does not need it — it simply acts for itself.
You may choose any key of the organisation. Every turn of the evaluation's runs then counts against that key's monthly cap, whoever starts the run. A schedule differs: a key that makes one through the API schedules as itself.
After Add you land on the new evaluation's Scenarios tab, because an evaluation with no scenarios can measure nothing.
Writing a scenario¶
A scenario is one conversation: a few turns in a row on one thread, so the agent remembers what was said in an earlier step. Choose Add scenario and fill in:
- Title and Description — how you find this conversation again in the table and in a run.
- Per step, When the user asks — the message that is sent.
- Per step, zero or more checks, the rules its answer is held to.
You save the whole scenario in one go. There is no separate route per step, deliberately: if two people edit at once, each writes the list they saw, and saving half a list is how you lose each other's work.
An evaluation holds at most 32 scenarios, a scenario at most 32 steps and a step at most 16 checks.
Prompt values¶
If any agent this evaluation measures pins a prompt with variables, a Prompt values section appears above the steps, with one field per variable those prompts declare — read off the agents themselves, so you never type a name. Every step of this scenario sends what you fill in.
One scenario is written for every agent the evaluation measures, so a value an agent's own prompt does not ask for is simply left out of that agent's turns. A field you leave blank sends nothing; a field the prompt marks as required has to be filled in before you can save.
Each field takes the type of value its variable declares: a number field for Number and Whole number, a choice between Yes (true) and No (false) for Yes or no, and a text field for Text and Choice; for a Choice you type one of its values. The scenario stores the value as that type. A value that does not fit what an agent's prompt allows, such as one outside the values of a Choice, fails that agent's every step, with a reason that names the variable.
A variable with a default is never required. Its empty field says what goes there instead, such as filled in automatically (Today's date) or default: «…». Leave it empty and Core fills in the default at every step; fill it in and your value wins.
A value stored under a name no linked agent asks for any more — the agent moved to a new prompt, or was unlinked — still gets a field, marked as such, so you can see it and clear it.
The checks¶
Pick Check for first; only the fields that kind uses appear afterwards. Eight checks are decided by a rule, four by a judge model.
| Check | Holds when | When to use it |
|---|---|---|
| Contains text | the answer contains the given text, ignoring case | a word, amount or deadline has to be in it |
| Does not contain text | that text is absent | something must never be mentioned |
| Holds no match of a pattern | the answer holds no match of the Pattern, a regular expression (RE2) whose match spans at most Longest match (characters), 1 to 200 | a bank account number, a phone number or another shape must never appear; the same rule a live check enforces |
| Exact match | the answer, stripped of whitespace, is literally equal | the answer is one fixed sentence or code |
| Close match | the answer is similar enough to the given text (Minimum similarity, 0.8 by default) | the wording may differ |
| Responds within time limit | the turn finished inside Maximum response time | slowness is a problem in itself |
| Must use tool | the turn called the named tool, or any tool at all when you name none | the agent has to look it up rather than guess |
| Found by a search | a search of the tool router in the turn, or the tools it picked in advance, listed the named tool among its hits | the agent's tools sit behind the router and you want to know whether it finds the right one |
| Correct format | a judge finds the answer matches the format you describe | "numbered list", "JSON only" |
| Polite tone | a judge finds the tone professional, helpful and polite | customer contact |
| No hallucination | a judge finds nothing was fabricated | the agent has to stay with its sources |
| LLM judge | a judge finds the answer meets the rule you write | anything that is not a string comparison |
Must use tool reads the turn's own tool calls, never its text: a model that writes "I will look that up" has looked nothing up.
For the four judged checks your text is the rule, not the right answer: write "the answer offers to escalate", not the sentence the agent should have said. The judge answers a score between 0 and 1 and the check passes from the Threshold, 0.7 by default. It sees only the step's message, the final answer and your rule — not the rest of the conversation and not the tool output.
Required to pass is on by default. A step passes when every required check passed; an optional check that does not hold gives a warning rather than a problem.
Versions and drafts¶
Every change to what an evaluation measures, its scenarios, its agents and its judge model, is a numbered version. A scenario you add, edit or delete on the Scenarios tab is published at once: every key and every schedule runs the new version from then on.
A draft is a version no key sees and no run of a key measures until you publish it. One arrives when you save a tried conversation as a scenario (see Agents in the console). Under the scenarios, the Versions card lists every version, a draft marked as one. Open a draft to see its scenarios, then:
- Measure draft queues a run of the draft, acting as the key the evaluation runs as. That run and everything it leaves are marked as a draft and are read in the console alone.
- Publish makes the draft the evaluation's scenarios.
- Discard draft removes it; its number is never handed out again.
Running an evaluation¶
There are three places to start one, and they differ only in how many runs they create:
- Run in the list's row menu, and Run all on the evaluation's page: one run per linked agent.
- Run beside an agent on the Settings tab, and the same button on that agent's own page: for that agent alone.
A run is queued and picked up by background processing; the page refreshes itself while anything is moving and stops on its own. Every run pins the agent version that was the newest at that moment and the evaluation's published version, and snapshots that version's scenarios, so a later edit changes nothing about what a finished run meant.
A run started here acts for nobody in particular; to measure the agent as it behaves for one person, start the run through the API (a run for one person).
When the button is off, the tooltip says why: there is no scenario yet, or no key has been chosen under Runs as.
The Settings tab also carries a Schedules card: the schedules that fire this evaluation, each a link to its own page, with Add schedule to create one with this evaluation already chosen (see Schedules).
Reading a run¶
The Runs tab lists every run with when it was queued, which agent and which version, how long it took, the score and the status. Click a row for the whole of it.
The header carries the agent and its version, the times, the duration, the score and the status, and, while the run is going, a Cancel run button. Under it sits a collapsible block per scenario, and inside it, per step, the message, the answer, how long the turn took and the checks.
What the colours mean¶
Each check carries a dot:
| Colour | Meaning |
|---|---|
| green | the check passed |
| red | the check did not pass |
| amber | the check could decide nothing — a judge model that did not answer, for instance |
Amber counts as "did not pass" towards the verdict, and that is deliberate: quietly counting a check nobody managed to run as a pass would be the more dangerous of the two. The dot is what tells "the agent got it wrong" from "we could not find out".
A whole run's verdict comes in three flavours:
| Status | Meaning |
|---|---|
| OK | every step passed |
| Warning | only optional checks did not hold |
| Issues | a step carrying a required check did not pass |
Failed and Cancelled are statuses, not verdicts: they say the run never got to have an opinion. The score is passed steps over the snapshot's own total, as a percentage, so a run halfway through reads as progress rather than as a mark the agent has already earned.
A cancelled run stops at its next step. The turn that was running really happened and stays on the bill; only the reading of that step is undone.
Deleting an evaluation or a scenario¶
Delete in the list's row menu removes the evaluation, its scenarios and all of its runs. This cannot be undone.
A scenario is deleted on the Scenarios tab. Runs that already exist keep their own snapshot and stay readable.
Empty lists and access restrictions¶
"No evaluations yet" means no evaluations have been created for this organisation.
If you see evaluations is off for this organisation, that capability is off in the organisation's ceiling. The administrator of the organisation above turns it on under Permitted capabilities on Settings; for an organisation with none above it, your operator does. Evaluations also require agents: without agents there is nothing to measure.
A key can be narrowed to particular agents. Naming an agent outside that set gives an error on the Agents field rather than an empty list, because you chose that agent yourself.