Evaluations through the API¶

The concept behind agents is explained in Agents.

On this page you write an evaluation through the API, start a run and read the outcome. An evaluation is a suite of conversations you write down once and then run against an agent. Each conversation is a scenario, each scenario is a list of steps, and each step is one user message plus the rules its answer has to hold to. A run replays those conversations against a pinned agent version and records what held and what did not. Use it to see, after a change to an agent, whether it still does its job.

Your key needs the evaluations capability. Evaluations apply to agents only: that is why evaluations needs agents, and why a suite names at least one agent. A run also needs threads: a scenario is a conversation, and its steps chain on one thread. You do the same with a mouse in Evaluations in the console.

Creating a suite¶

curl -s -X POST "$CORE_URL/evaluations" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{"name": "triage-quality",
         "description": "Does triage stay polite and name the ticket number?",
         "judge_model": "<a model id>",
         "run_as_key_id": "key_9Tc4...",
         "agents": ["agt_5Nj2..."]}'

A successful request answers 201.

An agent your key's policy does not name cannot be linked: that is 422 on agents. Nor can a key that is not your organisation's: that is 422 on run_as_key_id. PATCH /evaluations/{id} changes what you send; agents replaces the whole set, and an explicit null on judge_model or run_as_key_id clears that field.

Which key a run runs under¶

Every step of a run is an ordinary turn, and a turn is always recorded against a key. So every run records which key it runs under:

1. the key the suite names in run_as_key_id, when it names one;
2. otherwise the key that starts the run;
3. otherwise 409 — the suite has to name a key first.

That third case is an administrator in the management surface: a person has no key of their own to fall back on. Name one and the management surface can start runs like anyone else.

Whether the agent is reachable is decided against the policy of the key the run will run under, not against yours. If that key cannot see the agent you get 422 straight away rather than a run that can do nothing. If the named key has since been deleted, that is 422 on run_as_key_id: there is never a silent fallback to your own key.

created_by on a run stays whoever started it (a key id, or operator:<sub>); run_as_key_id says whose act its turns are.

A key that may manage evaluations may name any key of its organisation here, also one that holds more capabilities or a higher cap than it has itself. The turns of the suite's runs then fall under the named key's policy and count against the named key's monthly cap, not yours. A schedule works differently: your key schedules as itself.

Writing a scenario¶

curl -s -X POST "$CORE_URL/evaluations/evl_7Qa1.../scenarios" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{"title": "double charge",
         "steps": [
           {"input": "My card was charged twice.",
            "checks": [{"kind": "contains_text", "expected": "ticket"},
                       {"kind": "polite_tone", "required": false}]},
           {"input": "Can you refund it?",
            "checks": [{"kind": "must_use_tool", "expected": "api__refund"}]}
         ]}'

A scenario is appended at the end. You always write a scenario whole: PUT /evaluations/{id}/scenarios/{scenario_id} replaces the title, the description and the entire step list in one call. There is no per-step write, precisely because two people editing one scenario at the same time would otherwise lose each other's changes.

Every step gets an evt_ id from the server. Send that id back when you keep the step in a later PUT, and earlier runs' results stay attached to it. An id you send yourself has to be one the server could have minted (evt_ plus letters and digits) and has to be unique within that scenario; two steps of one scenario sharing an id is 422.

A suite holds at most 32 scenarios, a scenario at most 32 steps, and a step at most 16 checks.

Values for the agent's prompt¶

When an agent pins a prompt that declares variables, the scenario supplies them. variables is a name-to-value mapping on the scenario, and every step of that scenario sends the same values:

curl -s -X POST "$CORE_URL/evaluations/evl_7Qa1.../scenarios" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{"title": "an angry regular",
         "variables": {"customer_name": "A. Bakker", "tier": "gold"},
         "steps": [{"input": "My card was charged twice.", "checks": []}]}'

They sit on the scenario rather than on a step because one conversation is one persona: the caller does not change halfway through.

A suite measures several agents, and you write one scenario for all of them, so a name an agent's prompt does not declare is simply left out of that agent's turns instead of failing them — write customer_name for the agent that asks for it and tier for the one that does, in one scenario. An agent that pins no prompt at all is sent no values whatever the scenario says.

Two things do fail: a required variable with no value, and a value that does not fit the type or the enum the agent's prompt declares for it, such as the text "3" for a number. Either way that agent's every step comes back as a failed step whose reason names the variable. Neither is checked when you write the scenario, because the agents a suite measures can be linked after it. A variable with a default is not required: leave it out of the scenario and Core fills in the default at every step.

Names must look like prompt placeholders ([A-Za-z_][A-Za-z0-9_]*), at most 32 of them. A value is a JSON string, number or boolean -- send it as the type the prompt's input schema declares -- and a string is at most 8192 characters; anything else is 422 on variables. A PUT replaces the whole mapping, exactly as it replaces the steps.

tool_variables on the scenario works the same way for the measured agent's tools: every step sends those values as the request's tool_variables, and a name no tool of a step's turn declares is ignored there. Every reader of the suite sees them, so they are never a secret: a value for a name a tool of your organisation declares secret is refused 422 on tool_variables.<name>.

DELETE /evaluations/{id}/scenarios/{scenario_id} removes one. The order of the others does not shift: a run that already started carries its own snapshot.

Versions¶

Every change to what a suite measures -- a scenario written or removed, its agents or its judge model changed -- is a numbered version, published at once when a key makes it. You can also write the next version whole:

curl -s -X POST "$CORE_URL/evaluations/evl_7Qa1.../versions" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{"agents": ["agt_5Nj2..."], "scenarios": [{"id": "evs_2Kd9...", "title": "Returns",
         "steps": [{"input": "Can I return this?"}]}]}'

It sends every scenario: one with an id keeps it, one without is new, and one left out is removed. GET /evaluations/{id}/versions lists the versions, oldest first, and GET /evaluations/{id}/versions/{version} reads one. A key only ever sees published versions: a person can prepare a draft of a suite in the console, and a suite that is only a draft is not there for a key at all.

The checks¶

Every check carries required, true by default. A check refuses a field its kind does not use: write max_seconds on a contains_text and you get 422 rather than a rule that quietly never fires.

The four judged kinds are one model call per check. For llm_judge, your expected is the rule being tested, not a model answer: write "the answer offers to escalate", not the answer you expect. The judge sees the step's message, the agent's final text and that rule — nothing else. It returns a score from 0 to 1 and one sentence of reasoning; the check holds from threshold, 0.7 by default.

matches_pattern is the pattern a live check holds a real turn to, so a suite can measure the rule before the agent enforces it. A match is the failure. max_match, 1 to 200, is the longest match the pattern can have: a pattern whose match could be longer, such as .*, or that can match nothing, is 422.

must_use_tool reads the turn's own tool calls, not its text. A model that says it will search has not searched. tool_found tells a model that never found a tool from one that found it and did not use it; a turn that ran no search fails it (An agent with many tools).

Starting a run¶

curl -s -X POST "$CORE_URL/evaluations/evl_7Qa1.../runs" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{}'

This answers 202 with one run per linked agent. Send {"agent_id": "agt_5Nj2..."} to start one for that agent, which has to be linked to the suite. A suite with no scenarios is 409.

Nothing has run yet: each run is queued work, picked up by the worker. Each run pins the version the agent had at that moment and snapshots the scenarios as they read then, so a later edit no longer changes what that run measured.

Or let a clock start it. A schedule of kind: "evaluation" names this suite and queues a run on its own cron -- nightly, weekly, whatever you write -- as the key the suite nominates, skipping a night on which a run is still going. What such a schedule has done is this suite's own run list narrowed to it:

curl -s "$CORE_URL/evaluations/evl_7Qa1.../runs?schedule_id=sch_3Qp8..." \
    -H "Authorization: Bearer $CORE_KEY"

Every run says schedule_id -- null for the ones you asked for by hand -- and fired_for, the moment that was due.

A run for one person¶

A run can act for one person, so each step is the turn your agent would make for her: what it may search, which tools and which model it may use, and what it remembers about her, all decided by your rules for her exactly as on a live turn. Name her with X-Core-Subject on the request that starts the run, the way you name her on a turn (build for people):

curl -s -X POST "$CORE_URL/evaluations/evl_7Qa1.../runs" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -H 'X-Core-Subject: {"id": "anna", "attributes": {"department": "payroll"}}' \
    -d '{}'

Every run that request starts acts for her, and says so in subject_id. The attributes you assert apply to every step until the run ends, and are kept nowhere after that. The steps' conversations are hers, so they show up among her threads and responses, and leave with her when you erase her; the run itself stays and stops naming her. The checks are still judged as your organisation's measurement, not as her turn.

Two refusals are particular to it: a suite that runs under a key other than yours (run_as_key_id) answers 422 on header.X-Core-Subject, because a run acts for a person only as the key that named her; and an agent your rules do not let her use answers 422 on agents or agent_id before anything runs. A run started without the header acts for nobody, as before, and so does every run the management surface, a schedule or an improvement trial starts. A key whose policy requires a subject has to name one here too, as on every other evaluations call.

Stop a run with POST /evaluations/{id}/runs/{run_id}/cancel. A queued run stops at once; a running one stops at its next step, and the step that was in flight is not stored. A run that has already finished is 409.

Reading a run¶

curl -s "$CORE_URL/evaluations/evl_7Qa1.../runs/evr_3Kd8..." \
    -H "Authorization: Bearer $CORE_KEY"

Ask again to see progress: after every step, completed_steps, score and that step's result are in there.

If your organisation has set a retention window, a finished run older than the window keeps its status, score, cost and times, while scenarios, results and thread_ids come back empty with expired_at set. The threads they named are deleted by the same window. A comparison of such runs still gives the totals, but no steps to hold against each other.

A step passes when every required check holds. The run is failed as soon as a step carrying a required check did not pass, warnings when only optional checks did not hold, and passed when everything held. errored and cancelled say nothing about the agent: the run itself never got to an opinion.

A check has three outcomes. errored means the judge could not answer — a timeout, an unreachable model, an answer that is not the schema — and counts as "did not pass". That keeps the difference visible between "the agent got it wrong" and "we do not know".

Finding the turn behind a step¶

Every step is an ordinary turn: it sits on a thread, is readable through GET /responses/{response_id}, appears in your audit trail and counts towards your usage. results gives the response_id per step, and thread_ids the thread per scenario. Judged checks count as model_tokens, like any other model call.

Errors¶

The judged checks run under that same key: their tokens land on the same bill the turns do. If that key may not use the judge model, every judged check is errored with the policy's reason — never quietly passed.