Live checks through the API¶

The concept behind live checks is explained in Live checks.

On this page you give an agent version live checks through the API, read what a stopped turn tells your application, and read the results and pass rates back. A live check is a rule every real turn of the agent is held to: on the caller's question (input) or on the model's answer (output), deciding on the turn (block) or scoring a sample afterwards (background).

Your key needs the agents capability to write the checks. Reading one turn's results needs threads, as reading the turn does; reading an agent's pass rates needs evaluations. You do the same with a mouse in Live checks in the console.

Add checks to an agent version¶

Live checks are part of an agent version, written with the rest of it on POST /agents or POST /agents/{agent_id}/versions:

curl -s -X POST "$CORE_URL/agents/agt_5Nj2.../versions" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{"model": "<a model id>",
         "instructions": "Answer questions about orders in two sentences.",
         "blocked_reply": "I cannot help with that. Never share a password in this conversation.",
         "recover_attempts": 2,
         "live_checks": [
           {"name": "no-passwords", "stage": "input", "kind": "not_contains",
            "expected": "password"},
           {"name": "no-iban", "stage": "output", "kind": "matches_pattern",
            "pattern": "NL\\d{2}[A-Z]{4}\\d{10}", "max_match": 18, "recover": true,
            "description": "Never state a bank account number."},
           {"name": "polite", "stage": "output", "kind": "polite_tone",
            "mode": "background", "sample": 0.2,
            "alert": {"failure_rate": 0.3, "over_last": 50}}
         ]}'

This version refuses a question that mentions a password with the standard reply, never lets an answer carry a Dutch IBAN (and asks the model again when one does), and has a judge score the tone of one turn in five, raising a notification when 30% of the last 50 scores fail.

A version is the whole definition: a version you write without live_checks has no checks, and calls that name no version run it at once: a version written through the API is published. Save it as a draft in the console first when you want to measure it with an evaluation before your callers get it (Drafts).

The version's fields¶

A check's fields¶

The kinds¶

These are the kinds an evaluation uses, so a rule means the same in a suite and on a live turn. For a judged kind, expected is the rule, such as "the question is about our products", not an answer you expect. response_time is refused 422 as not yet supported.

A pattern needs a bound. pattern is RE2 syntax, at most 500 characters, without backreferences or lookaround. It is refused when its longest match could be longer than max_match, as with .* or \w+, or when it can match nothing at all. The bound is what keeps a streamed answer only a few characters behind the model: Core holds back max_match characters before it sends them. NL\d{2}[A-Z]{4}\d{10} matches exactly 18 characters, so max_match: 18 fits.

A judge model has to be a model of your organisation's catalogue, never a router. When the agent's own model is a router, every judged check names a judge_model, because a router picks a different model per turn. An agent without a model of its own, called with a router as the model, needs one too: a judged check without a judge_model then cannot decide (errored), and a blocking one stops the turn with check_errored, unless it says "on_error": "pass". The judge's tokens count as model_tokens on the key that ran the turn.

What your application receives¶

A turn a check stopped keeps the vendor's shape, and the X-Core-Check header says which check stopped it:

X-Core-Check: stage=input, check="no-passwords", code=input_check_failed

The header is a structured dictionary (RFC 8941) with stage, check and code. It names the first blocking check that did not pass, and is absent on every turn no check stopped, a recovered one included. It never carries a pattern, expected text, threshold, score or reason.

Read the refusal part as you read a model's own refusal: show it to the person. On POST /messages a refused question and a withheld answer are both stop_reason: "refusal", with the same header.

A streamed turn. An input check runs before the stream starts, so its header arrives on a streamed turn too. An output check decides after the headers were sent: the stream ends with response.incomplete and content_filter, and GET /responses/{id}/checks names the check. When the agent's blocking output checks are all patterns, the answer streams a few characters behind the model and stops where a match starts; with any other blocking output check, the answer arrives in one piece once it has passed.

A recovered turn. When a check with recover fails the answer, Core asks the model again without showing it the rejected answer, and the response holds only the answer that passed. Each attempt is a model call billed on your key and included in usage; the turn's deadline and your usage cap are checked before each one. Recovery is not attempted on a streamed turn whose answer was already reaching you: there the answer is cut.

A turn Core queues itself. A workflow step or a task started in the background is refused when an input check fails, since no one waits for the standard reply: the workflow step ends policy_refused, and the task is not started. A request you send with background: true is answered with the standard reply at once.

Read the results of one turn¶

curl -s "$CORE_URL/responses/resp_01M3.../checks" \
    -H "Authorization: Bearer $CORE_KEY"

{
  "items": [
    {"object": "response.check", "id": "rck_…", "response_id": "resp_01M3…",
     "name": "no-passwords", "stage": "input", "kind": "not_contains", "mode": "block",
     "attempt": 1, "outcome": "failed", "score": null,
     "reason": "The answer contains text it had to avoid.", "judge_model": null,
     "passed_on_error": false, "created_at": "2026-10-16T09:14:03Z"}
  ],
  "next_cursor": null
}

Every result of every check on the turn, in the order they were decided, paged by cursor (Cursor pagination). outcome is passed, failed or errored; passed_on_error is true on an errored result its check's "on_error": "pass" let through. attempt counts the answers: 1 is the first, 2 the first answer asked for again. reason is a fixed sentence for a rule and for a check that could not decide, and the judge's own sentence for a judged check; it never repeats the text the check read. The fixed sentences are the ones an evaluation uses, so they speak of "the answer" on an input check too. A background check's result appears once the judge has scored it, shortly after the turn.

You read the results of a turn wherever you can read the turn itself with GET /responses/{id}; a turn that was not stored answers 404.

Read an agent's pass rates¶

curl -s "$CORE_URL/agents/agt_5Nj2.../checks/summary?since=2026-10-09T00:00:00Z" \
    -H "Authorization: Bearer $CORE_KEY"

{
  "since": "2026-10-09T00:00:00Z",
  "items": [
    {"name": "no-iban", "passed": 412, "failed": 3, "errored": 0},
    {"name": "no-passwords", "passed": 409, "failed": 6, "errored": 0},
    {"name": "polite", "passed": 79, "failed": 2, "errored": 1}
  ]
}

Per check name, how many results of the agent's turns passed, failed or could not decide since since: seven days back when you leave it out, at most ninety. The counts cover every version of the agent, and never carry a reason or any text.

Alerts¶

A check with alert raises a notification when too many of its latest results did not pass:

"alert": {"failure_rate": 0.3, "over_last": 50}

Once the check has at least over_last results (1 to 1000), and the share of them that failed or could not decide is at or above failure_rate (above 0, at most 1), the organisation's administrators get one Live check failing notification by mail or Slack. It names the agent, the check and the count, never a question or an answer, and comes at most once a day per check. The results of every version of the agent count. An organisation that switched the kind check.failing off is not notified; see Notifications.

Events¶

Every result that did not pass, in either mode, is a response.check_failed event on your event feed and webhooks, with the check's name, stage, kind and outcome and the turn's response_id. Use it to follow failures as they happen rather than reading pass rates.

Errors¶