Insight¶

Core records what happened on every request. Four parts show it, each for a different question: Usage says how much was spent and what it cost, caps bound it, Audit says who did what and what was refused, and Traces show exactly what went to a model and what came back. Improvements reads that use back and proposes what could be better about your agents and skills.

When to use this¶

The request id Core returns with every answer ties them together: with that id you find the audit events, the usage and the trace of the same request.

Usage¶

Core meters what each request used: model tokens, embedding tokens, tool calls, retrievals, processed and crawled pages, images and the time a computer stood ready. Every record names the key, and the person when the request named a subject. Work that starts in the console is recorded under the administrator.

Usage is converted to euro with the price that applies to the organisation (Models and routing). Storage is a meter: you see how many bytes are stored, and that does not count against a cap.

Caps¶

A monthly cost cap is an amount in euro per calendar month. There are two:

• on an organisation, over all of its keys together. The administrator of the organisation above sets it; an organisation does not set its own ceiling.
• on a key. The organisation's cap always applies beside it.

When a cap is reached, Core answers a new turn with 429. A turn that was already running finishes. A background run that reaches the cap ends failed with the code usage_cap_exceeded and is not resumed. From the start of the next month Core accepts new turns again. You can get a notification by mail before the cap is reached (Notification mail).

Audit¶

The audit trail has two parts:

• Management: changes to the organisation and its keys, such as creating, renaming, suspending, registering and rotating. This is kept after an organisation is deleted.
• Calls: what requests did: responses, tool executions, prompts and skills used, files created. Every policy check produces a policy.decision event, also when the request was refused, with the rule that decided.

Next to these, a product built on Core can deliver its own events to the trail: what happened in the product that Core cannot see, such as a role assigned in it. Every event says its origin: core, or product with the product's namespace and name, so you read everything one person did in one place. As the product, deliver only what Core does not already record itself (Deliver your product's own events).

The audit trail holds no conversation content and no credentials. It says that something happened, by whom and with what outcome. Through the API, a key reads only its own call events.

Traces¶

A trace is the record of one model call: the request Core sent to the provider, the answer, and a readable rendering of both. One turn can make several model calls, for example during tool rounds; each call has a trace of its own, with the tools that ran after it.

Traces hold the full input and output. That is why they have rules of their own:

• Recording is on or off per organisation (Record model calls).
• Traces fall under the organisation's retention period and can get a shorter period of their own.
• Reading a trace's document itself produces an audit event (trace.read).

Improvements¶

An improvement analysis reads how your organisation's agents and skills are used and proposes concrete changes to them. It runs on a clock you choose, looks at the turns, tool calls and traces of a recent period, and writes proposals. A proposal is one finding: what stood out, the figures behind it, a short explanation and, where Core can make the change itself, a drafted change you apply in one step.

Eleven checks in all, each looking for something different. Four read conversations:

Two more checks read memory instead of conversations: they promote a fact several people hold to a shared memory record, and learn a skill from what several people's sessions share. Five newer checks are off until you switch them on (the newer checks):

A leaner agent is tried before it is proposed. A trial runs the agent's own evaluation twice, once on the published version and once on a draft holding the change, and the proposal is written only when the changed version did not do worse. A trial uses the agent's real tools: a tool that sends, writes or books does so for real while it runs. It runs as a key you choose, whose monthly cap is the trials' budget (Trials).

A proposal is open while nobody has decided on it. You can accept it (you think it is a good idea, but it is not carried out yet), dismiss it (it stays away for a while unless the problem doubles) or apply it. When a later analysis no longer finds the problem, the proposal becomes resolved.

Applying always creates something new. An analysis writes proposals. Applying one is done by a person in the console, by an application with a key, or by the analysis itself for the kinds of proposal your organisation chose (Apply without review). Core then creates a new version of the agent, the prompt or the skill, or a new agent, through the same route and under the same checks as when you create it yourself. Nothing is overwritten: whoever pins a version keeps it. Only a request for a new connector is not carried out by Core: someone has to build that connector.

Everything a model reads is pseudonymised. For the checks that read conversations (the four above, A procedure several agents repeat and An evaluation drawn from real conversations) a model reads a sample of conversations. Before it sees anything, Core replaces personal data with placeholders such as <PERSON_1>. That always happens, whatever the keys that held the conversations have set; you only choose which kinds of data are replaced and how strictly. The placeholders stay in the proposed text, and Core does not apply a change that still holds one: you replace it first.

For Always the same first call no model reads anything: Core compares the tool calls itself. The proposal then quotes the tool's result word for word, but only when that result was the same in the conversations of at least three different people. That keeps what belongs to one person out of a proposal.

Conversations can only be read while traces are on. With traces off, three checks produce figures only, and three are skipped altogether and write nothing: Always the same first call, A procedure several agents repeat and An evaluation drawn from real conversations.

Analyses have a model of their own. They use the analysis model, separate from the helper model that writes drafts in the console: an analysis reasons over dozens of conversations and needs a stronger model. If you choose none, the deployment's default applies. If there is none either, the three checks that need a model produce figures only, with no explanation and no change; Always the same first call works without a model and still drafts a change. Core never falls back to the helper model.

Switching on is consent, and it costs money. Analyses are off by default. Whoever switches them on consents to the analysis model reading the organisation's conversations in pseudonymised form. Every analysis makes model calls, up to a maximum you set, and they count towards the monthly cap: once it is reached, the analysis stops. What Core sent to the model is kept as a trace of your organisation.

Apply without review¶

An organisation chooses, per kind of proposal, how far an analysis applies those proposals itself, with nobody reading them first: propose (the default: the analysis only proposes), draft (it saves the change as a draft, which nobody's key uses until an administrator publishes it), or publish (it saves the draft and publishes it at once). Only a shared memory record goes straight from propose to publish, with no draft. The table says when each kind shows:

Some kinds are only ever proposed, whatever you choose: a connector request (there is nothing to apply), a skill retirement, a new evaluation, scenarios for an evaluation and a leaner agent.

The analysis applies as the organisation. No key and no person is involved. The analysis may do what the organisation's capabilities allow, and every rule in the policy of the organisation and of the organisations above it applies. Because nobody read the change, the whole policy applies, also where a person in the console is held only to its residency choices.

A refusal leaves the proposal open. When the analysis cannot apply a proposal, for example because the text still holds a placeholder, the agent has a newer version by now or a policy rule does not allow it, nothing changes. The proposal stays open with the reason attached, and a person decides on it. An analysis never applies a proposal someone has already accepted or dismissed.

What an analysis applied can be recognised. Such a proposal is applied, marked as done by the analysis, and the audit trail records it as it records a person's apply. Every analysis counts, per check, how many proposals it applied and how many it left open.

Undoing works as for any new version: pin the old version, or delete what was created. Nothing was overwritten.

You choose the kinds in the console or through the API.

Looking at a customer organisation from the organisation above¶

As a partner you want to see your customers' usage without signing in at each customer separately.

• In the console you open a customer organisation and read its Usage and Audit. On your own organisation you choose Including those below under Scope to add up the whole branch, with the cost per organisation. Where a markup is in force, Charged and Paid sit side by side.
• Through the API a key with management reads the usage and the audit events of an organisation below it at /organisations/{id}/usage and /organisations/{id}/audit/events.
• By reserving you keep the usage figures or the audit trail of calls to yourself: the customer's own administrators then do not see them. A key of the customer that reads its own data with GET /usage, GET /audit/events or GET /key notices nothing. A key of the customer with management does get 403 on the usage and the audit events of its own organisation under /organisations/{id}/..., and with key management reserved on changing keys.

Caps belong to one organisation. A parent organisation's cap is not the sum of the organisations beneath it.

How it fits together¶

• Usage and audit are attributed to a key and, when the request names one, to a person.
• Every policy decision is in the audit trail.
• A trace shows which tools a turn ran and which model answered, also after a fallback or a router's choice.
• An evaluation measures an agent beforehand; usage, audit and traces show what happened afterwards in real use.
• An improvement analysis reads the same turns and traces back and proposes a new version of an agent; an evaluation measures whether that version works better.

Read on¶

Tutorial: Build an agent that combines knowledge and tools.