Improvements in the console¶

The concept behind improvement analyses is in Insight.

On Improvements you read the proposals improvement analyses write about your agents and skills, and you decide on them: apply, accept or dismiss. You switch the analyses on and set them up on Settings, the Improvements tab; there you also choose which kinds of proposal an analysis applies itself. You do the same things with a key in Improvements through the API.

You need an administrator account of the organisation. Two more things must be on:

• The deployment runs improvement analyses. If it does not, the screen and the tab say This deployment does not run improvement analyses. Ask your operator to switch them on.
• The organisation may use the improvements capability. Without it, Improvements is not in the sidebar. The administrator of the organisation above, or your operator, switches it on under Permitted capabilities.

Switch analyses on¶

1. Open the organisation's Settings and choose the Improvements tab.
2. Switch on Improvement analyses. The card says what that means: Switching on is your consent: the analysis model reads pseudonymised conversations, tool calls and traces of this organisation. Every analysis costs model calls, up to the maximum below, and counts towards the spending cap. Below it: Analyses change nothing themselves, unless the organisation chose a level for a kind below under Apply without review.
3. Choose the clock, the checks, the model, the pseudonymisation and what an analysis may apply itself on the cards below it (below).
4. Choose Save in the bar at the bottom.

The first analysis runs at the clock's next moment. If you do not want to wait, choose Analyse now on Improvements.

Conversations can only be read while traces are on for the organisation (Traces). With traces off, three checks calculate the figures only, and Always the same first call, A procedure several agents repeat and An evaluation drawn from real conversations are skipped altogether and write nothing. The other three of the five newer checks, A skill nobody uses, A procedure inside the instructions and A leaner agent, read no traces and run with traces off.

If Read only shows at the top, the organisation above keeps key management to itself (What the organisation above reserves). You can see the settings, but only an administrator of the organisation above changes them. On a phone you switch analyses on or off and choose the clock; thresholds, the cron pattern and pseudonymisation are set on a larger screen.

Clock and period¶

On the When and about what card, Clock sets when an analysis runs: Every night 03:00, Every weekday 03:00, Every Monday 03:00, or Advanced for your own Cron pattern (five fields: minute, hour, day, month, weekday) and Time zone. The default is every night at 03:00 in the organisation's time zone. Below the clock you see when the next analysis runs.

Period is how many days an analysis reads back, from 1 to 30; the default is 7. An analysis never reads further back than the organisation keeps traces.

Checks¶

On the Checks card you switch each check on or off and set up to three numbers:

• Threshold: the value from which a finding becomes a proposal.
• Minimum: how many turns an agent or skill needs in the period before it counts.
• Sample: how many conversations an analysis reads per candidate, at most 50.

A check shows only the numbers that mean something for it. The five newer checks are off until you switch them on.

What the five newer checks do, and what they propose, is in the new checks. A last row, Repeated scripts, is greyed out and cannot be set.

Trials¶

The Trials card names the key a trial runs as. A trial is how A leaner agent finds out whether a change to an agent is safe to propose (Trials).

• Trial key is any key of the organisation, or None: no trial starts, which is what it says until you choose. Each key shows its monthly cap, or that it has none.
• The key's monthly cap is the trials' budget, the grading included. A key with no monthly cap starts no trial: the card then says This key has no monthly cap, so no trial starts. Give the key a cap: that amount is the trials' budget. Raise the cap on the key's page; View the key opens it. If the key you chose is gone, disabled or expired, the card says no trial starts until you choose another.
• Changing it needs management as well as improvements, because it chooses whose money trials spend. If the organisation above keeps key management to itself, the card is read only like the rest of the tab.

A trial uses the agent's real tools. The card says so, and so does the A leaner agent row on the Checks card: A trial run calls the same tools a real conversation calls, so a tool that sends, writes or books does so for real during a trial. Before you switch the check on, look at what the agents it may try can do.

Model and cost¶

Under Analysis model you choose which model reads the conversations:

• The platform's default follows the model your deployment has for analyses. The option names that model, or says there is none.
• Own choice opens a list of this organisation's chat models.

Core never falls back to the helper model. Without an analysis model the three checks that need one produce figures only, with no explanation and no proposed change. Always the same first call needs no model and still drafts its change.

Max. model calls per analysis bounds what one analysis may spend; the default is 40. Once the maximum is reached, whatever was not looked at comes first in the next analysis. A dismissed proposal stays quiet sets how many days a dismissed proposal stays away unless the problem doubles in that time; the default is 30.

At the bottom of the tab you see how many analyses ran last month and how many model calls they made. In Usage opens Usage, where the cost is.

Pseudonymisation¶

Everything a model reads is pseudonymised; that cannot be switched off. On the Pseudonymisation card you only choose how:

• Language: the language the conversations are read in.
• Confidence: how sure the detection has to be. Lower finds more, but also makes ordinary words unreadable.
• Kinds of data: what is replaced, such as Person, Email, Phone, IBAN and BSN. Choose at least one.

Data found becomes placeholders such as <PERSON_1>. If the service that does this cannot be reached, Core reads no conversations and that check does not run.

Always the same first call sends nothing to a model: Core compares the tool calls itself, and the proposal quotes the tool's result word for word. It does so only when that result was the same in the conversations of at least three different people.

Apply without review¶

On the Apply without review card you choose, per kind of proposal, how far an analysis goes by itself, with nobody reading the proposal first. What that means is in Insight. The card says it briefly: Per kind of proposal you choose how far an analysis goes by itself: only propose, make a draft a person publishes, or apply and publish. What it cannot apply stays open for a person. It can be undone: pin the old version or delete what was created.

Each kind has a choice of its own, and every choice starts on Only propose:

Choose Save in the bar at the bottom. What the analyses applied afterwards is read as described in What an analysis applied itself.

Read the proposals¶

Open Improvements in the sidebar, in the Insight group. Beside the name you see how many proposals are open. At the top are the last analysis and when the next one runs, with the Analyse now button. The Proposals and Analyses tabs divide the screen.

Narrow the list with three rows of choices:

• Status: All, Open, Accepted, Applied, Dismissed or Resolved.
• Check: one of the checks.
• Target: Agent, Prompt, Skill or Evaluation.

The filters are part of the address, so a link you share opens the same list. Each row names the proposal, its target, the Key figure (for example 82% of the conversations), the status and when an analysis last found the proposal. If an analysis applied the proposal itself, Applied automatically stands beside the status.

When the list is empty, the screen says why:

A proposal¶

Choose a proposal to open it. The page has these parts:

• Finding: what the check noticed.
• Explanation, by the analysis model: why, based on the conversations it read.
• Proposed change: what changes, per part (instructions, tools, skills or template) with the lines added and removed. A proposal for a new agent shows Proposed agent with a name, description and instructions. A rewritten skill shows Rewritten skill with the whole new SKILL.md and a Copy button.
• Figures: the share, for a skill the share Without the skill, and for a repeated first call how many Calls and Tokens a month you save. The newer checks add the Period, the Conversations and the People a finding rests on, the Tool sequence several agents repeat, and, for a leaner agent, the Steps passed, the Cost of the run and the Mean response time of both runs, and the Estimated saving a month.
• Example conversations: the turns the finding rests on, at most twenty, each with its time and its response id. The response id is a link that opens that turn's model calls on Traces (Read a trace). The proposal itself keeps no text from those conversations.
• History: in which analysis the proposal was found and refreshed, and who decided on it; for a proposal an analysis applied itself it says the analysis.

The newer checks add proposals of their own (the new checks): Proposed skill for a skill an analysis wrote, with the whole SKILL.md; an agent version that loads a skill or replaces the instructions whole (the part instructions (replaced)); Proposed evaluation or Scenarios to add, with each drafted scenario, its steps and the checks it holds the answer to; and Leaner agent, with the model, the tools the agent stops offering and the shorter instructions, whichever the trial changed. A skill that no agent loads and no request named has a proposal that carries figures only: you retire a skill yourself.

A proposal from one of the three model checks with no explanation and no change was made while there was no analysis model: No proposed change. It has the figures only; Set a model takes you to the model.

Above the change a message may say why you cannot apply yet:

The newer checks¶

All five are off until you switch them on (Checks).

An analysis never writes a skill that names a person: a draft that holds a name, an email address, a phone number, a link or an id is not proposed. The drafted scenarios of An evaluation drawn from real conversations are rewritten with invented details, name no person and no conversation, and are the organisation's own test data: they are not erased with a person. A skill an analysis wrote from the instructions replaces nothing until you apply it, and an agent version that replaces the instructions whole is never applied without review: at Apply and publish it waits for you.

Trials¶

A trial tries one change at a time, in this order: fewer tools (those no turn used in at least 14 days), then a cheaper model (at least 20% cheaper at the agent's own mix of input and output, one your policy permits and no person's rules refuse), then shorter instructions (at least a quarter shorter, drafted by the analysis model). It does this as two runs of the agent's own evaluation, with the same scenarios and the same judge: one on the agent's published version, one on a draft that holds the change. Nothing a trial writes reaches a key: the draft is not published, and a draft of a trial that proposes nothing is discarded.

A trial can take longer than an analysis, so the next analysis reads its result. It writes a Leaner agent proposal only when the changed version's pass score is at least the published version's, no step the published version passed failed, and its cost or its response time is at least 10% lower. Otherwise the trial ends without a proposal. At most three trials start per analysis, and one per agent at a time. An agent is tried once per published version.

A trial uses the agent's real tools. A trial run calls the same tools a real conversation calls, so a tool that sends, writes or books does so for real while the trial runs. Both runs do it. The proposal and the analysis say so again.

What a trial cannot show. Every Leaner agent proposal carries this sentence with the number of scenarios: A trial shows one thing: on these … scenarios, graded by this judge, the leaner agent did not do worse. It cannot show how the agent behaves on topics the evaluation lacks, whether a tool needed once a quarter is still needed, whether the tone drifted in a way the judge does not check, or that the result would repeat: each side is one run. That is why the proposal is only proposed, never applied without review.

What a trial costs. The turns and the judge's calls are billed to the trial key, so its monthly cap bounds them, inside the organisation's own cap. Before it starts a trial Core estimates the cost as twice what the agent's last finished run of that evaluation cost, the judge included; when that is more than what is left of the month, the trial is not started (Skipped: this month's trial budget is used up).

An agent with no evaluation of at least five scenarios is not tried at all. The analysis page points at the proposals of An evaluation drawn from real conversations, which drafts one.

Apply¶

Applying creates a new version of the agent, the prompt or the skill, or a new agent. The version the proposal is about stays, and whoever pins that version keeps it. For an agent, a prompt or a new agent the result is a draft: no key sees or runs it until you publish it (Applied as a draft).

1. Choose Apply, or Create agent for a proposal for a new agent. Edit on the change's card opens the same dialog.
2. The dialog names the new version (for example New version 4 of support-agent), what changes and what happens afterwards. For an agent or a prompt it says the version comes ready as a draft: no key runs it until you publish it, and calls without a pinned version stay on the version they use now. For a skill the dialog says: v… becomes the latest version. Agent versions keep the version they pin; a request that names the skill without a version uses v… at once.
3. Adjust the text if needed. The field is To add to the instructions, for a prompt To add to the template, for a new agent Instructions and for a skill SKILL.md of the new version. What you change here goes into the new version; the proposal keeps the original text. Replace every placeholder: while one remains, you cannot confirm.
4. Choose Create draft v… or Create draft of the agent.

Other kinds are applied the same way, with their own words in the dialog. New skill creates draft v1 of the skill. Proposed evaluation creates draft v1 of a new evaluation linked to the agent, and Scenarios to add creates the evaluation's next draft beside its published scenarios; neither has text to edit, and nothing is measured against a draft until you publish it, on the evaluation's page or from the proposal (Applied as a draft).

A leaner agent is the exception: applying it publishes. The dialog is Publish a leaner … and its button Publish version …. The version the trial measured becomes the latest at once, and every call without a pinned version runs it. A trial measured the evaluation, not every caller of the agent, so read the sentence on what a trial cannot show first. If the draft was discarded, Core writes the change again and publishes that. Edit changes the shorter instructions before they are published.

The console says what was created, the proposal is Applied, and View opens the new version or agent. Editing the text is done on a larger screen.

Applied as a draft¶

On a proposal that was applied as a draft, a message above the finding says This is a draft, not published yet, with View draft, which opens the agent's or prompt's page on that version (an evaluation's page for a drafted evaluation), and Publish. For a skill, View draft opens the skill's page, where its versions show the draft. Publish asks for a confirmation and makes the draft the latest version; if Core refuses, the confirmation shows its sentence. Once the draft is published, or discarded on the agent's, prompt's or skill's own page, the message says so instead.

If Core refuses, the dialog shows the reason as Core gave it. The most common: the target has a newer version by now (the next analysis refreshes the proposal), a placeholder is still in the text, or a rule of your organisation does not allow the change.

A new connector, or a change you made yourself¶

Core does not carry out a Request for a connector itself: give the description to whoever builds your API connectors or MCP servers. Once the connector exists, a later analysis proposes adding it to the agent.

Once you have made the change yourself, for example built a connector or put a rewritten skill into the skill yourself with Copy (Skills in the console), choose Mark as applied; you can add a note, for example where you made the change. Through the API that is PATCH .../improvements/proposals/{id} with "status": "applied" (Decide on a proposal).

Accept, dismiss and reopen¶

Every button opens a dialog first; nothing changes on the first click. You can add a Note of at most 1,000 characters, which goes into the proposal's history.

• Accept says you think it is a good idea, but it is not carried out yet. The proposal stays until someone applies it or the problem goes away.
• Dismiss keeps the proposal away for the days set in A dismissed proposal stays quiet, unless the problem doubles in that time.
• Reopen opens a dismissed proposal again. If a later analysis has meanwhile written the same finding as a new proposal, Core refuses to reopen; work with that new proposal instead.

On a phone the buttons are at the foot of the page.

What an analysis applied itself¶

For the kinds you set to Make a draft or Apply and publish under Apply without review, an analysis applies every proposal it writes or refreshes itself, straight away. A proposal someone has already accepted or dismissed is left alone.

Recognising it. A proposal an analysis applied is Applied with Applied automatically beside it, in the list and at the top of the proposal. The History says the analysis where it would otherwise name a person, and View opens what was created.

Refused. If the analysis could not apply a proposal, nothing changed and the proposal stays open. At the top of the proposal is Could not be applied automatically, with the reason:

Any other reason, such as a definition the target's route refused, the console shows as Reason: with the code Core gave. When a later analysis finds the same finding, it tries the proposal again.

Counting. On an analysis's page each check that ran also shows Applied automatically and Not applied, left open, with 0 when no kind is chosen. They are absent for Failed, Interrupted and Skipped · tracing off, and on older analyses (Analyses).

Undoing. Applying overwrites nothing, so you undo it as you would any new version. A draft of an agent or prompt that is not published yet: discard it on its page. A new version of an agent or prompt that is published: pin the previous version where it is used (Agents in the console, Prompts in the console). A new agent: delete it. A new skill version changes no agent as long as no agent version pins it; a request that names the skill itself goes back to the previous version by naming it, as name@N.

Analyse now¶

Analyse now queues an analysis at once, beside the clock's analyses. The button then shows Queued and Analysis running. Core refuses a new analysis while analyses are off or one is already queued or running; a message then gives the reason.

Analyses¶

The Analyses tab shows every analysis, newest first: when it started, whether the clock (schedule) or someone (manual) started it, how long it took, how many proposals it wrote or refreshed, how many model calls it made, and its status.

Choose an analysis for the details: when it started, how long it took, the Period read, the model calls and the proposals. Below that each check has its outcome with the counts Candidates, Conversations read, New proposals and Refreshed, and for a check that ran also Applied automatically and Not applied, left open:

When a check left findings out because applying them would be refused (by policy, a setting, or because the target changed or is gone), a line under the counts shows Not proposed with the total, then each reason and its number. Change the policy or setting the reason names; the next analysis proposes them again.

Under the counts a check may list What else happened: each count Core wrote, with its number. For A leaner agent these say why an agent was not tried (Skipped: no judge model on the evaluation, no finished run yet to estimate the cost, no trial key set, the trial key has no monthly cap, the trial key was deleted, disabled or expired, this month's trial budget is used up, left for a later analysis: too many trials in one analysis, too few conversations, the period is too short), why it got no cheaper or shorter version (every tool was used, the agent names no model, the agent uses a router, no cheaper model that qualifies), and how trials ended (Trials started, Trials that held, proposed, Trials that did not hold, Trials still running, and a trial dropped because the agent has a newer version, a run failed, or the agent was deleted). For the checks that draft text it says what was left out: a scenario that names a person or something real, or sits too close to the original message, and No proposal: too few scenarios or people left. A count the console has no words for shows under Core's own name. The page of A leaner agent also says again that a trial uses the agent's real tools.