Model routers¶
The concept behind models and routing is explained in Models and routing.
Use Routers to let Core choose the model for a turn. A caller names a router where it would name a model, and Core picks one model from the router's pool for that turn.
A router has an id that always begins with auto-, for example auto-cheap. Core reads the
last message of the turn, decides what kind of task it is and how hard, and runs the turn on the
first model in the router's table for that pair that the calling key may use. Everything that
holds for a model a caller named holds for the model the router chose: the key's policy, the
hosting requirement, the model's own fallback chain and its price.
A router is the organisation's own. A child organisation inherits none.
You need an administrator account of the organisation. When the organisation above reserves key
management, you can read routers and not change them (below). A key that holds
management does the same through the API, on /model-routers for its own organisation and
/organisations/{id}/model-routers for an organisation below it
(API reference).
Create a router¶
- Select New router and enter the Id.
auto-is fixed in front of it; use lower-case letters, digits, dots, underscores and hyphens. The id cannot change later. - Under Pool, add one to sixteen chat models, in order. Each must be a model the organisation may use.
- Under Default list, add the models a turn tries when its task type cannot be trusted. They come from the pool.
- Optionally fill the Table per task type (below).
- Review the thresholds, and leave Live off until the router has been measured.
- Save the router.
Core validates the whole router when you save it. An error names the list it concerns and the model it broke on, and shows under that list; a second router with an id the organisation already has is refused with a message above the form.
Order is the meaning of every list: a turn walks it from the top and uses the first model that survives the key's policy, the hosting requirement and the router's price ceiling. Use the arrows to change it.
The table per task type¶
The table has a row per task type — general, code, summarisation, extraction, reasoning, data analysis and acknowledgement — and a column for easy turns and one for hard turns. Each cell is an ordered list drawn from the pool. A cell left empty uses the default list. An acknowledgement ("thanks", "ok") is never hard, so it has an easy cell only.
Thresholds¶
| Field | What it decides | Default |
|---|---|---|
| Task type confidence | From this probability the task type is trusted; below it the turn uses the default list. | 0.7 |
| Easy-turn confidence | From this probability that a turn is easy it counts as easy; below it, hard. | 0.8 |
| Similarity to known examples | Below this similarity to the nearest example a turn counts as unknown and uses the default list. Empty switches the test off. | off |
| Maximum output price | A pool model that costs more, or that no price covers, is dropped. Empty is no ceiling. | none |
| Difficulty follows the conversation | A turn that continues a hard turn stays hard until an acknowledgement. | on |
The defaults come from the offline measurement the router was designed on; they are not tuned to your traffic. Measure a router before relying on them, as below.
Live¶
A router that is not live serves evaluations only. Ordinary turns that name it get the unknown-model error, and it is not in the list of models. Switch Live on when you are satisfied with the comparison. Saving a change to the pool, the lists, the table or a threshold writes a new version; the Live switch and the description do not.
Choose a router for an agent or a schedule¶
An agent version's model and a schedule's model offer your routers at the top of the model picker, marked not live while they are not. They are offered in no other model field: a judge, the helper model, summarising, a routing chain, a pipeline step and an index each name one model call, and Core refuses a router there.
See how a turn was routed¶
When a turn named a router, the console shows how it was routed wherever it shows that turn: on a call's page in Traces, on the turn's message in a conversation, and under a step of an evaluation run. The block lists the router and version, the task type with its confidence, the difficulty, why Core decided as it did, the list the model came from, the model chosen, and every pool member with Kept or why it was dropped. A turn that named a model has no such block.
Measure a router before it goes live¶
Measuring uses an evaluation: link two agents that differ only in their model, one naming the router and one a fixed model, and set a Judge model on the evaluation so both runs are judged alike. Run it, open one run and choose Compare with the other:
- The run's page shows the mean judged score beside the pass percentage, the mean turn latency, the cost and the tokens, the router version it pinned, and the judge.
- The comparison shows, per step, the model that answered and its score on each side, and the totals: how many steps the first run won, tied and lost, and each run's figures side by side. Two scores within 0.05 of each other count as a tie.
- If the two runs were not judged by one fixed model the comparison says so and the scores are not comparable.
Whether the router is good enough is your decision; Core does not set a threshold.
Delete a router¶
Select Delete in the row menu and confirm. Core does not check what still names the router: an agent version or a schedule that does fails its next turn with the unknown-model error, as for a model removed from the catalogue.
Reserved key management¶
If the parent organisation has reserved key management, routers are read-only here. Ask an operator or a parent organisation administrator to create, change or delete them.