Models and routing¶

An agent thinks with a model. Core keeps track of which models exist, where they run, what happens when a model does not answer, which model a turn gets when you let Core choose, and what a call costs.

1. The catalogue¶

The catalogue is the list of models configured on your installation. An organisation's Models section shows it, with for each model:

• the id a request puts in the model field;
• the kind: chat for a turn, embedding for making things searchable, image for generating images, or audio for turning a recording into text;
• the lab that developed the model, the provider that runs it, and the region it runs in;
• the price that applies to this organisation.

The catalogue is read-only: adding or changing a model is your operator's job. What an organisation sets itself are labels on a model, such as tier: premium.

The catalogue says what exists. What a key may use is decided by its policy with rules about Models, by name or by label. GET /models returns the models a key reaches.

Your operator can also retire a model: from a chosen date, another catalogue model, the replacement, takes its place. A turn naming the model after that date runs on the replacement, or is refused where the organisation's or the key's policy says so. What in your organisation still uses the model is on Models under Still in use, and a month before the date you get a notification about it.

When: a vendor stops offering a model, and you want to know what to move and by when.

2. Regions¶

Every model has a region: eu, us, global, in-cluster (the installation's own infrastructure) or a country code such as nl. Regions nest: nl and in-cluster lie in eu, and everything lies in global.

With Allowed regions in the policy of a key or an organisation you keep calls inside those regions. That covers the model that answers and also embedding models, the model steps of a knowledge pipeline, tools and computers. A destination outside the list is refused, unless the policy names it as an exception. A request can also demand a region itself with the hosting field.

When: a customer requires that data does not leave the EU, or the installation itself.

3. Fallback¶

A routing configuration fixes which model a turn falls back to when the requested model cannot answer: a timeout, overload or a server error at the provider. A configuration is a list of chains; each chain names one model and behind it, in order, one to four alternatives.

You make a configuration the organisation's default, or choose one on a single key. An agent version, or a single request, can name its own, which takes precedence over the key's, so one key can serve several assistants that each fall back along their own chains. The answer's model names the model that really answered, and the usage is attributed to that model. An alternative has to pass the policy and the region requirement itself: a chain grants no extra rights.

When: your application has to keep answering when one provider has an outage.

4. Routers¶

A router chooses the model for you. A request, agent or schedule names the router where it would otherwise name a model; the id always starts with auto-, for example auto-cheap. Core reads the turn's last message, determines the task type (general, code, summarisation, extraction, reasoning, data analysis or acknowledgement) and whether the turn is easy or hard, and uses the first model in the list for that combination that the key may use.

You put a router together yourself: a pool of models, a default list, and per task type a list for easy and for hard turns. Everything that holds for a model you chose yourself also holds for the model the router chose: the policy, the region, the fallback chain and the price.

A router only serves ordinary turns once you switch Live on. Measure it first with an evaluation: two agents that differ only in their model, one with the router and one with a fixed model.

When: you want simple questions to run on a cheaper model and hard ones on a stronger model, without your application making that choice.

Fallback and routers complement each other. A router chooses the model for a turn beforehand; fallback catches it when that model then does not answer.

5. Prices and markups¶

Core converts usage to euro with a price per model, per tool and per kind of usage. Your operator sets prices; an organisation reads its price and does not write it. The price that applies to your organisation is shown on Models.

A price is in the currency the vendor invoices in, dollars for example. Core converts it to euro with the European Central Bank's reference rate. While a month is open, Core uses the newest rate, so that month's cost moves with the rate. Once your operator closes a month, its rates are fixed and its cost no longer changes.

A markup is what your organisation puts on top of that price. You choose who it applies to:

• This organisation: changes what your own summaries, caps and model prices show.
• Everything below it: changes what every organisation beneath yours sees as its own price.

A markup is a percentage or an amount, for all usage or for one kind, model or tool. That is how you decide what you charge a customer. The customer sees only the raised price. In the usage of the whole branch you see Charged and Paid side by side: what the customer is charged, and what you pay for that same usage.

The monthly cost cap of an organisation or a key counts with the same prices, markup included.

What a child organisation does not inherit¶

Routing configurations and routers belong to the organisation itself. A customer organisation does not inherit them and has its own settings. A markup on Everything below it does carry down.

How it fits together¶

• An agent stores a model or a router.
• The policy decides which models and routers a key reaches and in which regions.
• Every model call is metered in tokens and converted with the price (Insight).
• Embedding models belong to knowledge: an index and a graph each name one.

Read on¶

Tutorial: Build your first agent.