Graphs through the API¶

The concept behind knowledge is explained in Knowledge.

A knowledge graph holds the entities your documents are about, such as companies, people, products and contracts, and the relations between them. A schema you write yourself decides which kinds exist. An index answers "which passages resemble this question"; a graph answers "what is connected to this, and how".

Your key needs the knowledge capability. A graph has no capability of its own. Only having a schema suggested needs assist as well. You do the same with a mouse in Graphs in the console.

Schema, facts and sources¶

1. Create a graph¶

curl -s -X POST "$CORE_URL/knowledge/graphs" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{
      "name": "suppliers",
      "description": "Suppliers, their products and the substances in them.",
      "visibility": "shared",
      "schema": {
        "entity_types": [
          {"name": "Company", "description": "A company that supplies or buys.",
           "properties": {"type": "object", "properties": {"country": {"type": "string"}}}},
          {"name": "Product", "description": "A product a company supplies.",
           "properties": {"type": "object", "properties": {"sku": {"type": "string"}}},
           "identity_keys": ["sku"]}
        ],
        "relation_types": [
          {"name": "supplies", "description": "The company supplies the product.",
           "from": ["Company"], "to": ["Product"],
           "properties": {"type": "object", "properties": {"since": {"type": "integer"}}}}
        ]
      }
    }'

The answer is 201 with the graph. Its id starts with kg_; the examples below use $GRAPH_ID for it.

GET /knowledge/graphs returns the graphs your key reaches, GET /knowledge/graphs/{id} one graph. PATCH changes description, labels, collection_ids, ingest_pipeline and embedding_model; not the name or the visibility. DELETE removes the graph with all its schema versions and facts. The collections and their sources stay.

2. The schema¶

An entity type has:

• name: unique within the schema, such as Company. Starts with an upper-case letter, then letters, digits and _, at most 64 characters;
• description: what belongs under the type. An extraction model reads this to decide what belongs, so write it the way you would explain it to a colleague. At most 1,024 characters;
• properties: a JSON Schema (Draft 2020-12) of the properties. Only references within the same document are followed;
• identity_keys: the properties that identify one entity, at most eight. Two extracted entities with the same values are the same entity. Without keys the normalised name decides: case, whitespace and surrounding punctuation do not count.

A relation type has a name (starts with a lower-case letter, then lower-case letters, digits and _, at most 64 characters, such as supplies), a description, from and to (each a list of entity types the relation may run between) and properties.

A schema has at least one and at most 50 entity types, and at most 100 relation types.

Between two entities a relation of one type exists at most once. When a connection occurs several times, such as one contract per year, make it an entity type of its own.

Write a new version¶

curl -s -X PUT "$CORE_URL/knowledge/graphs/$GRAPH_ID/schema" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d @schema.json

The body is the whole schema, in the same form as schema above. When the content differs from the latest version, Core writes a new version; when it is the same, nothing happens. Versions are immutable and numbered from 1.

• GET .../schema returns the latest version, GET .../schema/versions all versions and GET .../schema/versions/{version} one version.
• A new version does not re-check what is already in the graph. Every fact remembers in schema_version which version it was checked against.
• Writing and querying always use the latest version. A schema that leaves out an entity type or a relation type the graph still holds facts of is refused with a 409 that names each type and says how many facts have it; delete those facts first.
• The answer to a write that changes what is extracted carries reindex_required: true. Core then reads nothing again by itself; you start that yourself (Read again).

Have a schema suggested¶

POST /assist/graph-schema has your organisation's helper model write a schema. Besides knowledge, your key needs the assist capability for it; without assist the route answers 404. Nothing is stored: you get a suggestion back that you use yourself with PUT .../schema or in POST /knowledge/graphs.

curl -s -X POST "$CORE_URL/assist/graph-schema" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{"description": "Suppliers, the products they deliver and the substances in each product.",
         "collection_id": "col_2Pf7...", "sample_size": 3}'

At least one of description and collection_id is required. The answer holds schema (exactly the body PUT .../schema takes), explanation, the model that wrote it and sample: the sources the model was shown. The text of those sources goes to the helper model; that is a model call under your key's policy, and it is metered as model tokens.

3. Write facts yourself¶

curl -s -X POST "$CORE_URL/knowledge/graphs/$GRAPH_ID/entities" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{"type": "Company", "name": "Acme", "aliases": ["Acme BV"],
         "properties": {"country": "NL"}}'

curl -s -X POST "$CORE_URL/knowledge/graphs/$GRAPH_ID/relations" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{"type": "supplies", "from_entity_id": "kge_3Ta8...",
         "to_entity_id": "kge_7Lm2...", "properties": {"since": 2021}}'

Core checks every fact against the latest schema version: the type must exist, the properties must satisfy the type's JSON Schema, a type with identity_keys needs a value for each key, and a relation's two ends must exist in the graph and be of a type the relation type allows at that end.

A fact you write yourself has origin: manual. It has no source, does not disappear when a document disappears and is visible to every reader of the graph.

An extracted fact (origin: extracted) cannot be changed or deleted: PATCH and DELETE answer 409, because the next time its sources are read it would come back. Correct the source, or write a manual fact. When extraction finds an entity that already exists as a manual one, the source is attached to that entity and it stays manual, with its own values.

4. Fill from sources¶

A graph reads facts from the sources of the collections you link to it, with an ingest pipeline you choose:

curl -s -X PATCH "$CORE_URL/knowledge/graphs/$GRAPH_ID" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{"collection_ids": [{"collection_id": "col_2Pf7...",
                            "metadata_filter": {"department": "purchasing"}}],
         "ingest_pipeline": {"pipeline_id": "pip_...", "version": 1}}'

• collection_ids: at most 16 collections, each with an optional metadata_filter. Only sources whose metadata contains the filter are read; without a filter the whole collection.
• ingest_pipeline: a version of a pipeline of your own (pipeline_id and version) or a built-in template ({"template": "graph/..."}). The pipeline must end in the graph_store step. Without a pipeline the graph reads no source at all.
• embedding_model: the embedding model the names of entities are embedded with, so a search also finds them by meaning. null uses the installation's default.

Choose the pipeline¶

Three built-in templates end in graph_store:

The two templates without a model can be named directly. Their mapping is an example: graph/from-metadata reads a Company entity from the metadata key company, and graph/from-tables reads Company, Product and supplies from the columns supplier and product. When that does not fit your schema, make a pipeline of your own with your own mapping.

graph/llm-extract cannot be named directly: the graph_extract step has to name a model, and the template fixes none. Make a pipeline of your own with the same steps and the model filled in:

curl -s -X POST "$CORE_URL/knowledge/pipelines" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{
      "name": "suppliers-extract",
      "kind": "ingest",
      "config": {
        "nodes": [
          {"id": "input", "type": "input"},
          {"id": "load", "type": "load"},
          {"id": "parse", "type": "parse"},
          {"id": "split", "type": "sentence_splitter"},
          {"id": "extract", "type": "graph_extract", "config": {"model": "<a model id>"}},
          {"id": "gstore", "type": "graph_store"},
          {"id": "output", "type": "output"}
        ],
        "edges": [
          {"source": "input", "target": "load"},
          {"source": "load", "target": "parse"},
          {"source": "parse", "target": "split"},
          {"source": "split", "target": "extract"},
          {"source": "extract", "target": "gstore"},
          {"source": "gstore", "target": "output"}
        ]
      }
    }'

Put the pipeline you get back, with its version, in ingest_pipeline. A pipeline that does not end in graph_store, or a graph_extract without a model, answers 422 on ingest_pipeline. The same pipeline cannot fill an index and a graph at once.

A mapping, for metadata_to_graph and rows_to_graph, is one JSON object:

{
  "entities": [
    {"as": "supplier", "type": "Company", "name": "supplier"},
    {"as": "product", "type": "Product", "name": "product", "properties": {"sku": "article_number"}}
  ],
  "relations": [{"type": "supplies", "from": "supplier", "to": "product"}]
}

name and each property name the metadata key or the column the value comes from; a relation joins two entities by their as. When an entity's name is missing from a source, that entity is not written, and neither is a relation to it. See Pipelines for editing steps.

Every step holds what it finds to the schema: a type the schema does not name is dropped, as is a property the type does not have or a value that does not satisfy the schema. So nothing enters the graph that the schema does not allow.

When a source is read¶

A new source in a linked collection is read automatically. When you change a source's metadata, Core decides again which graphs it belongs in. Sources that were already there when you linked the collection or chose the pipeline are read only when you read again.

GET .../sources gives per source the status (queued, processing, ready or failed), the number of facts (fact_count) and on a failure a fixed error_code. A source that names nothing the schema knows is ready with fact_count: 0. A graph without a schema fails a source with no_schema, a pipeline that has since been deleted with no_pipeline, and a graph reads a source whole, so it fails one larger than the upload limit (25 MiB by default) with too_large.

When a source is deleted, the extracted facts that rested only on that source disappear. What other sources also state, stays.

Read again¶

curl -s -X POST "$CORE_URL/knowledge/graphs/$GRAPH_ID/reindex" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{"mode": "full"}'

full (the default) reads every source in scope again and takes out what sources said that the graph no longer reads. failed retries only the sources whose last reading failed. The answer is 202 with a run; GET .../runs and GET .../runs/{run_id} show the progress (sources_total, sources_done, sources_failed). When a run for this graph is already running or queued, the answer is 409.

Read again after you changed the schema, the pipeline or the embedding model, linked a collection that already held sources, or widened or narrowed a filter.

5. Query the graph¶

A query is a form, not query text. You start from entities of one type and follow at most three relations:

curl -s -X POST "$CORE_URL/knowledge/graphs/$GRAPH_ID/query" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{
      "start": {"type": "Company", "name": "Acme",
                "filters": [{"field": "country", "op": "eq", "value": "NL"}]},
      "follow": [
        {"relation": "supplies", "direction": "out", "to_type": "Product"}
      ],
      "limit": 50
    }'

This reads as: Dutch companies named Acme, and the products they supply.

A filter is {"field", "op", "value"} with eq, ne, lt, lte, gt, gte or contains; several filters apply together. contains works on text only.

The answer, abridged here to one entity, one relation and the fields you usually read:

{
  "entities": [{"id": "kge_3Ta8...", "type": "Company", "name": "Acme", "properties": {"country": "NL"},
                "origin": "extracted", "sources": [{"id": "src_...", "collection_id": "col_...",
                "title": "Framework contract Acme 2024"}]}],
  "relations": [{"id": "kgr_1Xc4...", "type": "supplies", "from_entity_id": "kge_3Ta8...",
                 "to_entity_id": "kge_7Lm2..."}],
  "paths": [["kge_3Ta8...", "kgr_1Xc4...", "kge_7Lm2..."]],
  "truncated": false
}

paths holds each path as ids: the start entity, then a relation and an entity per step. entities and relations hold every fact on a path once. sources names at most five sources per fact, and only sources the request's person may read.

Limits. A step follows at most 50 relations per entity and passes at most 200 entities on to the next step. When a limit or limit cut the answer short, truncated is true: the answer is then partial. A query has a time limit of its own, five seconds by default.

A query through this route is not metered as usage.

6. Let a model search¶

A model reaches a graph through a search tool, like an index. The search tool runs a retrieval pipeline with a graph source block. Three templates have one:

Copy a template and fill in its source blocks, then register the tool on that version:

curl -s -X POST "$CORE_URL/knowledge/pipelines/from-template" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{"template": "retrieval/hybrid-graph", "name": "suppliers-search",
         "sources": {"index": "idx_8Bd3...", "graph": "kg_5Wn1..."}}'

curl -s -X POST "$CORE_URL/knowledge/retrieval-tools" \
    -H "Authorization: Bearer $CORE_KEY" -H "Content-Type: application/json" \
    -d '{"name": "suppliers",
         "description": "Finds suppliers, their products and the contracts about them.",
         "pipeline": {"pipeline_id": "pip_...", "version": 1}}'

The source block is called graph in these templates; retrieval/graph-documents also has a collection block, and retrieval/hybrid-graph an index block. Offer the tool as {"type": "tool", "name": "knowledge__suppliers"}.

What the model gets back:

• the entities and relations found, as structured data, with the paths and truncated as well for a filled-in query;
• for each fact with a source the person may read, a passage with a citation, such as Acme -- supplies -> Widget (since: 2021). A manual fact has no source and reaches the model as data only.

A tool on retrieval/graph-documents offers the model a graph_query argument, with the schema's types and relations as choices. A query the schema refuses comes back to the model as invalid arguments, so it can correct the query; the turn does not fail because of it.

A graph is not filtered by metadata. A tool's filter_keys narrow only what is searched in indexes and collections. A metadata_filter on a tool whose pipeline has a graph block answers 422. What a graph holds is decided by the collections it reads, their filters and the visibility.

Every search step over a graph counts as one retrieval unit (retrieval), plus embedding tokens when it searched by meaning and model tokens when a step calls a model itself.

Who sees which facts¶

You choose the visibility at creation and it does not change afterwards.

Use shared for the organisation's own knowledge: a product catalogue, a supplier register. A person's sources never enter it, whatever the filter says. Note: inside a shared graph the labels on sources narrow nothing. A source a person may not see through the rules about Knowledge sources still contributes facts every reader of the graph sees. When labels on sources have to hold inside the graph too, choose per_person.

Use per_person for documents that belong to people: their own contracts, uploads, notes. A person then sees of an entity only the names and properties her own visible sources stated. A request names the person with the X-Core-Subject header (Build for people).

Which graphs a key reaches is decided by rules about Knowledge graphs in the policy, by name or by label. The default rules permit every graph. A graph no rule permits is missing from lists and answers 404; creating one under a name no rule permits answers 403.

What extraction costs¶

Extraction with a model costs model tokens in proportion to the text: one call per chunk, for every source the graph reads, and again every time you read again. The step's model decides the price, so choosing a cheaper model is done in the pipeline. The templates without a model cost nothing; embedding new entities costs embedding tokens. The usage counts against the organisation's cost cap.

What a graph does not do¶

• Entities are merged on identity keys or on the normalised name, not on similarity. Acme and Acme Holding are two entities, unless their type has identity keys and both carry the same values.
• A model cannot write facts into a graph during a turn.
• A graph cannot go into a publication.
• A query has no paths of unbounded length and no aggregation over a path.