Cache Lookup

A node that looks for a stored answer to the same request and, when it finds one, lets the flow answer without calling the model.

4 min read

Cache Lookup asks the cache whether the same request has been answered before. On a hit the flow answers with the stored answer and pays for nothing; on a miss it calls the model, and Cache Write stores the new answer.

Node

Cache Lookup

Type

cache_lookup

Category

Cache

Ports

Hit hit must be connected
Miss miss must be connected

Ends the path

What it does

The node makes a key out of the request and looks it up.

A hit leaves through Hit with the stored answer as the output of the node. Nothing is held and nothing is spent: no model is called for it, and the customer's credits are not charged.
A miss leaves through Miss and leaves its key for the Cache Write below, which stores the answer of the model under it.

When the answer of the execution is the one a hit gave back, the execution carries cache.hit set to true and cached_at, the moment the answer was stored. The Metrics tab of the node shows what its hits saved as Saved, estimated — at the estimate the stored answer was paid at.

When to use it

Put it before AI Model in a flow whose requests repeat word for word: the same question, the same document to summarize.

The key is an exact match. A prompt that differs by one space is another prompt.
The context of the call is not in the key. Two customers who send the same input share one answer, and the second is not charged.

Settings

Setting	What it is for
`scope`	Who shares an answer — operation, the requests of the operation on the published version that stored it, or project, every request of the project. A node without a scope takes operation.
`key_fields`	The paths inside the input the key is made of, written without input in front — messages, not input.messages. An empty list makes the whole input the key.

In the builder the settings are Shared across and Key fields.

Shared across	Who shares an answer
This operation	The requests of the operation, on the published version that stored it. A new version starts with an empty cache; a rollback finds what that version stored, while it is kept
The whole project	Every request of the project with the same key, whatever its operation and version

Key fields are the paths of the input the key is made of, written without input. in front: messages, not input.messages. Without any, the whole input is the key. A key field that a request does not have counts as null: the key of such a request is made of its other key fields alone.

Ports

The node has two ports, Hit and Miss, and both are required. A flow in which either leads nowhere is not published.

Output

On a hit the output is the answer the model gave when it was stored — the output of AI Model, with the usage of that call. On a miss the node leaves nothing. Return Response with the source Last model output answers with the stored answer when the hit is the last answer on the path.

Errors

The node raises no error: when the cache cannot answer, the node leaves through Miss, and a warning event of the trace says why. These are the warnings the node writes:

JSON

[
  { "type": "warning", "node_key": "lookup", "payload": { "code": "cache_unavailable" } },
  { "type": "warning", "node_key": "lookup", "payload": { "code": "cache_key_unresolved" } },
  { "type": "warning", "node_key": "lookup", "payload": { "code": "cache_key_partial" } }
]

Code	What happened
`cache_unavailable`	The cache store could not be reached. The key is left for Cache Write all the same
`cache_key_unresolved`	None of the key fields is in the input, so the key says nothing about the request. The cache is not asked, and nothing is stored under such a key
`cache_key_partial`	Some key fields are not in the input. The key is used without them

The settings are checked when the flow is validated and when it is published. A scope other than the two above is refused first:

In the API an entry of issues names such a check in params.detail; its code is invalid_node_config.

Check	Message in the builder
`cache_unsupported_scope`	“`{node}`” uses a cache scope that does not exist.

Then the key fields are checked:

In the API an entry of issues names such a check in params.detail; its code is invalid_node_config.

Check	Message in the builder
`cache_key_fields_not_list`	The key fields of “`{node}`” must be a list of input paths.
`cache_key_fields_empty_entry`	One of the key fields of “`{node}`” is empty.
`cache_key_fields_rooted_path`	“`{node}`” keys the cache by “`{path}`”, but key fields are read inside the input only: a path there does not start with “`{root}`”.
`cache_key_fields_padded_path`	“`{node}`” keys the cache by “`{path}`”, which has spaces at its ends: key fields are read exactly as written.

Example

A miss calls the model, and Cache Write stores its answer. A hit goes straight to Return Response, which answers with the stored answer.

Cache Write — the node that stores the answer.
Execution context and paths — what the output of a node is.
Send a request — cache and cost in the answer the caller gets.
AI Model — the call whose answer the cache keeps.

Built from openapi b070e1f, sdk contract 5cc7055, php ae4a53b, node 84bc62d