AI Model

AI Model calls a model. It is the only node of a flow that spends money: it holds the expected cost of the call in your budgets, sends the call to a provider and has the hold settled after the answer.

What it does

When an execution reaches the node, the call goes through three steps.

1. The hold. Slen checks the policies, estimates the call — the model, the messages, the output ceiling — and holds the estimate multiplied by a safety multiplier in the budgets that bind the call. A call made for one of your customers also holds its price in credits on the customer's account of the credit type the request names in context.credit_type. How budget reservations work explains the hold.

2. The call. A connection of a provider pool makes the call, attempt by attempt, as the next section says.

3. The settlement. When the provider answers, the execution leaves through Success, and the hold is captured at what the call cost, in the background. cost.actual of the execution is known once that is done. A call that failed or was refused gives its hold back.

The prompt is messages of the input of the request, sent as it came. An input without a non-empty messages ends the execution as failed with invalid_flow_input. The node adds the model, the output ceiling and the temperature of its settings, and a list of tools and a response format when they are set.

Which connection makes the call

• A node with a provider pool chosen calls through that pool.

• A node without one calls through the default pool of the provider of the model, and when there is none, through a single connection with one attempt — Provider pools says which.

• The node may set its own number of attempts and what follows a rate limit. Both win over the pool's, and both apply only while the call goes through a pool.

• A pool the node names that was deleted or is switched off, or a provider without a connection that can serve, ends the call with provider_not_configured.

When the execution waits at the node

The node can make an execution wait. Its status is then waiting, and the request that started it has been answered with 202.

• Waiting for a connection. When the node or its pool answers a rate limit with queue and no connection has room for the call, the execution waits as long as the provider asked, or a minute, and the node tries again with a new hold. Nothing is held while it waits. After three waits by default the call ends with pool_exhausted.

• A call the provider runs in the background. When the service is set up for it, the call of an execution sent in async mode may be handed to a provider that can run it in the background. The execution waits while Slen asks the provider how the call goes, and continues with the answer. The hold of such a call stands 30 minutes from the start of the call by default; that time bounds the hold, not the call. The end of the call is announced by operation.completed, operation.failed or operation.uncertain.

  • When the provider cannot be asked any more, the execution ends uncertain, and the hold stays until it expires.

  • When the hold was settled by another capture while the call waited, the call ends without asking the provider for its answer, with background_hold_expired. Nothing is given back, and the provider may still bill the call.

  • When the provider still reports the call as running after the execution has waited 72 hours by default, the execution is closed as failed with parked_expired. No event about the call is sent, and its operation stays pending.

The overrun policy

A new node has no overrun policy. A policy, chosen under On overrun, makes Slen estimate the call a second time just before the provider is called and compare the figure with the hold. Both estimates come from the same request a moment apart, so a policy practically never fires — and it adds the second estimate to every call. When it does fire, the call goes on, stops with overrun_stopped, waits for a person's approval, or moves to a cheaper model with a second hold, as the policy says. Approvals describes the request a person decides on.

When to use it

• Every flow that calls a model has the node: it is the call.

• Put Budget Check before it to refuse before money is held, and Cache Lookup to answer a repeated question without a call.

• Connect Deny, Error and Pool exhausted to choose what the caller hears: a cheaper model through Fallback, a fixed answer, a refusal with your own code.

• A flow may call several models. Each call holds and is settled on its own and is an operation of its own, and cost.estimated of the execution is the sum of the estimates of the calls made.

Settings

In the builder the Settings tab holds Model, Temperature, Max output tokens and On overrun. The Connections tab holds Provider pool, Max attempts and On rate limit (429). The Advanced tab shows the whole config of the node as it is saved, the keys the tabs have no field for included: those are set through the API.

A new node has no model, and a flow is not published until one is chosen. An empty Max output tokens caps the answer at 1024 tokens unless the service is set up otherwise, and the hold is sized on that ceiling.

Ports

• The last three ports are optional. When Pool exhausted has no connection and Error has one, the execution takes the Error branch. What an unconnected port does is in Ports and branching.

• When the provider was called and its answer was lost, the node takes no port: the execution ends uncertain with operation_uncertain, and the hold stays until it expires. Do not run such a request again as if it had failed.

• An execution that ends rejected after the call was refused keeps the estimate of the refused call as its cost.estimated: money the refusal saved, not money spent. A refusal on the credits of your customer leaves it empty. A Deny branch that answers without calling a model ends completed with the same estimate and no actual cost.

Output

When the provider answers, the node leaves the answer for the nodes that follow.

{
  "model": "gpt-4o-mini",
  "content": "Thank you for your order. It ships on Monday.",
  "finish_reason": "stop",
  "tool_calls": null,
  "usage": { "input_tokens": 1180, "output_tokens": 212, "cached_input_tokens": 0 }
}

A later node reads it by the path nodes.<key>.output.content, where <key> is the key of this node in the flow. Return Response answers with the output of the last model by default.

When the execution leaves through Deny, Pool exhausted or Error, the node leaves what ended the call instead:

{ "error_code": "budget_exceeded", "http_status": 402, "message": "Budget 74438200973595008 (monthly) exceeded" }

Errors

The settings of the node are checked when the flow is validated and when it is published.

The check also reads what the node names. A model the catalogue does not have (unknown_model), a pool that does not exist (unknown_pool) and a pool of another provider that takes no other providers (pool_provider_mismatch) stop the publication. A pool that is switched off, has no enabled connection or has none that can serve the model is a warning (pool_disabled, pool_without_connections, pool_cannot_serve_model), and so is a node without an Error branch (error_port_not_connected).

What happens to a call that is refused or fails is in Ports above; the meaning of every code and what to do about it is in Error codes.

Example

AI Model holds the money and calls the model, and its answer goes to Return Response. When the pool runs out or the provider fails, the execution passes Fallback to a second AI Model with a cheaper model and a hold of its own. When the call is refused, Reject Request ends the request with its own code.

Related

• The money chain — the nodes that can stand around the call.

• Provider pools — how a pool chooses a connection and what a failure does.

• How budget reservations work — the hold of a call and how it ends.

• Send a request — what your application gets for a refusal or a failure.