Concurrency limits — at most N model calls at once

A concurrency limit lets at most N model calls of one key run at the same time — of one customer, of the flow, of the whole project or of one value of a field of the request. A call over the limit waits its turn and goes on the moment a slot is free, or is refused when it has waited long enough. The limit is the Concurrency limit mode of Queue.

Why a limit

Before a call Slen knows two numbers. The estimate is what the call will probably cost; it learns from the history of the node and is right on average, not for every call. The ceiling is the most one call can cost — from the limits of the node and the prices of the catalog — and it holds for every call.

A budget is checked against estimates and holds. A limit counts calls. With at most N calls of a key in flight, the worst case of that key at any moment is N times the ceiling of the call, however wrong an estimate was. The panel of the node says that sum: with a limit of 3 and a ceiling of $0.42, the worst case of a key is $1.26.

A limit does more than that:

• one noisy customer does not take every call a shared provider key allows, and the other customers keep getting answers;

• a client that loops runs into a refusal after a short queue, instead of into a bill;

• the calls that wait cost nothing while they wait: nothing is reserved before the model node.

Only calls are counted. The money a key has in flight is not a limit of its own.

A limit and a budget

The two work together: a budget caps what a period may spend, a limit caps what can be at risk at once. A budget of the kind Concurrent holds (retired) used to count open holds; it is no longer created, and its archived budgets are not restored.

Add a limit

1. Open the flow in the builder and put Queue in front of the model it should guard — before Reserve Budget, if the flow has one. A new node is a limit already.

2. Choose Count by: Customer (the customer of the SDK, user_id of the context), This flow, Whole project or Request field and its path.

3. Set At most (calls at once), and under When full choose Wait in the queue with Wait at most (seconds), or Refuse at once.

4. Lead Busy to an answer of its own — Reject Request with a code your application knows, or a cheaper model behind a limit of its own.

5. Publish. The limit is on the plans from Pro up; a draft may hold it on any plan.

The template At most 3 calls per customer is this flow — AI flow templates.

Which call holds a slot

The node guards the first AI Model the path from it reaches. An execution takes a slot when it leaves the node and gives it back when that call ends — whatever port the call leaves by, and when the execution is closed. A slot a worker lost is given back by its lease. Between the node and the model may stand the nodes of the money chain, Condition, Transform, Trace Note, Cache Lookup and other limits; a wait there — a delay, Require Approval — is refused when the flow is published, because the execution would wait with the slot taken.

A fallback model added with Add a fallback model on a guarded model gets a copy of the limit in front of it, with the same key and limit: the slot of the first call is free once it leaves by a failure port, and the fallback call waits in the same queue — The builder.

Stack limits

One node counts by one key. For two, put two nodes one after the other: «customer, at most 2», then «this flow, at most 20», then the model. A noisy customer takes no more than 2 of the 20 slots, and no scheduler is needed.

Put the narrow key first. While an execution waits at the second node it keeps its slot of the first. Two flows that take the same keys in a different order can hold each other up; the deadline of the wait ends it, and the execution leaves by Busy.

Every limit node of the project with the same key shares one count, whatever its flow, and each keeps its own limit. A queue is served strictly in order, so an execution that waits under a smaller limit holds up the ones behind it; the panel warns when two limits of one flow count by the same key with different numbers.

Waiting instead of a refusal

A call over the limit is not refused at once. A request whose execution waits for a slot is answered with 202, waiting_on of the type queue_slot and retry_after_seconds of 2: the SDK's wait reads it again until the result is there, and a slot freed by another call wakes the execution within seconds. The deadline of the wait is kept to the minute. At most 100 executions wait for one key; the next leaves by Busy at once — Pauses, queues and resuming.

An execution that leaves by Busy with no edge from it ends with 402 concurrency_limit_exceeded, and the details of the error carry the output of the node — why, the key and its counts:

{ "reason": "queue_full", "key": "acme_42", "limit": 3, "in_flight": 3, "waiting": 100, "waited_seconds": 0 }

Watch the limits

• In Live mode the card of a limit says how many calls of its key are in flight and how many executions wait, and the panel of the canvas counts the executions that wait for a slot apart from the other waits.

• The Now section of the node lists its busiest keys — by their labels, never by the value of a field — with their calls in flight, the executions that wait and the oldest wait. The API answers the same with GET /v1/flows/{id}/limits — Flows.

• The trace of an execution names every step of its slot: slot_acquired, slot_waiting, slot_granted, slot_released, slot_busy — Inspect an execution.

Where to go next

• Queue — every setting, the output by each port and the checks of the node.

• Budgets — the ceiling on what a period may spend.

• Plans and limits — the plans that have the limit.