Put provider connections in a failover order
Group connections into a pool, choose how the pool picks a connection for a call and what it does when a provider refuses or fails.
5 min read
A provider pool groups connections to providers and decides which of them serves a model call, and what happens when an attempt fails. This page is about the provider pools page of the console.
Where the pools are
Open Settings, then Provider Pools. The page lists the pools of the project chosen in the header and the pools of the whole organization. A row shows the provider of the pool, its strategy, where it applies, how many connections it has and up to how many attempts a call makes.
The page opens for whoever has the management:read permission. Creating and changing a pool need management:write. A pool of the whole organization is changed only by a member who reaches every project.
Which calls a pool serves
An AI Model node that names a pool calls the model through that pool.
Any other call goes through the default pool of the provider of the model: a pool of that provider in the project, otherwise one of the whole organization, otherwise a pool that takes connections of other providers.
When no pool serves the call, Slen calls the provider through one connection, with one attempt — Connections says which one.
A pool chooses only among its own connections that can serve: enabled in the pool and not Disabled, Paused or in Error. When none can, the call is refused with provider_not_configured.
Create a pool
1. Start a new pool
Press New pool.
2. Fill in the basics
Under Basics fill in Name, choose the Provider and choose where the pool applies under Applies to: This project or Whole organization. The provider cannot be changed after the pool is created. Turn on Connections of other providers to let the pool hold them too: such a connection runs the equivalent model of its own provider, and one with no equivalent is skipped.
3. Choose the routing
Under Routing choose the Strategy and Max attempts, from 1 to 5. An attempt is one request to one connection.
4. Say what a failure does
Under On failures choose what follows a Rate limit (429) and a Provider error.
5. Create the pool
Press Create pool. The page of the new pool opens. Add connections to it before calls rely on it.
How a connection is chosen
| Strategy | The order of the connections for a call |
|---|---|
| By priority | The lowest priority number first |
| Least utilized | The connection furthest from its capacity limits first |
| By weight | A random draw in which a connection with a larger weight comes first more often. A connection with the weight 0 is tried only after the others, in the order of priority |
After a failed attempt the call moves on to the next connection in that order while attempts are left and the failure allows it. A connection that answered with a rate limit or an outage is set aside for a while and shows as Paused.
What a failure does
| Choice | After a Rate limit (429) | After a Provider error |
|---|---|---|
| Switch connection | The next attempt goes to another connection | The next attempt goes to another connection |
| Reject | The call ends with the refusal of the provider, provider_rate_limit | The call ends with the error of the provider |
| Fallback model (cross-provider pools) | The next attempts skip connections that would run the same model, so the call moves to another model. A pool with no such connection runs out of them | — |
Queue (a flow waits and retries) is listed too and cannot be chosen in the console. With it, a rate-limited call waits for a free connection, and its model node tries again.
When the attempts or the connections run out without an answer, the call ends with pool_exhausted — unless an AI Model node routes it to one of its ports, as Ports and branching explains. An AI Model node may set its own number of attempts and its own behaviour after a rate limit; they win over the pool's.
Add and arrange the connections
On the page of a pool the Connections tab lists its connections. Press Add connection, choose the Connection and set its Priority: a lower value is tried first. Under Advanced settings set the Weight for the strategy By weight and the Cost multiplier: the cost of an attempt through this connection is multiplied by it, from the next attempt on. Press Attach.
The list offers the connections of the provider of the pool. A connection of another provider is offered only when the pool takes them; through the API it is refused with
connection_incompatible.Enabled in a row takes the connection out of the pool and puts it back. A row also detaches its connection from the pool; the connection and its key stay.
Failover, under the list, changes the strategy, the attempts and the two failure choices. Press Save settings.
The Analytics tab counts, for the chosen period, the attempts and operations through the pool, the failovers and the calls a retry saved, and shows how the traffic was spread over the connections.
Change or delete a pool
The menu in the row of a pool holds Edit and Delete. Edit changes the name, the strategy, the attempts, the failure choices and whether other providers are taken; the provider and where the pool applies stay. Delete removes the pool after a confirmation and keeps its connections. A pool that the active version of a flow, or an execution that has not finished, still calls through is not deleted: the request is refused with the code pool_in_use.
Where to go next
Connections — add the connections a pool groups.
Glossary — a provider, a connection and a pool in one line each.
Provider pools — the same pools over the API.
AI Model — the node of a flow that calls a model through a pool.