Be told when a budget, a pool or the traffic crosses a line
Read the feed of alerts, acknowledge one, and write the rules that decide what Slen tells you about.
6 min read
An alert is a notice that a condition you set a rule for has come true. This page is about the alerts page of the console: the feed of alerts and the rules behind them.
Where the alerts are
Open Alerts. The page has two tabs. Feed is what has come true, and Rules is what you asked to be told about. Both show the project chosen in the header together with what belongs to the whole organization.
The feed opens for whoever has the read permission, which every role has. The Rules tab needs management:read, and changing a rule needs management:write. A rule of the whole organization is changed only by a member who reaches every project.
How an alert appears
Slen checks the active rules once a minute. Without a rule nothing is watched.
When the condition of a rule holds and the rule has no open alert, an alert opens.
While the condition holds, the same alert stays open, and every check refreshes its severity and its numbers. A rule has one open alert at most.
When the condition no longer holds, the alert is resolved. An alert has no button that closes it: pausing or removing its rule resolves it too.
Webhooks and notifications report these changes. A webhook gets the event alert.opened when an alert opens and alert.resolved when it is resolved, and a change of the severity of an open alert sends it no event. Notifications tell people the same in email, Telegram, Slack, Microsoft Teams or Discord, and also tell them when an open alert becomes critical.
Read the feed
A row of the feed says what happened in one sentence. Under it stand the severity, Warning or Critical, the type of the rule, the project the alert is about — or Whole organization when it is about no single project — the time, and the numbers behind the sentence.
Open only is on when the page opens. Turn it off to see the resolved alerts too.
Acknowledge records that you have seen the alert, and when. It does not resolve the alert, and the alert is still resolved when its condition ends.
The feed does not update by itself. Press Refresh to read it again.
The types of rule
| Type | The alert opens when | It is critical when |
|---|---|---|
| Budget threshold | A budget has used the threshold share of its limit or more. What is reserved counts as used, like what is captured | The budget is used up |
| Unusual spend | The spend of the current hour is the multiplier times the average of the hours before it, or more | Never |
| Pool capacity | The share of the connections of a pool that are healthy is below the minimum | No connection of the pool is healthy |
| Rate-limit spike | Providers answered with a rate limit the set number of times in the window, or more | Never |
| Error spike | The set share of the attempts in the window failed, or more | At least half of the attempts failed |
| Cache hit drop | The share of requests answered from the cache over the window is below the minimum | Never |
| Overrun rate | The set share of the holds captured in the window cost more than was held, or more | At least half of them did |
A rule of the type Unusual spend, Error spike, Cache hit drop or Overrun rate stays silent until there is enough to judge. Each of these types has a minimum among its settings: of spend, of requests or of captured holds.
The counters behind the rules are kept by the hour. A window shorter than an hour is judged over the hour it falls in.
Write a rule
1. Start a new rule
On the Rules tab press New rule.
2. Name the rule
Fill in Name. The name is what tells one rule from another in the list.
3. Choose the condition
Under Condition choose the type of the rule.
4. Choose what it watches
Under Watches choose This project or The whole organization. The second choice is offered only to a member who reaches every project.
A rule of the whole organization judges every project of the organization on that project's own numbers, and a rule of the type Pool capacity watches every pool of the organization, each with the project it belongs to. A budget or a pool of the whole organization is judged on its own, and its alert shows Whole organization in place of a project. The rule still keeps one open alert. The alert is about whichever of them is worst off when it opens, and it stays with it while the condition holds there. When that one recovers and another is still past the line, the alert is resolved and a new one opens about the other.
5. Set the numbers
Fill in the settings of the type. Each field says the range it takes and the value it starts with.
6. Create the rule
Press Create rule. The rule is active and is checked within a minute.
After a rule is created, its condition and what it watches cannot be changed: the alerts already filed under it were about them.
Who to notify
The drawer of a rule has the block Who to notify. Add channels of the team, roles or members to it, and Slen tells them when an alert of the rule opens, becomes critical and is resolved. The block keeps a route of notifications bound to the rule: the first recipient creates it, and removing the last one deletes it. Without management:write the block is read-only. Notifications says how such a route works.
Pause, change or remove a rule
The menu in the row of a rule holds three actions.
| Action | What it does |
|---|---|
| Edit | Changes the name and the settings |
| Pause | Stops the checks of the rule and keeps its settings. Resume starts them again |
| Remove | Deletes the rule after a confirmation. The alerts it filed stay in the feed |
Pausing or removing a rule resolves its open alert at once, and Slen sends alert.resolved. After Resume the next check opens a new alert if the condition still holds.
Where to go next
Policies — refuse a request instead of being told about it.
Alerts — the feed over the API.
Glossary — an alert and a webhook in one line each.
Budgets — the ceilings an alert about a budget watches.
Webhooks — hear the events of Slen in your service.
Notifications — tell people about alerts in email, Telegram, Slack, Microsoft Teams and Discord.
Receive the events of Slen — the delivery of
alert.openedandalert.resolved, and how to check its signature.