Notifications

The page where you decide which alarms turn into notifications, and where those notifications are delivered. Two tabs: Routing Rules and Channels.

The appliance has an alarm engine that watches the system and raises alarms when something is wrong (see Alarms, chapter 11). The Notifications page sits one layer above the alarm engine: it routes those alarms to one of two destinations — the in-app bell icon, or an external pipeline.

A notification rule is a filter (“any system disk-space alarm at warning or worse”) plus a delivery destination (“send to Vector”, “ring the bell”). Without at least one rule, alarms still exist on the Alarms page but nothing notifies anyone about them.

Page Layout

The page is a two-tab panel: Routing Rules and Channels.

+-----------------------------------------------------------+
| Notifications  -  Manage notification routing rules and   |
|                   delivery channels                        |
+-----------------------------------------------------------+
| [ Routing Rules ]  [ Channels ]                            |
+-----------------------------------------------------------+
| (active tab content)                                       |
|                                                            |
+-----------------------------------------------------------+

Switch tabs to manage either side. They share state — routing rules can reference any channel defined on the Channels tab.

Routing Rules Tab

A table of every notification rule, with create / edit / delete / enable / disable controls.

What a Rule Does

A rule has five filters and two output settings. When an alarm fires (or resolves), the dispatcher walks the rules in order. For each rule whose filters match the alarm, the rule’s delivery channel receives the alarm.

A single alarm can match multiple rules — for example one that goes to the bell and another that ships to Vector. All matching rules fire.

Rules Table

Column	Description
Name	Short label for the rule.
Description	Optional free text.
Channel	Which alarm channels the rule listens for: `system`, `action`, or `any`.
Min Severity	Lowest severity that triggers the rule: `ok`, `warning`, `critical`, `unknown`.
Alarm Category	A friendly label for the alarm source-key match — e.g. “System: Disk”, “Action: Block”.
Delivery	Where the rule sends notifications: `bell` or `vector`.
Notify On	`fired` (alarm enters firing), `resolved` (alarm clears), or `all` (both).
Enabled	Per-rule toggle. Disabled rules stay in the list but are skipped by the dispatcher.
Created	Timestamp the rule was added.
Actions	Pencil to edit, trash to delete.

The list paginates at 20 rules per page. The toolbar shows the total rule count and a Create Rule button.

Creating a Rule

Press Create Rule. A modal opens with the following fields:

Field	Options
Name (required)	Free text, e.g. `Critical system alerts`.
Description	Free text, optional.
Channel Match	`system` (engine-health alarms), `action` (enforcement alarms), `any` (both).
Min Severity	`ok`, `warning`, `critical`, `unknown`.
Alarm Category	Drop-down of canned patterns: “All alarms”, “All system alarms”, “System: ClickHouse”, “System: Disk”, “System: License”, “All action alarms”, “Action: Block”, “Action: Deny”, “Action: Throttle”, “Action: Automation”.
Delivery Channel	`vector` or `bell`.
Notify On	`fired`, `resolved`, `all`.

Press Create Rule to save. The new rule appears in the table immediately and starts matching against new alarms within seconds.

Editing a Rule

Click the pencil icon on any row. The same modal opens with the rule’s current values pre-filled. Changes save when you press Save Changes and take effect on the next alarm.

Enabling or Disabling a Rule

Toggle the switch in the Enabled column. Disabled rules stay in the list but the dispatcher skips them — useful for temporarily silencing a noisy rule without losing its configuration.

Deleting a Rule

Click the trash icon. A confirmation modal asks you to confirm; the deletion is permanent and cannot be undone (the rule is removed, not soft-deleted). Existing notifications already delivered to the bell or to Vector are unaffected.

Channels Tab

Manages the external delivery destinations. Today there is one channel — Vector — but the design accommodates more.

The bell channel is implicit: every operator’s bell-icon dropdown receives whatever a bell-delivery rule matches. There is no bell-channel configuration to do.

The Vector channel needs an endpoint URL and an optional bearer token. The card on the Channels tab is where you configure that.

Vector Channel Card

The Vector card shows:

Status badge next to the channel name:
- Connected (green) — endpoint configured, enabled, and reachable.
- Half-Open (yellow) — the circuit breaker is sampling after a failure burst.
- Open (failing) (red) — recent deliveries failed; the breaker is paused.
- Inactive (grey) — endpoint configured but the channel is toggled off.
- Not configured (grey) — no endpoint URL saved yet.
Failure counter next to the badge — number of consecutive recent failures, hidden when zero.
Enable toggle — switches the channel on or off. Disabled until a valid endpoint is saved.

Below the header:

Endpoint URL — the HTTP/HTTPS address of your Vector receiver. The card warns in yellow if the URL is http:// rather than https:// — notifications would be sent in plaintext.
Token — optional bearer token sent as Authorization: Bearer ... with every notification. If a token is already saved, the field shows (set) and you can leave it alone. To change the token, clear and retype it.

Two buttons at the bottom:

Test Connection — sends a synthetic notification through the configured endpoint, then refreshes the status. The result appears under the buttons as a green check or a red cross with the failure message.
Save — commits the endpoint and (when changed) the token.

The card polls the channel status every 30 seconds so the badge stays current. A failing endpoint will turn the badge red within a couple of poll cycles after the failure starts.

How Vector Delivery Works

When a vector-delivery rule matches a fired or resolved alarm, the dispatcher posts a JSON payload to the configured endpoint with the bearer token (if any). The payload contains the alarm fields visible on the Alarms page: severity, channel, description, source key, metric value, threshold, timestamps, and the firing/resolved state.

The dispatcher protects the channel with a circuit breaker. If several consecutive deliveries fail, the breaker opens and the dispatcher pauses external delivery for a cool-down period, then samples one delivery to test recovery (the half-open state). Once a sample succeeds, normal delivery resumes.

Vector is the integration point with your downstream pipeline. From Vector you can fan out to Slack, email, PagerDuty, an SIEM, or anything else Vector knows how to talk to.

Tip. Start by routing every critical alarm to the bell (Channel: any, Min Severity: critical, Delivery: bell). Add a separate vector rule for the subset you want piped to an external system. The bell rule gives every operator immediate visibility; the Vector rule gives you durable, off-box history.

Example Rule Configurations

Three common starting patterns:

Route All Critical Alarms to the Bell

Channel Match: any
Min Severity: critical
Alarm Category: All alarms
Delivery: bell
Notify On: fired

This is the typical “everyone in the GUI sees the red badge appear” rule.

Page External Monitoring on ClickHouse Failures

Channel Match: system
Min Severity: warning
Alarm Category: System: ClickHouse
Delivery: vector
Notify On: all

Pipes both firing and resolution events into your external pipeline so the on-call channel sees clear-up notifications too.

Quiet “Device Blocked” Bell Spam in Steady State

If your environment legitimately blocks lots of devices and you don’t want the bell to ring every time, route action alarms only at critical severity:

Channel Match: action
Min Severity: critical
Alarm Category: All action alarms
Delivery: bell
Notify On: fired

Or use a more specific category like Action: Automation to only ring on rule-driven enforcement.

Testing Notifications

Two ways to verify rules end-to-end.

Test the Channel

The Test Connection button on the Vector card sends a synthetic alarm through the channel without involving the rule engine. Use it to confirm your Vector endpoint is reachable and that the token is correct.

Test the Rule

To verify a rule matches the alarms you expect, look at the alarm list on the Alarms (chapter 11) page. Resolve a stale alarm and watch a corresponding notify_on: resolved rule fire — or wait for a real alarm and see it land at the right destination.

The dispatcher does not currently expose a per-rule “send synthetic alarm” button. The combination of channel-level test plus the real alarm list is the supported way to verify rule behaviour.

Where Notifications Show Up

Bell delivery and Vector delivery have very different UX.

Destination	Where it appears	Persistence	Severity styling
Bell	Bell icon at the top-right of every page, dropdown listing the 20 most recent	Stored on the appliance until you mark read	Colour-coded by severity
Vector	Wherever your Vector pipeline forwards it (Slack, email, SIEM, etc.)	Outside the appliance	Determined by the downstream system

The bell icon is described in detail on the Alarms (chapter 11) page.

Troubleshooting

Bell shows nothing when alarms fire. Check that you have at least one rule with Delivery: bell and that its filters actually match the alarm — Channel Match, Min Severity, and Alarm Category all have to align. Try a permissive starter rule (any / warning / All alarms) to confirm the engine is working at all.

Vector test says “Connection error”. Confirm the endpoint URL is reachable from the appliance host. The appliance does not proxy through your workstation — if you can reach the URL from your browser but not the appliance, the issue is network-side. Use the Terminal (chapter 33) page on the appliance to curl the endpoint and prove reachability.

Vector test succeeds but real alarms do not arrive. Verify that at least one routing rule has Delivery: vector and that the rule is enabled. The test bypasses the rule engine; real alarms need a matching rule.

Badge stays “Open (failing)” after fixing the endpoint. The circuit breaker cools off for a fixed window before retrying. Wait one poll cycle (30 seconds) or press Test Connection to force a probe.

An alarm I expected stays unread on the bell. Bell entries auto-clear by being clicked or by Mark All Read in the bell dropdown. Resolving the alarm does not remove the bell entry.

Alarms (chapter 11) — the alarm engine itself, the lifecycle of an alarm, severity scheme, and source key conventions. Read this first if rules are not behaving as expected.
User Profile — your personal preferences. Notifications are appliance-wide, not per-user, but the bell icon you see is shared across all operators.