Set In-Stream Alerts for Pipeline Nodes

You can set specific conditions to trigger an alert on any Source, or Processor in your Pipeline. Alerts are based on the aggregation of metrics or events within the data stream, and are triggered when when the aggregated data reaches meets specific conditions.

General Configuration

1. Log into the Mezmo Web App and select the Pipeline where you want to set the alert, then click Edit Pipeline.
2. Select the Source, or Processor in your Pipeline where you want to set the alert, then click Alerts.
3. Click New Alert.
4. Enter a Name and Description for the alert, then click Next.

Evaluation Configuration

Set the criteria for evaluation of the metric or event data.

1. Select the Alert Type: Threshold Alert, Change Alert, or Absence Alert.

2. Select the Event Type: Metric, Event.

3. Select the field paths to use for aggregating and grouping events based on matching values. Default field paths are.name, .namespace, .tags.

4. Select the type of aggregating Operation to use: Sum, Minimum, Maximum, Average, Set Intersection, Distribution Cocantenation, or Custom. If you select Custom, enter the Script you want to use. Check out the topic Script Execution Processor for more information about syntax for scripts.

5. Select the Window Type: Tumbling or Sliding.

   1. Tumbling windows are a series of fixed-sized, non-overlapping and contiguous time intervals. For example, if you set it to a five-minute tumbling window, the elements with timestamp values [0:00:00-0:05:00) are in the first window. Elements with timestamp values [0:05:00-0:10:00) are in the second window.
   2. A sliding window has a fixed time length, and it moves forward or “slides” at a time interval smaller than the window’s length. For example, a sliding window can be five minutes long, and slide every one minute and capture five minutes of data. The length of the slide is not user-configurable by user, the system will automatically calculate an appropriate slide based on the window size.

6. Set the Window Duration for aggregation, in minutes.

Alert Condition Configuration

Set the conditions that will trigger an alert based on the data evaluation. All conditions are based on an IF statement.

1. Enter the Field to use for the conditional statement.
2. Select the Operator to use in the statement.
3. Enter the Value to use.
4. Click Next.

As you enter the values, a conditional statement will be generated. For example:

if (.log_volume value_change_greater_or_equal 60)

Alert Event Structure

Internally, an alert instantiates an instance of the Aggregate Processor in order to track the configured Evaluation over time. The resulting alert event will consist of the event that triggered the alert, along with the aggregate results, which are stored in metadata . Any of this information can be extracted via templates for use in the Alert Payload.

This is an example of a Demo Logs Source that has an alert configured for it. Note the aggregate results inside of metadata, and the Demo Logs event that triggered the alert as the message.

{

  "message": {

    "bytes": 8655,

    "datetime": "21/Aug/2024:17:35:34",

    "host": "100.49.129.84",

    "method": "HEAD",

    "protocol": "HTTP/1.1",

    "referer": "johns.com",

    "request": "/assets92ab70912ac7aca655b1016e84c1508",

    "status": 200,

    "user-identifier": "dockbradtke"

  },

  "metadata": {

    "aggregate": {

      "end_timestamp": 1724261783647,

      "event_count": 1,

      "start_timestamp": 1724261753647

    }

  }

}

Alert Payload Configuration

The Alert Payload enables you to customize the alert message based on the incoming alert event structure prior to sending it to an external service. Throttling is also used to control the frequency of alerts.

1. Select one of the available services and configure its options.
2. Configure the options for the selected service.
3. Set the Throttling options.
4. Click Save.

Available Services

The available services to send alerts are Slack, PagerDuty, Webhook or Mezmo Log Analysis. These are the options for each service:

Slack

• URI - The full URI for the Slack API
• Message - A static or templated string to be the Slack message

PagerDuty

• URI - The full URI for the PagerDuty API
• Summary - Details of the alert, as specified by their API docs. Supports templating.
• Severity - INFO, WARNING, ERROR or CRITICAL
• Source - The source of the alert. Supports templating.
• Routing Key - For PagerDuty authorization
• Event Action - trigger, acknowledge or resolve

Webhook

• URI - The full URI for the Webhook
• HTTP Method - post, put, patch, delete, get, head, options or trace
• Message - A static or templated string as the webhook body. This can be a text string or serialized JSON. If the message can be parsed as JSON, it will be sent as such.
• HTTP Headers (optional)
• Authentication Options (optional)

Log Analysis

When using Mezmo Log Analysis, the Subject is the primary message in the payload. There will be additional properties appearing for body and level (Severity).

• Severity - The level of the log line: INFO, WARNING, ERROR or CRITICAL
• Subject - This text supports templating, and will be easily seen in the Log Analysis UI.
• Body - Additional information to include in the payload. Supports templating.
• Mezmo Ingestion Key - the key to access your Log Analysis instance.

View a Configured Alert

If a Source, Destination, or Processor has an alert configured, you will see an indicator on the node like this:

You can see the alert configuration by clicking on the alert indicator.

Interactive Demo

This demo illustrates the configuraiton of an in-stream alert for a Pipeline node. You will need to have pop-ups enabled for your browser or docs.mezmo.com to view the demo. You can also view the demo without a pop-up at mezmo.com.