Configure Responsive Pipelines

Responsive Pipelines enable you to pre-configure a Pipeline to change behavior automatically in the case of an incident. This makes it easier to balance the need for high-fidelity data required during incident response, with the need to reduce data load for cost reduction.

Responsive Pipelines adjust their behavior in response to specific triggers, such as the detection of a new critical incident. State change can be triggered through API calls from external incident response systems, such as PagerDuty, or you can activate it manually in the Pipeline in the Mezmo Web App.

This screenshot shows a Pipeline that, in Monitoring mode, drops Status 200 events, and converts other HTTP Status events to metrics. In Incident Mode, the Route Processor sends the full data stream to Mezmo Log Analysis.

Beta Limitations

The Beta release of Responsive Pipelines has these limitations:

• You need to use the Script Execution Processor to access the pipeline state variable and add it to the event metadata as described below.
• You can only change the Pipeline operational state via the Mezmo Telemetry Pipelines API

How it Works

Mezmo Responsive Pipelines introduces a state variable operational_state that is associated with each Pipeline. The value for this state is either normalor incident, and the Processors in the Pipeline can change their functioning based on the value. For example, the Sample Processor can be disabled if operational_state=incident so that during the incident, you can have high fidelity data for further analysis.

Set the Pipeline Operational State

You can change the operational state of a Pipeline manually through the Mezmo Web App, or through the Telemetry Pipelines API.

Mezmo Web App

You can change the operational state of a Pipeline in the Mezmo Web App by selecting the state in the upper-left corner of the Pipeline Map. The state of the Pipeline is also shown in the Pipeline name, with an orange flag indicating the Incident state.

Changing the operational state of a Pipeline in the Mezmo Web App

Script Execution Processor

You can use the Script Execution Processor to add the operational state of a Pipeline as metadata to a message to change the functioning of downstream Processors, such as the Route Processor.

This code shows how to get the Pipeline state variable and set it as message metadata for downstream Processors:

function processEvent(message, metadata, timestamp, annotations) {

  const state = getPipelineStateVariable("operational_state")

  message.op_state = state

  return message

}

This screenshot shows the configuration of the Script Execution Process to get the operational state of a Pipeline and use it in a message for downstream Processors

You can these use the value of op_state to set the routing conditions for the Pipeline, as shown in this example:

Route Processor Configuration

​

<Normal State>

if (metadata._op_state equal 'normal' OR is_null(metadata._op_state))

​

<Incident State>

if (metadata._op_state equal 'incident')

This screenshot shows the configuration of the conditions in the Route Processor for both Monitoring and Incident Modes

API

You can get and set the value of the Pipeline operational state with the Telemetry Pipelines API:

Get the state_id:

curl -s --request GET \

  --url 'https://api.mezmo.com/v3/pipeline/state-variable' \

  -H 'Authorization: Token <<PIPELINE_SERVICE_TOKEN>>' \

  -H 'Content-Type: application/json' \

  --data '{"pipeline_id": "<<PIPELINE_ID>>"}' | jq '.data[0].state'

Set the state:

curl -i --request PUT \

  --url 'https://api.mezmo.com/v3/pipeline/state-variable/<<STATEID>>' \

  -H 'Authorization: Token <<PIPELINE SERVICE_TOKEN>>' \

  -H 'Content-Type: application/json' \

  --data '{"pipeline_id": "<<PIPELINE_ID>>", "state": {"operational_state": "incident"}}'

Interactive Demo

This interactive demo illustrates a telemetry pipeline that is configured to respond to a Datadog data volume spike incident. You will need to have pop-ups enabled to view the demo. You can also view the demo without pop-ups at mezmo.com