Skip to main content
One of the most important functions of a telemetry pipeline is the ability to process log data to reduce its size, re-format and standardize it, or extract information from it for use in analytical operations. Mezmo Telemetry Pipelines include processors for an extensive set of use cases, that can be used independently for basic processing, or chained together for more complex operations. In this section you’ll find a complete listing of Mezmo Pipeline Processors, including their common use cases and configuration.
  1. Log into the Mezmo Web App.
  2. In the left-hand menu, click Pipelines. This will open the Edit Pipeline interface.
  3. In the left-hand navigation, click + New Pipeline.
  4. Enter a name for the Pipeline, then click Save.
  5. Set up your Pipeline Sources and Pipeline Destinations.
  6. After you connect the Source and Destination, click the connector and the Insert Node dialog will open.
  7. Select Insert Node and choose the Processor you want to apply to that Pipeline segment.
The topic Supported Processors provides a full list of available sources and their configuration instructions.

​
Processor Module Tutorials

These tutorials provide an overview of typical Processor configurations for specific use cases. They include a schematic of the “Pipette” and configuration instructions for each module, an interactive demo, and step-by-step instructions for how to create your own version using Demo Logs and a Blackhole Destination, and then connecting your own data sources and destinations.

​
Video Overview

Aggregate Processor

Aggregate metric and log fields over tumbling or sliding windows and trigger threshold or change alerts when conditions are met.

Compact Fields Processor

Recursively remove empty arrays and objects from a specified field to clean up null values and empty spaces in messages.

Data Profiler Processor

Analyze Source data to surface the most common message patterns and the apps and hosts driving log volume, with field mapping options.

Decrypt Field Processor

Decrypt an encrypted string field using a secret key and initialization vector, reversing the Encrypt Field Processor.

Remove Fields Processor

Drop specified JSON fields from each record in the data stream before sending it to storage or other processors.

Dedupe Processor

Remove duplicate log records across a set of events using Match or Ignore comparison on specified fields to reduce log chatter.

Encrypt Field Processor

Apply an encryption algorithm and key to a specified field to protect sensitive log data before sending it to storage.

Event to Metric Processor

Create a counter, sum, or gauge metric event from a log message, drawing the metric values from event fields or static values.

Filter Processor

Pass or drop events using conditional statements with content, string, comparison, and type operators to reduce forwarded data.

Flatten Fields Processor

Recursively collapse nested JSON objects into a single level, joining parent and child keys with a configurable delimiter.

Script Execution Processor

Transform events using a subset of JavaScript to filter, drop, map, and cast data within a single script, with supported functions and operators.

Map Fields Processor

Move or copy fields within an event, including nested fields, with options to drop the source field or overwrite an existing target.

Tag Cardinality Limit Processor

Cap the number of unique tag values per metric using exact or probabilistic matching, dropping the offending tag or the entire event when the limit is reached.

Parse Processor

Convert incoming data of a known format into parsed values using parsers like CSV, Grok, JSON, Regex, and timestamps, plus AI-assisted pattern matching.

Parse Sequentially Processor

Configure multiple parsers on a single node that are tried in order, applying the first parser that matches the incoming data.

Redact Processor

Identify and redact PII such as SSNs, emails, credit cards, and phone numbers in your data stream using pattern detection, hashing, or replacement.

Reduce Processor

Combine multiple log events into a single event over a time window using group-by fields and configurable per-field merge strategies.

Route Processor

Separate events from a single stream into multiple streams using conditional statements, so you can send events to different processors or destinations.

Sample Processor

Filter incoming events at a configurable 1/N rate to reduce event volume, with conditional rules to always forward matching events.

Set Timestamp Processor

Override an event's timestamp with a field and parser of your choice, trying multiple fields and formats until one parses successfully.

Stringify Processor

Render an entire event's JSON object into a single text string for downstream steps that require string input, such as the Encrypt processor.

Throttle Processor

Apply rate limiting to an event stream to protect downstream destinations, throttling globally or per key field within a configurable time window.

Trace Sampling Processor

Sample OpenTelemetry traces at a given rate using the Trace ID, supporting both head-based and conditional tail-based sampling strategies.

Unroll Processor

Convert a JSON object array within a field into individual objects, commonly used when a source packages multiple events in a single JSON object.