Skip to main content
In the Mezmo O’Reilly Report The Fundamentals of Telemetry Pipelines, telemetry data is described as a raw resource that must be refined in order to become useful information. The process of refinement is carried out through a telemetry pipeline. There is, however, no one-size-fits-all approach to telemetry pipeline design - you must design the pipeline to produce the type of information that suits your purpose. While the word “pipeline” brings to mind images of pipes, valves, and other plumbing fixtures, a data pipeline is better thought of as an algorithm - a series of operations executed in a specific order to produce a result. Within a telemetry data pipeline, the operations are represented by processors or processor groups that perform specific functions. In this guide you’ll find examples of Mezmo Telemetry Pipelines that are designed for specific purposes, along with descriptions of the processors typically used in each type of pipeline. You will also find tutorials for building “Pipettes” using Mezmo Demo Source Data, and interactive demos to help you understand how data is transformed into information as it passes through the pipeline.

​
Data Ingestion Pipelines

Data ingestion pipelines are designed to send log data to Mezmo Log Analysis.

​
Data Optimization Pipelines

Data optimization pipelines are designed to optimize specific types of data before sending it to observability tools and storage. These pipelines typically use processors like Filter, Route, Event to Metric, and Aggregate to transform the data into the format required for the destinations. The Mezmo Data Profiler can help you understand your data and provide recommendations for how to optimize it.

​
Data Archiving and Rehydration Pipelines

Data archiving and rehydration pipelines are designed to optimize data for storage by reducing its volume, but then being able to restore or “rehydrate” it as needed for incident or other investigations. The archiving pipeline will typically include elements of a data optimization pipeline, as well as processors to Mask and Encrypt Data in situations in which the data potentially includes Personally Identifying Information (PII). The rehydration pipeline will typically include processors like Filter and Map Fields, to make sure the data is in the correct format for your log analysis tool.

​
Responsive Pipelines

Responsive Pipelines are designed to change the pipeline’s data processing operations when a defined condition is detected in the data, for example a surge in data from a particular source, or the detection of PII data. This enables you to preserve full-fidelity copies of your data during an incident, for example, and is intended to help you reduce Mean Time to Resolution (MTTR).