Getting Started with Mezmo Telemetry Pipelines

Organizations of all sizes rely on data to drive critical business decisions. Whether it's troubleshooting critical operations, optimizing performance, fixing security flaws, or ensuring compliance to policy, every action starts with understanding the data.

Data visibility is a prerequisite for action. Given that the size of data volumes generated by organizations is increasing, storing all of the data in a single location is no longer practical or effective.

Mezmo Telemetry Pipelines provide a way for organizations to collect and process large data streams in motion to enable visibility. Beyond enabling visibility, users of Mezmo Pipelines can also define scalable operations that help them monitor, alert, and manage data flowing as it passes through the Pipeline.

Create an Account and Organization

1. Click Sign Up on the Mezmo website. You will be automatically enrolled into a 30-day free trial.
2. Your organization is an independent workspace where you can access and configure your logs, add members, change billing plan, and manage other aspects of your account. You can find more information in the Organization Management Overview section of the Mezmo Log Analysis product guide.

Pipeline Components

Pipelines are composed of Sources, Processors, and Destinations.

• Sources are systems that create data during regular operation
• Processors are data operators that are used to act on the data from one or more sources
• Destinations are external targets where data is sent once it has been processed

Sources

Pipelines start by taking data in from one or more sources. Example sources include web applications, firewalls, operating systems, and IoT devices. Any networked system will usually generate data during normal operation in the form of logs, but source data can also include metrics.

A single Pipeline can allow for more than one source, but each source is parsed according to its own structure at the ingestion point. This means that syslog data is parsed as syslog at ingestion, while Fluent data would be parsed according to the Fluent standard.

Processors

Data that is sent into a Mezmo Pipeline is acted upon in accordance with the configuration of processors. Processors are discrete functionalities that can be chained together to act on data.

Example processors include filtering events based on defined criteria, deduplicating events based on matching field criteria, reducing event fields that are extraneous, and routing events to the next operation or destination.

Destinations

Destinations are targets for data streams external from a Pipeline. Destinations are typically data storage solutions, but also can include data handling or visualization tools.

Data sent to destinations counts as egress, which is measured in bytes.

Pipeline Usage Metrics

The key metrics are ingress and egress, measured in bytes. Computation is another metric that will be exposed in the future based on how much processing is configured in any Pipeline, but it is not currently shown.

Ingress

Ingress measures the incoming bytes added to a Pipeline. Ingress is measured and aggregated for you based on the sources you have configured.

Egress

Egress measures the outgoing bytes coming from a Pipeline. Egress is measured and aggregated for you based on which destinations you have configured.

Pipeline billing is based on egress. Note that any destination results in egress, but data sent to Mezmo Log Analysis is automatically subtracted from your billing metric..

All pipelines created are by default durable, meaning that if you send data to a Mezmo pipeline and we acknowledge that with a HTTP 200 response, we have recorded that data and it will not be lost.

View the Usage Metrics for a Pipeline

You can view the Ingress and Egress volume for the past 24 hours for any Pipeline by selecting it in the Pipelines interface of the Mezmo Web App. The Ingress and Egress volume will be displayed above the Pipeline map, as shown in this example.

An example of the Ingress/Egress volume for a Pipeline

Build a Pipeline

• Build and Deploy a Mezmo Telemetry Pipeline contains an overview of the basic process for building a Pipeline
• Build a Mezmo Telemetry Pipeline with Terraform provides instructions for using Terraform and the Mezmo Terraform Provider to build a Pipeline