Route Data

In more complex architectures, you will often have several Sources feeding into the same Pipeline, with the data for each needing different types of processing before being sent to multiple destinations. A key component of these Pipelines is a Route Processor, which uses conditional statements to match data and send it along its particular processing route.

This topic describes a typical use of a Route Processor, with examples of the Processor configurations.

Interactive Demo

You can see how data is processed and routed through a Pipeline with this interactive demo.

Overview

This schematic illustrates the configuration of a Routing group, which includes a Script Execution Processor to format raw strings to JSON, that routes different data types from several sources through specialized processing chains to several destinations.

Interactive Demo

You can see how data is processed and routed through a Pipeline with this interactive demo.

1 - Sources

The Sources represent three different types of data flowing through the Pipeline that need to be routed to separate processing chains:

1. Financial Data that needs to have Personally Identifying Information encrypted before being sent to storage and the observability tool.
2. JSON Data that needs to have Status - 200 events routed and dropped.
3. Apache Errors that need to be converted to JSON format and all info messages dropped.

2 - Script Execution Processor

The $link[page,228924,auto$] is configured to convert the Apache errors from raw strings to JSON format.

3 - Route Processor

The $link[page,220155,auto$] uses three conditional statements to identify and route specific components of all three data types:

Apache Info Messages

Because both the JSON and Apache errors data contain .message fields, this statement uses AND to make sure that that .messages that don't contain the INFO event won't generate a "field not found" error. All messages meeting this criteria are sent to the Drop Destination.

Status 200 Events

All events that meet these criteria are sent to the Drop Destination.

Transaction Events

All events that meet these criteria are send to the Encrypt Processor.

4 - Encrypt Processor

Because transaction events contain Personally Identifying Information (PII), such as credit card numbers, this information needs to be encrypted before being sent to storage and observability tools. For more information, check out the topic $link[page,278104,auto$].

Encrypt Processor Configuration

5 - Destinations

The routed data is sent to three destination, represented in this schematic by the $link[page,227761,Black Hole Destination]:

1. Drop, where the unnecessary INFO and Status - 200 mesages are sent.
2. Storage, where all unmatched data and encrypted PII data is sent.
3. Observability Tool, where all unmatched data and encrypted PII data is sent.

The $link[page,227761,auto$] Destination drops all data sent to it. This makes it useful for testing your Processor chain to make sure you are getting the expected results before sending them on to a production Destination. Mezmo supports a wide variety of popular Destinations including $link[page,220137,auto$], $link[page,227633,auto$], and $link[page,223093,auto$].