Dedupe Processor

Description

The Dedupe processor removes duplicate values from log data.

Use

This processor is most useful for reducing “chatter” in logs. The overlap of data across fields is the key to having this processor work effectively. This processor will emit the first matching record of the set of records that are being compared.

Configuration

There are three options to configure for this processor.

Option	Description	Example
Number of Events	The number of events to compare across. Limited to 5000.	`5000`
Comparison Type	Match will remove duplicate records based on the specified fields. Ignore will remove duplicate records based on all fields except those specified.	`Match`
Fields	The field or fields to apply the Dedupe processing to.	`.foo`

Example - Match

Before

{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, 'bat': true}
{"foo": "qux", "baz": 3}
{"foo": "qux", "baz": 4}
{"foo": "qux", "baz": 5}

Dedupe Options

Option	Value
Number of Events	`5`
Comparison Type	`Match`
Fields	`.foo`

After

{"foo": "bar", "baz": 1}
{"foo": "qux", "baz": 3}

Example - Ignore

Before

JSON
{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, "bat": true}
{"foo": "qux", "baz": 3}
{"foo": "qux", "baz": 4}
{"foo": "corge", "baz": 5}

Dedupe Options

Option	Value
Number of Events	`5`
Comparison Type	`Ignore`
Fields	`.baz`

After

{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, "bat": true}
{"foo": "qux", "baz": 3}
{"foo": "corge", "baz": 5}

About Pipelines

Sources

Processors

Destinations

Pipeline Management

Other Topics

Description

Use

Configuration

Example - Match

Before

Dedupe Options

After

Example - Ignore

Before

Dedupe Options

After

​Description

​Use

​Configuration

​Example - Match

​Before

​Dedupe Options

​After

​Example - Ignore

​Before

​Dedupe Options

​After

Description

Use

Configuration

Example - Match

Before

Dedupe Options

After

Example - Ignore

Before

Dedupe Options

After