Dedupe Processor

Description

The Dedupe processor removes duplicate values from log data.

This processor is most useful for reducing “chatter” in logs. The overlap of data across fields is the key to having this processor work effectively. This processor will emit the first matching record of the set of records that are being compared.

Configuration

There are three options to configure for this processor.

Option	Description	Example
Number of Events	The number of events to compare across. Limited to 5000.	`5000`
Comparison Type	Match will remove duplicate records based on the specified fields. Ignore will remove duplicate records based on all fields except those specified.	`Match`
Fields	The field or fields to apply the Dedupe processing to.	`.foo`

Example - Match

Before

JSON
    
 
{"foo": "bar", "baz": 1}{"foo": "bar", "baz": 2, 'bat': true}{"foo": "qux", "baz": 3}{"foo": "qux", "baz": 4}{"foo": "qux", "baz": 5}
Copy

Dedupe Options

Option	Value
Number of Events	`5`
Comparison Type	`Match`
Fields	`.foo`

After

JSON
    
 
{"foo": "bar", "baz": 1}{"foo": "qux", "baz": 3}
Copy

Example - Ignore

Before

JSON
    
 
JSON{"foo": "bar", "baz": 1}{"foo": "bar", "baz": 2, "bat": true}{"foo": "qux", "baz": 3}{"foo": "qux", "baz": 4}{"foo": "corge", "baz": 5}
Copy

Dedupe Options

Option	Value
Number of Events	`5`
Comparison Type	`Ignore`
Fields	`.baz`

After

JSON
    
 
{"foo": "bar", "baz": 1}{"foo": "bar", "baz": 2, "bat": true}{"foo": "qux", "baz": 3}{"foo": "corge", "baz": 5}
Copy

Last updated on Nov 1, 2024

Was this page helpful?