Dedupe Processor
Description
The Dedupe processor removes duplicate values from log data.
Use
This processor is most useful for reducing “chatter” in logs. The overlap of data across fields is the key to having this processor work effectively. This processor will emit the first matching record of the set of records that are being compared.
Configuration
There are three options to configure for this processor.
| Option | Description | Example |
|---|---|---|
| Number of Events | The number of events to compare across. Limited to 5000. | 5000 |
| Comparison Type | Match will remove duplicate records based on the specified fields. Ignore will remove duplicate records based on all fields except those specified. | Match |
| Fields | The field or fields to apply the Dedupe processing to. | .foo |
Example - Match
Before
{"foo": "bar", "baz": 1}{"foo": "bar", "baz": 2, 'bat': true}{"foo": "qux", "baz": 3}{"foo": "qux", "baz": 4}{"foo": "qux", "baz": 5}Dedupe Options
| Option | Value |
|---|---|
| Number of Events | 5 |
| Comparison Type | Match |
| Fields | .foo |
After
{"foo": "bar", "baz": 1}{"foo": "qux", "baz": 3}Example - Ignore
Before
JSON{"foo": "bar", "baz": 1}{"foo": "bar", "baz": 2, "bat": true}{"foo": "qux", "baz": 3}{"foo": "qux", "baz": 4}{"foo": "corge", "baz": 5}Dedupe Options
| Option | Value |
|---|---|
| Number of Events | 5 |
| Comparison Type | Ignore |
| Fields | .baz |
After
{"foo": "bar", "baz": 1}{"foo": "bar", "baz": 2, "bat": true}{"foo": "qux", "baz": 3}{"foo": "corge", "baz": 5}Was this page helpful?