> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mezmo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Dedupe Processor

> Remove duplicate log records across a set of events using Match or Ignore comparison on specified fields to reduce log chatter.

## Description

The Dedupe processor removes duplicate values from log data.

## Use

This processor is most useful for reducing “chatter” in logs. The overlap of data across fields is the key to having this processor work effectively. This processor will emit the first matching record of the set of records that are being compared.

## Configuration

There are three options to configure for this processor.

| **Option**           | **Description**                                                                                                                                                         | **Example** |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| **Number of Events** | The number of events to compare across. Limited to 5000.                                                                                                                | `5000`      |
| **Comparison Type**  | **Match** will remove duplicate records based on the specified fields. <br /><br />**Ignore** will remove duplicate records based on all fields except those specified. | `Match`     |
| **Fields**           | The field or fields to apply the Dedupe processing to.                                                                                                                  | `.foo`      |

## Example - Match

### Before

```json theme={null}
{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, 'bat': true}
{"foo": "qux", "baz": 3}
{"foo": "qux", "baz": 4}
{"foo": "qux", "baz": 5}

```

### Dedupe Options

| Option           | Value   |
| ---------------- | ------- |
| Number of Events | `5`     |
| Comparison Type  | `Match` |
| Fields           | `.foo`  |

### After

```json theme={null}
{"foo": "bar", "baz": 1}
{"foo": "qux", "baz": 3}

```

## Example - Ignore

### Before

```json theme={null}
JSON
{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, "bat": true}
{"foo": "qux", "baz": 3}
{"foo": "qux", "baz": 4}
{"foo": "corge", "baz": 5}

```

### Dedupe Options

| Option           | Value    |
| ---------------- | -------- |
| Number of Events | `5`      |
| Comparison Type  | `Ignore` |
| Fields           | `.baz`   |

### After

```json theme={null}
{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, "bat": true}
{"foo": "qux", "baz": 3}
{"foo": "corge", "baz": 5}

```