Throttle Processor

This Pipeline component is in Beta development, and should be used in Production environments with caution. Contact your Mezmo Account Manger to have this feature enabled. If you encounter any issues, please notify Mezmo Support.

Description

The Throttle processor applies rate limiting to a stream of events to limit load on downstream destinations.

Use

This processor is used for cases where a high-throughput of low-value events may be produced from a source, as a means of protecting destinations and services. The Throttle processor allows for granular rate limiting of events, either across the entire event stream or bucketed by the configured Key Field. Events that exceed the configured Limit within the time Window are discarded.

Configuration

Option	Description	Example
Limit	The number of events to allow during the time window. If you specify a Key Field , the limit will be applied to events for each unique value of that field.	100
Window	The time window during which the configured limit is applied. Options include `Milliseconds`, `Seconds`, `Minutes`, and `Hours`.	1min
Key Field	When configured, the value from this field will be used to group events into separate buckets for throttling. If left blank, or if the event does not contain a value in this field, the event will be throttled as part of the default bucket.	`.app`
Exclude	A conditional statement that will be applied to exclude events from throttling.	`if (.host equal 'host-4')`

Examples

Limit Events by Field

This example shows a configuration that allows only 1 event every 1 second, for each unique value of the .app field. Events produced by host-4 are excluded from throttling.

Before

[
  {"host": "host-1", "app": "app-1", "message": "GET /server HTTP/1.1"},
  {"host": "host-1", "app": "app-1", "message": "POST /widgets/152 HTTP/1.1"},
  {"host": "host-2", "app": "app-1", "message": "GET /widgets HTTP/2"},
  {"host": "host-3", "app": "app-2", "message": "GET /widgets HTTP/2"},
  {"host": "host-3", "app": "app-2", "message": "PUT /widgets/5593 HTTP/1.1"},
  {"host": "host-4", "app": "app-2", "message": "GET /server/status HTTP/1.1"},  
]

Options

Option	Value
Limit	1
Window	1s
Key Field	`.app`
Exclude	Field: `.host` Operator: `equal` Value: `host-4`

After

Three events are produced - the first event in the stream for each unique value of .app , along with any events excluded from throttling (when .host == "host-4" ).

{"host": "host-1", "app": "app-1", "message": "GET /server HTTP/1.1"}
{"host": "host-3", "app": "app-2", "message": "GET /widgets HTTP/2"}
{"host": "host-4", "app": "app-2", "message": "GET /server/status HTTP/1.1"}

Redact Processor

Parse Sequentially Processor

⌘I

About Pipelines

Sources

Processors

Destinations

Pipeline Management

Other Topics

Description

Use

Configuration

Examples

Limit Events by Field

Before

Options

After

​Description

​Use

​Configuration

​Examples

​Limit Events by Field

​Before

​Options

​After

Description

Use

Configuration

Examples

Limit Events by Field

Before

Options

After