Pipeline Event Data Model

Event Envelope

When data is moving from Sources through Processors to Destinations in a Pipeline, the key details of these events come through the message object. While this may be all you need in your Pipeline, there is other data that may be available to you in what we refer to as the event envelope.

Envelope Fields

Field	Purpose
`message`	The main field that carries the important data elements of an event. This field will always be present. Depending on the source, this field could be a string or an object. Metrics: while there is no prescribed schema for log events, metrics have a prescribed data model you must follow.
`metadata`	This field contains properties that describe the message. In most cases this data is only accessible in a Pipeline, and is not passed to the Destination. You can move Items in this field into `message` with the Map Fields Processor.
`timestamp`	This field is a ISO8601/RFC3339 formatted timestamp. For most Sources, this will be the date and time the event was received into the Pipeline. When there is a standard data format (OTLP, Auto-parsed logs in known formats, etc), Mezmo will set this field to the timestamp it finds in the standard field. You can override this value with the Set Timestamp Processor or the Script Execution Processor.

{
  "message":{
    "bytes": 22322,
    "datetime": "24/Jan/2023:18:54:09",
    "host": "127.219.215.140",
    "method": "DELETE",
    "protocol": "HTTP/1.1",
    "referer": "https://names.de/apps/deploy",
    "request": "/secret-info/open-sesame",
    "status": "300",
    "user-identifier": "meln1ks",
    "data": {
      "extra-data": "mydata"
    }
  }
  "metadata":{
    "headers":{}
    "query":{}
  }
  "timestamp":"2024-06-28T19:41:00.995+00:00"
}

{
  "bytes": 22322,
  "datetime": "24/Jan/2023:18:54:09",
  "host": "127.219.215.140",
  "method": "DELETE",
  "protocol": "HTTP/1.1",
  "referer": "https://names.de/apps/deploy",
  "request": "/secret-info/open-sesame",
  "status": "300",
  "user-identifier": "meln1ks",
  "data": {
  	"extra-data": "mydata"
  }
}

View Envelope Fields

You can view the contents of the envelope with the Pipeline Tap and Simulation features. If it isn’t displayed, make sure Show data envelope is set to ON.

Fields in message, metadata and timestamp can be referenced in most processors and destinations. The message field also has a shortcut syntax of . For example, to access host from the example above, you could reference it as message.host or .host

Edit or Delete a Pipeline

Simulate and Test Pipeline Data Streams

⌘I

About Pipelines

Sources

Processors

Destinations

Pipeline Management

Other Topics

Pipeline Event Data Model

Event Envelope

Envelope Fields

View Envelope Fields

​Event Envelope

​Envelope Fields

​View Envelope Fields

Event Envelope

Envelope Fields

View Envelope Fields