Skip to main content

​
Description

You can configure any source to send data via a RESTful POST to the Mezmo Pipeline.
When using the HTTP source, your content must be both encoded appropriately and packaged in way that enables it to be parsed after ingestion. You must also use a Parse Processor explicitly after ingestion in order to make use of any structured data. Structured formats, such as JSON, do not require additional parsing unless you want to further parse a specific value within the JSON.
You would typically use an HTTP request as a source when the specific source type you wish to send from is not supported. For example, you may have a not well known open source application that you wish to process. As long as you’re able to use a RESTful POST transport to send the data to an endpoint, you can send the data into the pipeline. All data sent to an HTTP endpoint must be parsed within the pipeline to extract the fields and act on it. Once the data is parsed, you can use it for any subsequent desired use case, such as remove or encrypt any sensitive data before sending them to other tools for application performance monitoring, log analysis, or security information and event monitoring.
If you use an individual request per log, you will quickly exceed your network capacity, so we recommend using a buffering solution.

​
Configuration

In the HTTP source configuration, use the pipeline data endpoint and configure the appropriate content type in your request. You must ensure that the Access key is included in the header with the configuration for Authorization: <access key> .

​
Mezmo Configuration Options

​
Parsing and Subsequent Processing

  • Always use a Parse Processor following your HTTP source to ensure the data in your payload is in a structured format.
  • You may also need to use an Unroll Processor if your events are included in an array.

​
Included Metadata

Note that you must explicitly turn on the metadata in the source to capture it. Metadata recorded by the source is split into two types: the header and query parameters. Each of these types can include values based on the sender. Certain fields will be included typically, such as ip and user_agent .

​
Examples

​
Reference Guides

The topic Set Up and Test an HTTP Endpoint Source up provides a basic example of how to set up your HTTP endpoint Source and typical Processors you would use.

​
Basic Example for Sending Data to an HTTP Source

Let’s set up an HTTP source and use curl to send in a sample payload with a set of key-value pairs.
  1. Create a new pipeline.
  2. Add an HTTP source to the graph with text decoding.
  3. Save the source, then generate an API key.
  4. Save the API key somewhere safe, such as a password manager
  5. Copy the HTTP URL endpoint. It should follow the format https://pipeline.mezmo.com/v1/%YOUR PIPELINE ID%
  6. Deploy the pipeline with just the HTTP source. You can ignore the warnings for missing connections.
  7. Start a tap on the output of the HTTP source by clicking on the right end of it and pressing the play button.
    1. Leave the tap running
  8. Open a terminal on your computer
  9. Enter this curl command, replacing the %PIPELINE ID% and the %TOKEN% with the information from your newly created pipeline
You should see the message payload msg=Hello_world show in the tap view after a few seconds as plain text with no color coding. If you want to make the message into a useful structured data object, you could also use the Parse Processor with a key-value pair parse algorithm to turn it into a JSON formatted schema.

​
Source Data Example with Metadata

Here’s a payload representing an example of what the inbound HTTP payload looks like when including all of the metadata from headers and query parameters.