HTTP
Description
You can configure any source to send data via a RESTful POST to the Memzo Pipeline.
When using the HTTP source, your content must be both encoded appropriately and packaged in way that enables it to be parsed after ingestion. You must also use a Parse Processor explicitly after ingestion in order to make use of any structured data. Structured formats, such as JSON, do not require additional parsing unless you want to further parse a specific value within the JSON.
You would typically use an HTTP request as a source when the specific source type you wish to send from is not supported. For example, you may have a not well known open source application that you wish to process. As long as you're able to use a RESTful POST transport to send the data to an endpoint, you can send the data into the pipeline.
All data sent to an HTTP endpoint must be parsed within the pipeline to extract the fields and act on it. Once the data is parsed, you can use it for any subsequent desired use case, such as remove or encrypt any sensitive data before sending them to other tools for application performance monitoring, log analysis, or security information and event monitoring.
If you use an individual request per log, you will quickly exceed your network capacity, so we recommend using a buffering solution.
Configuration
In the HTTP source configuration, use the pipeline data endpoint and configure the appropriate content type in your request. You must ensure that the Access key is included in the header with the configuration for Authorization: <access key> .
Mezmo Configuration Options
| Setting | Description |
|---|---|
| Title | A name for your source. |
| Description | A short description of the source. |
| Decoding Method | The decoding method to use to convert frames to data events Note: We recommend using Auto for convenience and to prevent potential errors in type mismatch |
| Access Key Management |
Make sure to copy the Access Key and note the Ingestion URL, which should resemble You can include your Access key in any of the following headers for authentication:
|
Parsing and Subsequent Processing
- Always use a Parse Processor following your HTTP source to ensure the data in your payload is in a structured format.
- You may also need to use an Unroll Processor if your events are included in an array.
Included Metadata
Note that you must explicitly turn on the metadata in the source to capture it.
Metadata recorded by the source is split into two types: the header and query parameters. Each of these types can include values based on the sender. Certain fields will be included typically, such as ip and user_agent .
| Object | Description |
|---|---|
header | This metadata object will include the fields that exist such as ip, user_agent, accept, and accept_encoding. |
query | Any parameters passed in the query string format, ?<parameter_name>=<parameter_value> appended to the URL for the source will be parsed and passed in. |
Examples
Reference Guides
The topic Set Up and Test an HTTP Endpoint Source up provides a basic example of how to set up your HTTP endpoint Source and typical Processors you would use.
Basic Example for Sending Data to an HTTP Source
Let's set up an HTTP source and use curl to send in a sample payload with a set of key-value pairs.
Create a new pipeline.
Add an HTTP source to the graph with
textdecoding.Save the source, then generate an API key.
Save the API key somewhere safe, such as a password manager
Copy the HTTP URL endpoint. It should follow the format
https://pipeline.mezmo.com/v1/%YOUR PIPELINE ID%Deploy the pipeline with just the HTTP source. You cand ignore the warnings for missing connections.
Start a tap on the output of the HTTP source by clicking on the right end of it and pressing the play button.
- Leave the tap running
Open a terminal on your computer
Enter this
curlcommand, replacing the%PIPELINE ID%and the%TOKEN%with the information from your newly created pipeline
You should see the message payload msg=Hello_world show in the tap view after a few seconds as plain text with no color coding.
If you want to make the message into a useful structured data object, you could also use the Parse Processor with a key-value pair parse algorithm to turn it into a JSON formatted schema.
Source Data Example with Metadata
Here's a payload representing an example of what the inbound HTTP payload looks like when including all of the metadata from headers and query parameters.
{ "message": { "availability_zone": "us-east-1b", "event": { "dest_ip": "10.1.3.81", "dest_port": 17778, "event_type": "netflow", "flow_id": 694693483348034, "netflow": { "age": 0, "bytes": 44, "end": "2023-06-13T22:31:41.163906+0000", "max_ttl": 242, "min_ttl": 242, "pkts": 1, "start": "2023-06-13T22:31:41.163906+0000" }, "proto": "TCP", "src_ip": "205.210.31.133", "src_port": 50813, "tcp": { "syn": true, "tcp_flags": "02" }, "timestamp": "2023-06-13T22:32:49.446239+0000" }, "event_timestamp": "1686695569", "firewall_name": "AWS-Network-Firewall-Multi-AZ-firewall" }, "metadata": { "headers": { "accept": "*/*", "accept-encoding": "gzip, deflate", "connection": "keep-alive", "content-length": "603586", "host": "pipeline", "user-agent": "python-requests/2.28.2", "x-consumer-id": "68b52623-9642-4322-9e27-8218d8c7bb37", "x-consumer-username": "p_1f799b82-53e2-11ee-a88d-26dab184329f", "x-credential-identifier": "7c419a2c-ccf7-480b-9a9e-3fd94e3f6b28", "x-forwarded-for": "76.253.171.67", "x-forwarded-host": "pipeline.mezmo.it", "x-forwarded-path": "/v1/1f799b82-53e2-11ee-a88d-26dab184329f", "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-kafka-key": "76.253.171.67", "x-pipeline-capture-metadata": "true", "x-pipeline-source-type": "http", "x-real-ip": "76.253.171.67" }, "query": { "test": "parameter" } }}