HTTP Client

Description

This is a pull source that periodically polls a remote HTTP endpoint to collect data. Unlike push sources where external systems send data to the Pipeline, the HTTP Client source actively requests data from a specified URL at a configurable interval.

You would typically use the HTTP Client source when:

• The service you want to collect data from does not support push-based delivery
• You need to poll health check, status, or metrics endpoints at regular intervals
• You want to ingest data from external REST APIs or HTTP services

This source is available for both SaaS and Edge deployments.

Configuration

Configuration Options

Authentication

The HTTP Client source supports two authentication methods:

Basic Authentication

Select basic as the authentication strategy and provide:

• User: The user for authentication.
• Password: The password for authentication.

Bearer Token Authentication

Select bearer as the authentication strategy and provide:

• Token: The bearer token for the Authorization header.

Decoding Options

• bytes: Treats the response body as plain text and places it in the message field as-is. Useful for text-based responses or when parsing with a subsequent processor.
• json: Parses the response body as JSON and places the resulting object in the message field. Useful when the endpoint returns JSON data.

Example Use Cases

Polling a REST API

To poll a REST API that returns JSON data:

1. Set Endpoint to your API URL, for example https://api.example.com/status.
2. Set Method to GET.
3. Set Decoding to json to parse the response automatically.
4. Set Authentication Strategy to bearer and provide your API token.
5. Adjust Scrape Interval based on how frequently the data changes.

Collecting Metrics from a Health Endpoint

To poll a service health check endpoint:

1. Set Endpoint to the health check URL.
2. Set Scrape Interval to a short interval like 15 or 30 seconds for frequent monitoring.
3. Add any required Headers for authentication or content negotiation.