> ## Documentation Index > Fetch the complete documentation index at: https://docs.mezmo.com/llms.txt > Use this file to discover all available pages before exploring further. # Update sink > Updates a sink of a pipeline ## OpenAPI ````yaml /pipeline.yaml put /pipeline/{pipeline_id}/sink/{pipeline_sink_id} openapi: 3.1.0 info: title: Pipeline API description: API for the Pipeline product version: 3.345.5 servers: - url: https://api.mezmo.com/v3 security: - PipelineServiceKey: [] tags: - name: Internal description: Routes not accessible by outside users - name: Experimental description: Experimental routes, not intended for public consumption - name: Account description: Perform account level actions - name: Alerting description: API to control alerts on pipelines - name: Data Classification & Profiling description: API for classifying and profiling data - name: Edge - Local Deploy description: User routes that are available to use with local Edge instances - name: Incident Mode description: Change or read state of pipeline (incident mode on/off) - name: Log Volume Reduction description: CRUD operations for log volume reduction - name: Metrics description: Usage metrics for pipelines - name: Pipeline Export description: Export pipeline to terraform - name: Pipeline Management description: CRUD Operations for pipeline - name: Pipeline Nodes description: CRUD operations for sources, processors, and destinations - name: Processor Groups description: CRUD operations for Processor Groups - name: Shared Sources description: Create and manage sources for use across multiple pipelines - name: Sampling and Simulation description: >- Create samples of data and run them through a simulated pipeline without affecting your published pipeline externalDocs: url: https://docs.mezmo.com/pipeline-api paths: /pipeline/{pipeline_id}/sink/{pipeline_sink_id}: put: tags: - Pipeline Nodes summary: Update sink description: Updates a sink of a pipeline parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_sink_id required: true requestBody: content: application/json: schema: type: object allOf: - if: properties: type: enum: - mezmo deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Mezmo Log Analysis description: Send log data to Mezmo Log Analysis additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true mezmo_host: type: string title: Mezmo Host description: The host for your Log Analysis environment default: null minLength: 1 ingestion_key: type: string title: Ingestion Key description: Ingestion key ui: display_type: mezmo-ingestion-key minLength: 1 query: type: object title: Query Parameters default: hostname: mezmo properties: hostname: type: string title: Hostname description: >- Hostname string or template to extract event values to form a hostname. Depending on the type of ingestion, this value is usually set from other places so the default value is often sufficient. If the value in the event is null, empty, or doesn't exist, when using a template then it will be set to "mezmo" default: mezmo minLength: 1 ui: display_type: data-template tags: type: array title: Tags description: >- List of tag strings or templates to attach to logs. Template must resolve to either an array of strings, or a string. Commas are treated as tag delimiters. items: type: string title: Tag description: Tag string or template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template ip: type: string title: IP description: IP address template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template mac: type: string title: MAC description: MAC address template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template message: title: Log Construction Scheme default: scheme: Explicit field selection allOf: - if: properties: scheme: enum: - Explicit field selection then: properties: line: type: string title: Line description: >- Template or field reference to use as the log line. Field reference can point to an object. minLength: 1 maxLength: 1024 meta_field: type: string title: Meta Field description: >- Field containing the meta object for the log pattern: >- ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: >- Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference maxLength: 512 app: type: string title: App description: App name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template file: type: string title: File description: File name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template timestamp_field: type: string title: Timestamp description: >- Field containing the timestamp for the log pattern: >- ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: >- Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference maxLength: 512 env: type: string title: Env description: >- Environment name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template properties: scheme: type: string title: Scheme description: How to construct the log message. enum: - Explicit field selection - Message pass-through default: Explicit field selection required: - scheme required: - mezmo_host - ingestion_key - if: properties: type: enum: - mezmo-archive deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Mezmo Archive Destination description: >- Store logs for future imports into mezmo pipeline or log analysis. additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: >- The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true destination: title: Destination description: '' allOf: - if: properties: archive_provider: enum: - Azure then: properties: container_name: type: string title: Container Name minLength: 1 description: >- The name of the container for blob storage. connection_string: type: string title: Connection String minLength: 1 description: >- A connection string for the account that contains an access key required: - container_name - connection_string - if: properties: archive_provider: enum: - S3 then: properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret bucket: type: string title: Bucket pattern: >- (?!(^xn--|.+-s3alias$))^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ description: >- The S3 bucket name. Do not include a leading s3:// or a trailing / region: type: string title: Region minLength: 1 description: >- The name of the AWS region that is targeted. required: - access_key_id - secret_access_key - bucket - region properties: archive_provider: type: string title: Archive Provider description: The storage destination for your data. enum: - Azure - S3 default: Azure required: - archive_provider required: - destination - if: properties: type: enum: - http deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: HTTP description: Send data to a HTTP endpoint additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: >- The full URI to make HTTP requests to. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none auth: title: Authentication description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 description: The basic authentication user password: type: string title: Password minLength: 1 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none headers: type: array title: HTTP Headers description: >- An array of key/value objects for the headers to be sent with the request. items: type: object title: Headers description: >- A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 description: The value of the header. required: - header_name - header_value advanced_options: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-http-advanced-options title: Advanced Options description: >- Settings available for fine-tuning the destination behavior. additionalProperties: false dependencies: payload_prefix: - payload_suffix payload_suffix: - payload_prefix properties: max_bytes: type: integer title: Payload Size description: >- The maximum number of uncompressed bytes when batching data to the destination. minimum: 1 maximum: 2097152 timeout_secs: type: integer title: Timeout Seconds description: >- The number of seconds before a destination write timeout. minimum: 5 method: type: string enum: - post - put - patch - delete - get - head - options - trace title: HTTP Method description: The HTTP method to use for the destination. payload_prefix: type: string title: Payload Prefix description: >- Add a prefix to the payload. Only used for serialized JSON chunks. This option also requires "Payload Suffix" to form a valid JSON string. examples: - '{"event":"' - '{"extra_prop": true, "our_thing":"' minLength: 1 payload_suffix: type: string title: Payload Suffix description: >- Used in combination with "Payload Prefix" to form valid JSON from the payload. examples: - '}' - '"extra_prop": true }' minLength: 1 proxy: type: object additionalProperties: false title: Proxy Settings description: Configure proxy settings for the destination. properties: enabled: type: boolean title: Enabled description: Turns proxying on/off default: false endpoint: type: string minLength: 1 title: HTTP(S) Endpoint description: HTTP or HTTPS Endpoint to use for traffic. format: uri errorMessage: format: Must be a valid URI hosts_bypass_proxy: type: array minItems: 1 title: Hosts to Bypass description: >- A list of hosts to bypass proxying. Can be specified as a domain name, IP address, or CIDR block. Wildcards are supported as a dot (.) in domain names, or as a star (*) to match all hosts. examples: - 93.184.216.34 - ignorehost.example.com - .ignoreall.example.com - 93.184.216.0/24 items: type: string title: Host description: Domain name, IP address, or CIDR block. minLength: 1 tls_protocols: type: array title: TLS Protocols description: >- A list of ALPN protocols to use during TLS negotiation. They are attempted in the order they appear. items: type: string title: Protocol description: An ALPN protocol name minLength: 1 minItems: 1 rate_limiting: type: object additionalProperties: false title: Rate Limiting description: >- Settings for controlling rate limiting to the destination. properties: request_limit: type: integer minimum: 1 title: Request Limit description: >- The max number of requests allowed within the specified "Duration Seconds" setting. duration_secs: type: integer default: 1 title: Duration Seconds description: >- The window of time used to apply "Request Limit." minimum: 1 required: - uri - encoding - compression - if: properties: type: enum: - cloudwatch-logs deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: AWS Cloudwatch Logs description: Publish log events to AWS Cloudwatch Logs additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none group_name: type: string title: Group Name minLength: 1 description: >- The name of the log group for the targeted log stream. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the targeted log stream. required: - auth - encoding - compression - group_name - region - stream_name - if: properties: type: enum: - cloudwatch-metrics deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: AWS Cloudwatch Metrics description: Send metric events to AWS Cloudwatch Metrics additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none namespace: type: string title: Namespace minLength: 1 description: >- Name for the container that will isolate metrics from one another. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - auth - compression - namespace - region - if: properties: type: enum: - sqs deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: AWS Simple Queue Service (SQS) description: Publish log events to AWS Simple Queue Service additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text sqs_queue_url: type: string title: SQS Queue URL format: url errorMessage: format: Must be a valid URL description: >- The URL of the SQS queue that the data is publishing to. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - auth - encoding - sqs_queue_url - region - if: properties: type: enum: - notification-channel deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Notification Channel description: Send a Message to a 3rd Party Service. additionalProperties: false properties: service: title: Service description: >- Configure a service to receive the payload. Currently supports Slack, PagerDuty, Webhook or Mezmo (Log Analysis). Where allowed, messagetemplating can be used. To use a template, surround field names with double curly braces, e.g., {{.my_field}} or {{."my-object".thing}}. See [documentation](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) for more examples. additionalProperties: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: >- The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 allOf: - if: properties: name: enum: - slack required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: >- The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The text to be sent as a Slack message. required: - uri - message_text - if: properties: name: enum: - pager_duty required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: >- The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 summary: type: string title: Summary description: >- Summarize the alert details for PagerDuty. minLength: 1 ui: display_type: data-template severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. source: type: string title: Source description: The source of the alert. minLength: 1 ui: display_type: data-template routing_key: type: string title: Routing Key description: The service integration routing key. minLength: 1 ui: display_type: secret event_action: type: string enum: - trigger - acknowledge - resolve title: Event Action description: >- The event action associated with the alert. required: - uri - summary - severity - source - routing_key - event_action - if: properties: name: enum: - webhook required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: >- The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 method: type: string title: HTTP Method description: The HTTP method to use for the request. enum: - post - put - patch - delete - get - head - options - trace ui: default_hint: post message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: >- The Webhook payload to send. This can be a text string or stringified JSON. If the message can be parsed as JSON, it will be sent as such. headers: type: array title: HTTP Headers description: >- A list of key/value pairs to be included in the HTTP request. items: type: object title: Headers description: >- A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 maxLength: 1024 description: The value of the header. required: - header_name - header_value maxItems: 20 auth: title: Authentication Options description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 description: The basic authentication user password: type: string title: Password minLength: 1 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: >- Choose basic or token-based authentication. default: none required: - uri - message_text - if: properties: name: enum: - log_analysis required: - name then: properties: severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. default: INFO subject: type: string title: Subject description: >- A string to be used as the subject of the alert. minLength: 1 maxLength: 200 ui: display_type: data-template body: type: string title: Body description: >- A string to be used as the alert message body. minLength: 1 maxLength: 1024 ui: display_type: data-template ingestion_key: type: string title: Ingestion Key description: >- The key required to communicate to Log Analysis for sending the alert. ui: display_type: mezmo-ingestion-key minLength: 1 required: - severity - subject - body - ingestion_key properties: name: title: Name description: >- The name of the service that will receive the notification. ui: display_type: list errorMessage: must be a valid service name oneOf: - label: Slack enum: - slack - label: PagerDuty enum: - pager_duty - label: Webhook enum: - webhook - label: Mezmo Log Analysis enum: - log_analysis required: - name throttling: type: object title: Throttling description: >- Only allow a certain number of notifications to be sent within a given time window. properties: window_secs: type: integer title: Window Seconds description: >- The time frame during which the number of notifications (set by the Threshold) is permitted. default: 60 minimum: 1 threshold: type: integer title: Threshold description: >- The maximum number of notifications allowed over the given time window set by "Window Seconds" (default: 1). default: 1 minimum: 1 required: - service - throttling - if: properties: type: enum: - azure-blob-storage deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Azure Blob Storage description: Publish log events to Azure Blob Storage additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: >- The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none container_name: type: string title: Container Name minLength: 1 description: The name of the container for blob storage. connection_string: type: string title: Connection String minLength: 1 description: >- A connection string for the account that contains an access key prefix: type: string title: Prefix minLength: 1 description: A prefix to be applied to all object keys ui: display_type: data-template file_consolidation: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-azure-blob-file-consolidation title: File Consolidation description: >- This sink writes many small files out to azure blob storage. Enabling this process will allow the automatic consolidation of these small files into larger files of your choosing. This process will enable upon deployment and run on the chosen interval from `Processing Interval` creating files named `merged_[timestamp].log` where `timestamp` is the time since epoch when the actual file was created. The process will recursively access all files under the `Base Path` to handle merging sub-directory logging structures. additionalProperties: false properties: enabled: type: boolean default: false title: Enabled description: Toggles whether the process is enabled. process_every_seconds: type: integer title: Processing Interval (seconds) description: How often to run the consolidation process. default: 600 minimum: 300 maximum: 3600 requested_size_bytes: type: integer title: Requested File Size (bytes) description: The requested size of the consolidated files. default: 500000000 minimum: 50000000 maximum: 10000000000 base_path: type: string title: Base Path default: '' maxLength: 512 description: >- The path from the container to begin recursivelylooking for files to merge. A blank value indicates root.Merged files will only contain data from the respectivefolder. required: - encoding - compression - container_name - connection_string - if: properties: type: enum: - kinesis-firehose deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: AWS Kinesis Data Firehose description: Publish log events to AWS Kinesis Data Firehose additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the Kinesis Firehose stream required: - auth - encoding - compression - region - stream_name - if: properties: type: enum: - kinesis-streams deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: AWS Kinesis Streams description: Publish log events to AWS Kinesis Streams additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none partition_key_field: type: string title: Partition Key Field pattern: >- ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ description: >- The field in the log whose value is used as Kinesis' partition key. errorMessage: pattern: >- Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the targeted log stream. required: - auth - encoding - compression - region - stream_name - if: properties: type: enum: - elasticsearch deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Elasticsearch description: Publish log events to Elasticsearch additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none auth: title: ElasticSearch Auth Strategy description: ElasticSearch authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - aws then: properties: access_key_id: type: string title: Access Key ID minLength: 1 maxLength: 1024 description: AWS access key ID ui: display_type: secret secret_access_key: type: string title: Secrete Access Key minLength: 1 maxLength: 1024 description: AWS secrete access key ui: display_type: secret region: type: string title: Region minLength: 1 maxLength: 128 description: >- The name of the AWS region that is targeted. required: - access_key_id - secret_access_key - region properties: strategy: type: string title: Strategy description: The ES authentication strategy to use enum: - basic - aws required: - strategy endpoints: type: array title: Endpoints uniqueItems: true minItems: 1 description: An array of ElasticSearch endpoints items: type: string title: URL format: url errorMessage: format: Must be a valid URL description: The full Elasticsearch URL to send data to. minLength: 1 maxLength: 1024 pipeline: type: string title: Pipeline minLength: 1 description: Name of an ES ingest pipeline to include. index: type: string minLength: 1 maxLength: 512 title: Custom Index description: >- Index to use when writing ES events (default = mezmo-%Y.%m.%d) default: mezmo-%Y.%m.%d required: - compression - auth - endpoints - if: properties: type: enum: - gcp-cloud-monitoring deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: GCP Cloud Monitoring description: Publish metrics events to GCP Cloud Monitoring additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. resource_type: type: string title: Resource Type minLength: 1 description: >- The `monitored-resource` type as defined in Monitoring. resource_labels: type: array title: Resource Labels description: >- An array of key/val objects describing labels of the Monitoring resource. items: type: object title: Label/Value description: >- A key/val object describing a label name and its value. additionalProperties: false properties: label_name: type: string title: Label Name minLength: 1 description: The name of the label. label_value: type: string title: Label Value minLength: 1 description: The value of the label. required: - label_name - label_value required: - credentials_json - project_id - resource_type - if: properties: type: enum: - gcp-cloud-storage deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: GCP Cloud Storage description: Publish log events to GCP Cloud Storage additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: >- The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file encoding: type: string description: >- Dictates how the data will be serialized before storing. enum: - json - ndjson - text title: Encoding default: text bucket: type: string title: Bucket minLength: 1 description: >- The name of the bucket in GCP where the data will be stored. compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none bucket_prefix: type: string title: Bucket Prefix minLength: 1 description: >- The prefix applied to the bucket name, giving the appearance of having directories. required: - credentials_json - encoding - bucket - compression - if: properties: type: enum: - gcp-cloud-operations deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: GCP Cloud Operations description: Publish log events to GCP Cloud Operations additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file log_id: type: string title: Log ID minLength: 1 description: Concise reference for the log stream name. project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. resource_type: type: string title: Resource Type minLength: 1 description: >- The `monitored-resource` type as defined in Monitoring. resource_labels: type: array title: Resource Labels description: >- An array of key/val objects describing labels of the Monitoring resource. items: type: object title: Label/Value description: >- A key/val object describing a label name and its value. additionalProperties: false properties: label_name: type: string title: Label Name minLength: 1 description: The name of the label. label_value: type: string title: Label Value minLength: 1 description: The value of the label. required: - label_name - label_value required: - credentials_json - log_id - project_id - resource_type - if: properties: type: enum: - gcp-cloud-pubsub deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: GCP Cloud PubSub description: Publish log events to GCP Cloud PubSub additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file encoding: type: string description: >- Dictates how the data will be serialized before storing. enum: - json - text title: Encoding default: text project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. topic: type: string title: Topic minLength: 1 description: The name of the topic to publish messages to. required: - credentials_json - encoding - project_id - topic - if: properties: type: enum: - honeycomb-logs deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Honeycomb Logs description: Send log data to Honeycomb additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Honeycomb API key description: Your Honeycomb API key minLength: 1 maxLength: 512 ui: display_type: secret dataset: type: string title: Dataset description: The name of the targeted dataset. minLength: 1 maxLength: 512 required: - api_key - dataset - if: properties: type: enum: - indexed-search deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Mezmo Indexed Search description: >- Publish log events to Mezmo's data store for Indexed Search capabilities additionalProperties: true properties: search_field: type: string title: Full Text Search Field pattern: >- ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: .line maxLength: 256 description: >- If your data reaches this stage as a plain-text string, the entire payload will be indexed for full-text searching. If your data reaches this stage as an object, the field you enter here will be used as the full-text search field. Nested fields are allowed. errorMessage: pattern: >- Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference timestamp_field: type: string title: Timestamp Field pattern: >- ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: timestamp maxLength: 256 description: >- If your data reaches this stage as an object, this field represents the timestamp value to be cataloged for searching. If the field does not exists the current time will be used. We accept UNIX timestamps (in milliseconds), ISO8601/RFC3389, and RFC2822 formats. errorMessage: pattern: >- Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - search_field - timestamp_field - if: properties: type: enum: - kafka deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: title: Kafka description: Publish log events to Kafka additionalProperties: type: string allOf: - if: properties: sasl_enabled: enum: - true then: properties: sasl_mechanism: type: string enum: - SCRAM-SHA-512 - SCRAM-SHA-256 - PLAIN title: SASL description: The Kafka SASL/SCRAM mechanism default: SCRAM-SHA-512 sasl_username: type: string title: Username description: The SASL/SCRAM username minLength: 1 sasl_password: type: string title: Password description: The SASL/SCRAM username ui: display_type: secret minLength: 1 required: - sasl_mechanism - sasl_username - sasl_password - if: properties: tls_enabled: enum: - true then: properties: tls_ca_certificate: type: string title: TLS CA Certificate Chain minLength: 1 description: The CA certificate chain in the PEM format ui: display_type: secret-file format: x509_certificate errorMessage: format: Must be a valid PEM or DER certificate tls_verify_certificate: type: boolean title: TLS Verify Certificate description: Verify the TLS certificates default: true required: - tls_verify_certificate properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - lz4 - snappy - zstd - none title: Compression default: none event_key_field: type: string title: Event Key Field minLength: 1 description: >- The field in the log whose value is used as Kafka's event key. brokers: type: array title: Brokers uniqueItems: true minItems: 1 description: >- An array of key/val objects for each host/port of a Kafka broker. items: type: object title: Host/Port description: The host and port for the Kafka broker. additionalProperties: false properties: host: title: Host description: The hostname or IP address of the broker. anyOf: - type: string format: ipv4 errorMessage: format: Must be a valid IPv4 address - type: string format: hostname errorMessage: format: Must be a valid hostname port: type: number title: Port description: The port that the broker listens on. required: - host - port topic: type: string title: Topic minLength: 1 description: The name of the topic to publish to. group_id: type: string title: Group ID minLength: 1 description: The consumer group identifier sasl_enabled: type: boolean title: SASL/SCRAM Enabled description: Enable SASL/SCRAM authentication with Kafka default: false tls_enabled: type: boolean title: Use TLS description: Enable TLS during connections to the remote default: false required: - encoding - compression - brokers - topic - if: properties: type: enum: - loki deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Loki description: Publish log events to Loki additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: title: Loki Auth Strategy description: Loki authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token (passed as-is) ui: display_type: secret required: - token properties: strategy: type: string title: Strategy description: >- The authentication strategy to use (only basic supported) enum: - basic - bearer required: - strategy encoding: type: object title: Encoding description: Configures how event are encoded properties: codec: type: string title: Codec enum: - json - text required: - codec endpoint: type: string title: Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Loki base URL path: type: string title: Path minLength: 1 maxLength: 512 description: >- The path appended to the Loki base URL(defaults to /loki/api/v1/push) default: /loki/api/v1/push labels: type: array title: Loki Labels description: Key/Value pair used to describe Loki data items: type: object title: Labels description: Label name and values additionalProperties: false properties: label_name: type: string title: Label Name description: The name of the label minLength: 1 maxLength: 512 label_value: type: string title: Label Value description: The value of the label minLength: 1 maxLength: 512 required: - label_name - label_value required: - auth - encoding - endpoint - labels - if: properties: type: enum: - new-relic deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: New Relic description: Publish to New Relic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true account_id: type: string title: New Relic Account ID description: Account ID for your New Relic account minLength: 1 maxLength: 1024 ui: display_type: secret api: type: string title: API description: New Relic API endpoint (events not supported) enum: - logs - metrics default: logs license_key: type: string title: New Relic License Key description: License key for your New Relic account minLength: 1 maxLength: 1024 ui: display_type: secret required: - account_id - api - license_key - if: properties: type: enum: - prometheus-remote-write deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Prometheus Remote Write description: Publish metrics to Prometheus Remote Write endpoint additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Endpoint format: uri errorMessage: format: Must be a valid URI description: >- The full URI to make HTTP requests to. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. Example: http://example.org:8080/api/v1/push minLength: 1 maxLength: 2000 auth: title: Authentication description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none required: - strategy required: - endpoint - if: properties: type: enum: - pulsar deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Pulsar description: Publish log events to Apache Pulsar additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text url: type: string title: URL format: url errorMessage: format: Must be a valid URL description: The full Pulsar URL to send data to. topic: type: string title: Topic minLength: 1 description: The name of the Pulsar topic to publish to. required: - encoding - url - topic - if: properties: type: enum: - redis deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Redis description: Publish log events to Redis additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text url: type: string title: URL format: uri errorMessage: format: Must be a valid URL description: The full Redis URL to send data to. key: type: string title: Key minLength: 1 description: The name of the Redis key to write data to. required: - encoding - url - key - if: properties: type: enum: - splunk-hec-logs deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Splunk HTTP Event Collector description: Publish log events to Splunk HTTP Event Collector additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none url: type: string title: Endpoint format: url errorMessage: format: Must be a valid URL description: >- The base URL for the Splunk instance. The collector path, such as `/services/collector/events`, will be automatically inferred from the destination's configuration. token: type: string title: Token minLength: 1 ui: display_type: secret description: The default token to authenticate to Splunk HEC. tls_verify_certificate: type: boolean title: Verify TLS Certificate description: >- If unchecked, the TLS certificate of the remote host will not be verified. **This is insecure**, but may be useful for testing or when evaluating Splunk in a trial instance. default: true host_field: type: string title: Host Field description: >- The field that contains the hostname to include in the event. default: metadata.host timestamp_field: type: string title: Timestamp Field description: >- The field that contains the timestamp to include in the event. default: metadata.time pattern: >- ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 maxLength: 512 errorMessage: pattern: >- Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: feature_flag: pipeline-splunk-sink-v2 display_type: field-reference source: title: Source description: >- The source of events sent to this sink. This is typically the filename the logs originated from. Use the field path ".metadata.source" to use the upstream source value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: >- ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: >- Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: >- Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - 'null' required: - value_type sourcetype: title: Source Type description: >- The sourcetype of events sent to this sink. Use the field path ".metadata.sourcetype" to use the upstream sourcetype value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: >- ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: >- Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: >- Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - 'null' required: - value_type index: title: Index description: >- The name of the index to send events to. Use the field path ".metadata.index" to use the upstream index value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: >- ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: >- Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: >- Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - 'null' required: - value_type required: - compression - url - token - source - sourcetype - index - if: properties: type: enum: - sumo-logic-logs deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Sumo Logic Logs description: Publish log data to Sumo Logic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: >- The Sumo Logic Collector URL that receives the data ui: display_type: secret compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none category: type: string minLength: 1 maxLength: 512 title: Category description: >- Category to use when publishing data to Sumo Logic (default = mezmo_default_category) default: mezmo_default_category required: - endpoint - compression - if: properties: type: enum: - sumo-logic-metrics deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Sumo Logic Metrics description: Publish metrics data to Sumo Logic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: >- The Sumo Logic Collector URL that receives the data ui: display_type: secret compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none category: type: string minLength: 1 maxLength: 512 title: Category description: >- Category to use when publishing data to Sumo Logic (default = mezmo_default_category) default: mezmo_default_category required: - endpoint - compression - if: properties: type: enum: - vector deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Vector description: Publish log events to Vector additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none vector_ip: type: string title: Vector IP format: ipv4 description: The IP portion of the Vector host vector_port: type: number title: Vector Port description: The port that the Vector host listens on. version: type: string enum: - '1' - '2' title: Version default: '2' description: >- The version of the Vector sink which must match the source version. required: - compression - vector_ip - vector_port - version - if: properties: type: enum: - s3 deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: AWS S3 description: Publish events as objects in AWS S3 additionalProperties: false errorMessage: properties: bucket: must be a valid bucket name tags: must be a valid tag identifier properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: >- The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key bucket: type: string title: Bucket pattern: >- (?!(^xn--|.+-s3alias$))^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ description: >- The S3 bucket name. Do not include a leading s3:// or a trailing / prefix: type: string title: Prefix description: A prefix to apply to all object key names. default: / ui: display_type: data-template tags: type: array maxItems: 10 title: Tags description: >- A set of tags (key/value pairs) to apply to the object in S3. items: type: object properties: key: type: string title: Name description: The tag name pattern: ^[A-Za-z0-9_.:=+@/\-\s]*$ minLength: 1 maxLength: 128 value: type: string title: Value description: The tag value default: '' pattern: ^[A-Za-z0-9_.:=+@/\-\s]*$ minLength: 0 maxLength: 256 required: - key - value encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. file_consolidation: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-s3-file-consolidation title: File Consolidation description: >- This sink writes many small files out to S3. Enabling this process will allow the automatic consolidation of these small files into larger files of your choosing. This process will enable upon deployment and run on the chosen interval from `Processing Interval` creating files named `merged_[timestamp].log` where `timestamp` is the time since epoch. *NOTE*: only `text` and `ndjson` files written by the sink are consolidated. additionalProperties: false properties: enabled: type: boolean default: false title: Enabled description: Toggles whether the process is enabled. process_every_seconds: type: integer title: Processing Interval (seconds) description: How often to run the consolidation process. default: 600 minimum: 300 maximum: 3600 requested_size_bytes: type: integer title: Requested File Size (bytes) description: The size of the requested files. default: 500000000 minimum: 50000000 maximum: 10000000000 base_path: type: string title: Base Path default: '' maxLength: 512 description: >- The path from the container to begin recursivelylooking for files to merge. A blank value indicates root.Merged files will only contain data from the respectivefolder. required: - auth - bucket - prefix - encoding - compression - region - if: properties: type: enum: - blackhole deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Blackhole description: Drop data arriving at this destination additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true - if: properties: type: enum: - datadog-logs deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Datadog Logs description: Publish log events to Datadog additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Datadog API Key description: Datadog log application API key minLength: 1 ui: display_type: secret compression: type: string description: >- The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: gzip site: type: string description: The Datadog site (region) to send logs to. enum: - us1 - us3 - us5 - eu1 title: Datadog Site default: us1 required: - api_key - compression - site - if: properties: type: enum: - datadog-metrics deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Datadog Metrics description: >- Publish metrics events to Datadog. Sending custom metrics to Datadog may result in extra charges on your Datadog bill. Please go to https://docs.datadoghq.com/metrics/custom_metrics/ for more information on Datadog custom metrics pricing. additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Datadog API Key description: Datadog metrics application API key minLength: 1 maxLength: 512 ui: display_type: secret site: type: string description: The Datadog site (region) to send metrics to. enum: - us1 - us3 - us5 - eu1 title: Datadog Site default: us1 required: - api_key - site - if: properties: type: enum: - clickhouse deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: Clickhouse description: Deliver log data to the ClickHouse database additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string minLength: 1 maxLength: 2048 title: Endpoint description: Endpoint of the Clickhouse server format: uri errorMessage: format: Must be a valid URI auth: title: Auth Strategy description: Authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 2048 description: >- The bearer token to use for authentication ui: display_type: secret required: - token properties: strategy: type: string title: Strategy description: The ES authentication strategy to use enum: - basic - bearer required: - strategy table: type: string title: Table description: The table that data is inserted into pattern: ^[a-zA-Z0-9_]+$ errorMessage: Invalid table name ui: display_type: data-template database: type: string title: Database description: >- The database that contains the table that data is inserted into pattern: ^[a-zA-Z0-9_]+$ errorMessage: Invalid database name ui: display_type: data-template compression: type: string enum: - gzip - none - snappy - zlib - zstd title: Compression description: The compression algorithm to use default: gzip skip_unknown_fields: type: boolean title: Skip Unknown Fields description: >- Allows ClickHouse to discard fields not present in the table schema default: false required: - endpoint - auth - table - if: properties: type: enum: - opentelemetry deploy_type: anyOf: - enum: - saas - type: 'null' required: - type then: properties: user_config: type: object title: OpenTelemetry description: Publish data to OpenTelemetry additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: >- The OpenTelemetry Collector URL that receives the data auth: title: Authentication description: Configures HTTP authentication additionalProperties: type: string allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none headers: type: array title: HTTP Headers description: >- An array of key/value objects for the headers to be sent with the request. items: type: object title: Headers description: >- A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 description: The value of the header. required: - header_name - header_value compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none required: - endpoint - compression properties: title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: 'null' errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: 'null' errorMessage: invalid description type: type: string enum: - mezmo - mezmo-archive - http - cloudwatch-logs - cloudwatch-metrics - sqs - notification-channel - azure-blob-storage - kinesis-firehose - kinesis-streams - elasticsearch - gcp-cloud-monitoring - gcp-cloud-storage - gcp-cloud-operations - gcp-cloud-pubsub - honeycomb-logs - indexed-search - kafka - loki - new-relic - prometheus-remote-write - pulsar - redis - splunk-hec-logs - sumo-logic-logs - sumo-logic-metrics - vector - s3 - blackhole - datadog-logs - datadog-metrics - clickhouse - opentelemetry deploy_type: type: string enum: - saas - local default: saas user_config: type: object generation_id: type: integer inputs: type: array items: type: string required: - type - user_config - generation_id - inputs required: true responses: '200': description: Default Response components: securitySchemes: PipelineServiceKey: type: apiKey name: Authorization in: header description: 'Enter your token in the format: Token mytokengoeshere' ````