openapi: 3.1.0 info: title: Pipeline API description: API for the Pipeline product version: 3.345.5 components: securitySchemes: PipelineServiceKey: type: apiKey name: Authorization in: header description: "Enter your token in the format: Token mytokengoeshere" schemas: http-error-item: type: object title: HTTP Error Item additionalProperties: true properties: keyword: type: string maxLength: 30 description: An indication of the type of rule or error that caused the request to fail instance_path: type: string maxLength: 200 nullable: true default: null description: JSON Pointer to the location in the data instance schema_path: type: string maxLength: 200 nullable: true default: null description: JSON Pointer to the location of the failing keyword in the schema params: type: object additionalProperties: true description: Additional information about the error that can be used to generate error messages property_name: type: string maxLength: 100 nullable: true default: null description: Set for errors in `propertyNames` keyword schema message: type: string maxLength: 200 description: A message describing the nature of the error or failure required: - keyword - params - message http-error: type: object title: HTTP Error description: A normalized representation of an error that is returned from an http request properties: status: type: integer minimum: 300 maximum: 599 format: int32 description: The HTTP status code the error is associated with context: type: string maxLength: 15 description: The portion of the inbound request which triggered the validation failure enum: - body - querystring - params - headers code: type: string maxLength: 35 description: A textual classifier of the error or failure message: type: string maxLength: 200 description: A message describing the nature of the error or failure errors: type: array maxItems: 200 description: List of metadata objects describing validation failures or contextual information items: $ref: "#/components/schemas/http-error-item" required: - status - code - message validation-error: $ref: "#/components/schemas/http-error" type: object title: Validation Error description: >- A normalized representation of an error that is returned from an http request in response to failed input validation properties: context: type: string maxLength: 15 description: The portion of the inbound request which triggered the validation failure enum: - body - querystring - params - headers legacy-validation-error: description: A normalized representation of an error that is returned from an http request type: object title: Validation Error (Legacy) properties: status: type: integer minimum: 300 maximum: 599 format: int32 description: The HTTP status code the error is associated with context: type: string maxLength: 15 description: The portion of the inbound request which triggered the validation failure enum: - body - querystring - params - headers code: type: string maxLength: 35 description: A textual classifier of the error or failure message: type: string maxLength: 200 description: A message describing the nature of the error or failure errors: type: array maxItems: 200 description: List of metadata objects describing validation failures items: additionalProperties: true type: object title: HTTP Error Item (Legacy) properties: keyword: type: string maxLength: 30 description: An indication of the type of rule or error that caused the request to fail params: type: object additionalProperties: true description: Additional information about the error that can be used to generate error messages message: type: string maxLength: 200 description: A message describing the nature of the error or failure schemaPath: type: string maxLength: 200 nullable: true default: null description: JSON Pointer to the location of the failing keyword in the schema instancePath: type: string maxLength: 200 nullable: true default: null description: JSON Pointer to the location in the data instance propertyName: type: string maxLength: 100 nullable: true default: null description: where the validation error occurred required: - keyword - params - message required: - status - code - message legacy-http-error-item: additionalProperties: true type: object title: HTTP Error Item (Legacy) properties: keyword: type: string maxLength: 30 description: An indication of the type of rule or error that caused the request to fail params: type: object additionalProperties: true description: Additional information about the error that can be used to generate error messages message: type: string maxLength: 200 description: A message describing the nature of the error or failure schemaPath: type: string maxLength: 200 nullable: true default: null description: JSON Pointer to the location of the failing keyword in the schema instancePath: type: string maxLength: 200 nullable: true default: null description: JSON Pointer to the location in the data instance propertyName: type: string maxLength: 100 nullable: true default: null description: where the validation error occurred required: - keyword - params - message alert-config: type: object title: Alert Configuration description: Configure an alert to be triggered when conditions are met. additionalProperties: false properties: general: type: object title: General description: Basic properties that describe the alert. additionalProperties: false properties: name: type: string title: Name minLength: 1 maxLength: 200 description: The name of the alert description: type: string title: Description minLength: 1 maxLength: 1024 description: An optional description describing what the alert is for. required: - name evaluation: title: Evaluation description: Multiple events (Log or Metric) will be collected into a single event (via aggregation) over a specified time window. That event is what the alert conditions will evaluate against. This section specifies how that aggregation should work. For Log event types, a `custom` JavaScript operation is required because the event shape is not standardized. additionalProperties: type: string title: Operation description: Specifies the type of aggregation operation to use with the window type and duration. ui: display_type: list oneOf: - const: DEFAULT label: add description: Adds new values together if they are of the same type. For counters and gauges the values are added together. For sets, this is a union of the elements. For distribution, the samples are concatenated. For histograms, values in the same bucket will be added together. - const: SUM label: sum description: Values are added together. Works for the GAUGE and COUNT metric types. - const: MIN label: minimum description: Get the minimum value for a series of events. Works for the GAUGE and COUNT metric types. - const: MAX label: maximum description: Get the maximum value for a series of events. Works for the GAUGE and COUNT metric types. - const: AVG label: average description: Values are averaged together. Works for the GAUGE and COUNT metric types. - const: SET_INTERSECTION label: set intersection description: The aggregate produces a set of unique values. - const: DIST_CONCAT label: distribution concatenation description: The aggregate produces a list of values combined together. - const: CUSTOM label: custom description: Use a custom script to perform aggregation. allOf: - if: allOf: - type: object properties: event_type: enum: - log required: - event_type - type: object properties: alert_type: not: enum: - absence required: - alert_type then: properties: operation: title: Operation description: Specifies the type of aggregation operation to use with the window type and duration. ui: display_type: list oneOf: - label: custom description: Use a custom script to perform aggregation. enum: - CUSTOM default: CUSTOM group_by: type: array minItems: 1 maxItems: 50 uniqueItems: true title: Group By Field Paths description: When aggregating, group events based on matching values from each of these field paths. Supports nesting via dot-notation. items: type: string maxLength: 100 title: Field description: The dot-notated property path. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ ui: default_hint: - .name - .namespace - .tags event_timestamp: type: string title: Event Timestamp description: The path to a field on the event that contains a epoch timestamp value. If an event does not have a timestamp field, events will be associated to the wall clock value when the event is processed. maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: default_hint: timestamp pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ required: - operation - if: allOf: - type: object properties: event_type: enum: - metric required: - event_type - type: object properties: alert_type: not: enum: - absence required: - alert_type then: properties: operation: title: Operation description: Specifies the type of aggregation operation to use with the window type and duration. ui: display_type: list oneOf: - label: add description: Adds new values together if they are of the same type. For counters and gauges the values are added together. For sets, this is a union of the elements. For distribution, the samples are concatenated. For histograms, values in the same bucket will be added together. enum: - DEFAULT - label: sum description: Values are added together. Works for the GAUGE and COUNT metric types. enum: - SUM - label: minimum description: Get the minimum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MIN - label: maximum description: Get the maximum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MAX - label: average description: Values are averaged together. Works for the GAUGE and COUNT metric types. enum: - AVG - label: set intersection description: The aggregate produces a set of unique values. enum: - SET_INTERSECTION - label: distribution concatenation description: The aggregate produces a list of values combined together. enum: - DIST_CONCAT - label: custom description: Use a custom script to perform aggregation. enum: - CUSTOM default: SUM event_timestamp: not: type: string title: Event Timestamp description: The path to a field on the event that contains a epoch timestamp value. If an event does not have a timestamp field, events will be associated to the wall clock value when the event is processed. maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ required: - operation - if: properties: operation: enum: - CUSTOM required: - operation then: properties: script: type: string title: Script description: A custom JavaScript function that will control the aggregation. At the time of flushing, this aggregation will become the emitted event. ui: display_type: code-editor options: language: js minLength: 1 maxLength: 5000 default: > // Receives the current event, metadata and an accumulator object. Logic can be performed // on the event properties and added to the accumulator object for later analysis. // The accumulator is persisted, and will become the emitted event when // conditions are true, OR if the time window naturally expires. Those same conditions // will also be evaluated against the emitted event to determine if the alert should // be triggered. function alertAggregation(accum, event, metadata) { // This function *must* return the accumulated event return accum } required: - script - if: properties: alert_type: enum: - threshold required: - alert_type then: properties: conditional: title: Alert Conditions description: Evaluate these conditions against the aggregated event, and trigger an alert if they are met. If the conditions are true, it will also flush the current aggregated event and end the time window. allOf: - $ref: "#/components/schemas/alert-condition-group-threshold" ui: display_type: rules-engine options: operators: alert-condition-group-threshold - if: properties: alert_type: enum: - change required: - alert_type then: properties: conditional: title: Alert Conditions description: Evaluate these conditions against the aggregated event, and trigger an alert if they are met. If the conditions are true, it will also flush the current aggregated event and end the time window. allOf: - $ref: "#/components/schemas/alert-condition-group-change" ui: display_type: rules-engine options: operators: alert-condition-group-change - if: properties: alert_type: enum: - absence required: - alert_type then: properties: conditional: title: Alert Conditions description: No conditions are necessary for alerting on the absence of events. not: anyOf: - $ref: "#/components/schemas/alert-condition-group-threshold" - $ref: "#/components/schemas/alert-condition-group-change" ui: display_type: rules-engine options: hidden: true operation: not: type: string title: Operation description: Specifies the type of aggregation operation to use with the window type and duration. ui: display_type: list oneOf: - label: add description: Adds new values together if they are of the same type. For counters and gauges the values are added together. For sets, this is a union of the elements. For distribution, the samples are concatenated. For histograms, values in the same bucket will be added together. enum: - DEFAULT - label: sum description: Values are added together. Works for the GAUGE and COUNT metric types. enum: - SUM - label: minimum description: Get the minimum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MIN - label: maximum description: Get the maximum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MAX - label: average description: Values are averaged together. Works for the GAUGE and COUNT metric types. enum: - AVG - label: set intersection description: The aggregate produces a set of unique values. enum: - SET_INTERSECTION - label: distribution concatenation description: The aggregate produces a list of values combined together. enum: - DIST_CONCAT - label: custom description: Use a custom script to perform aggregation. enum: - CUSTOM script: not: type: string title: Script description: A custom JavaScript function that will control the aggregation. At the time of flushing, this aggregation will become the emitted event. ui: display_type: code-editor options: language: js minLength: 1 maxLength: 5000 properties: alert_type: title: Alert Type description: Specifies what kind of alert this should be. ui: display_type: list errorMessage: must be one of the allowed alert types oneOf: - label: Threshold Alert description: Trigger an alert when the evaluation is higher or lower than a set value. enum: - threshold - label: Change Alert description: Trigger an alert when there is a change in percentage or value thresholds. enum: - change - label: Absence Alert description: Trigger an alert when there is an absence of an event. enum: - absence event_type: title: Event Type description: The type of event is either a Log event or a Metric event. ui: display_type: list errorMessage: must be one of the allowed event types oneOf: - label: Metric description: Alert configuration based on a Metric event enum: - metric - label: Log description: Alert configuration based on a Log event enum: - log group_by: type: array minItems: 1 maxItems: 50 uniqueItems: true title: Group By Field Paths description: When aggregating, group events based on matching values from each of these field paths. Supports nesting via dot-notation. items: type: string maxLength: 100 title: Field description: The dot-notated property path. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ window_type: type: string title: Window Type description: Sliding windows can overlap, whereas tumbling windows are disjoint. For example, a tumbling window has a fixed time span and any events that fall within the "window duration" will be used in the aggregate. In a sliding window, the aggregation occurs every "window duration" seconds after an event is encountered. enum: - tumbling - sliding default: tumbling window_duration_minutes: type: integer title: Window Duration (minutes) description: The duration of the aggregation window in minutes. default: 5 minimum: 1 maximum: 1440 conditional: type: object additionalProperties: false properties: expressions: type: array items: type: object logical_operation: type: string enum: - AND - OR required: - alert_type - event_type - window_type - window_duration_minutes alert_payload: type: object title: Alert Payload description: Determine the content of the alert and where to send it. additionalProperties: false properties: service: title: Service description: Configure a service to receive the payload. Currently supports Slack, PagerDuty, Webhook or Mezmo (Log Analysis). Where allowed, messagetemplating can be used. To use a template, surround field names with double curly braces, e.g., {{.my_field}} or {{."my-object".thing}}. See [documentation](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) for more examples. additionalProperties: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 allOf: - if: properties: name: enum: - slack required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The text to be sent as a Slack message. required: - uri - message_text - if: properties: name: enum: - pager_duty required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 summary: type: string title: Summary description: Summarize the alert details for PagerDuty. minLength: 1 ui: display_type: data-template severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. source: type: string title: Source description: The source of the alert. minLength: 1 ui: display_type: data-template routing_key: type: string title: Routing Key description: The service integration routing key. minLength: 1 ui: display_type: secret event_action: type: string enum: - trigger - acknowledge - resolve title: Event Action description: The event action associated with the alert. required: - uri - summary - severity - source - routing_key - event_action - if: properties: name: enum: - webhook required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 method: type: string title: HTTP Method description: The HTTP method to use for the request. enum: - post - put - patch - delete - get - head - options - trace ui: default_hint: post message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The Webhook payload to send. This can be a text string or stringified JSON. If the message can be parsed as JSON, it will be sent as such. headers: type: array title: HTTP Headers description: A list of key/value pairs to be included in the HTTP request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 maxLength: 1024 description: The value of the header. required: - header_name - header_value maxItems: 20 auth: title: Authentication Options description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 description: The basic authentication user password: type: string title: Password minLength: 1 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: Choose basic or token-based authentication. default: none required: - uri - message_text - if: properties: name: enum: - log_analysis required: - name then: properties: severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. default: INFO subject: type: string title: Subject description: A string to be used as the subject of the alert. minLength: 1 maxLength: 200 ui: display_type: data-template body: type: string title: Body description: A string to be used as the alert message body. minLength: 1 maxLength: 1024 ui: display_type: data-template ingestion_key: type: string title: Ingestion Key description: The key required to communicate to Log Analysis for sending the alert. ui: display_type: mezmo-ingestion-key minLength: 1 required: - severity - subject - body - ingestion_key properties: name: title: Name description: The name of the service that will receive the notification. ui: display_type: list errorMessage: must be a valid service name oneOf: - label: Slack enum: - slack - label: PagerDuty enum: - pager_duty - label: Webhook enum: - webhook - label: Mezmo Log Analysis enum: - log_analysis required: - name throttling: type: object title: Throttling description: Only allow a certain number of notifications to be sent within a given time window. properties: window_secs: type: integer title: Window Seconds description: The time frame during which the number of notifications (set by the Threshold) is permitted. default: 60 minimum: 1 threshold: type: integer title: Threshold description: 'The maximum number of notifications allowed over the given time window set by "Window Seconds" (default: 1).' default: 1 minimum: 1 required: - service - throttling required: - general - evaluation - alert_payload alert-expression-threshold: title: Conditional Expression description: Define an expression for field comparison additionalProperties: type: - string - number allOf: - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: errorMessage: pattern: Value must be a number or numeric string to use the provided operator type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 pattern: ^\s*-?(\d*\.)?\d+\s*$ required: - value - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix - percent_change_greater - percent_change_greater_or_equal - percent_change_less - percent_change_less_or_equal - value_change_greater - value_change_greater_or_equal - value_change_less - value_change_less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 required: - value - if: title: Operator properties: str_operator: type: string enum: - is_ip_in_cidr_range required: - str_operator then: properties: value: type: string minLength: 3 format: cidr_range errorMessage: Value must be an IPv4 or IPv6 CIDR range required: - value - if: title: Operator properties: str_operator: type: string enum: - is_metric required: - str_operator then: title: Report errors properties: report_errors: type: boolean description: Report diagnostic errors when a value is not a valid metric? - if: title: Operator properties: str_operator: type: string enum: - contains - equal - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix ui: display_type: list-operators title: Operator required: - str_operator then: title: Case sensitive properties: case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive comparison? - if: title: Operator properties: str_operator: type: string enum: - regex_match ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Pattern properties: value: type: string title: Pattern description: The regular expression pattern to match against minLength: 1 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" match_newline: type: boolean title: Dot all mode description: Dot matches newline character multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines required: - value properties: field: type: string title: Field description: The JSON field whose value will be used in the comparison pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - is_metric - is_array - is_boolean - is_empty - is_null - is_number - is_object - is_string - exists - ends_with - starts_with - contains_token_term - contains_token_prefix - regex_match ui: display_type: list-operators title: Operator description: The comparison operator negate: type: boolean title: Negate description: Negate the comparison expression required: - field - str_operator alert-expression-change: title: Conditional Expression description: Define an expression for field comparison additionalProperties: type: - string - number allOf: - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: errorMessage: pattern: Value must be a number or numeric string to use the provided operator type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 pattern: ^\s*-?(\d*\.)?\d+\s*$ required: - value - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix - percent_change_greater - percent_change_greater_or_equal - percent_change_less - percent_change_less_or_equal - value_change_greater - value_change_greater_or_equal - value_change_less - value_change_less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 required: - value - if: title: Operator properties: str_operator: type: string enum: - is_ip_in_cidr_range required: - str_operator then: properties: value: type: string minLength: 3 format: cidr_range errorMessage: Value must be an IPv4 or IPv6 CIDR range required: - value - if: title: Operator properties: str_operator: type: string enum: - is_metric required: - str_operator then: title: Report errors properties: report_errors: type: boolean description: Report diagnostic errors when a value is not a valid metric? - if: title: Operator properties: str_operator: type: string enum: - contains - equal - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix ui: display_type: list-operators title: Operator required: - str_operator then: title: Case sensitive properties: case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive comparison? - if: title: Operator properties: str_operator: type: string enum: - regex_match ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Pattern properties: value: type: string title: Pattern description: The regular expression pattern to match against minLength: 1 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" match_newline: type: boolean title: Dot all mode description: Dot matches newline character multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines required: - value properties: field: type: string title: Field description: The JSON field whose value will be used in the comparison pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference str_operator: type: string enum: - percent_change_greater - percent_change_greater_or_equal - percent_change_less - percent_change_less_or_equal - value_change_greater - value_change_greater_or_equal - value_change_less - value_change_less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator negate: type: boolean title: Negate description: Negate the comparison expression required: - field - str_operator alert-condition-group-threshold: type: object title: Condition Group description: A group of expressions joined together by an and/or operator properties: expressions: type: array title: Expressions description: List of conditional expressions to be combined with the designated operator minItems: 1 items: anyOf: - $ref: "#/components/schemas/alert-condition-group-threshold" - $ref: "#/components/schemas/alert-expression-threshold" logical_operation: type: string enum: - AND - OR default: AND title: Logical Operation description: The logical operation (AND/OR) to be applied to the list of conditionals required: - expressions - logical_operation alert-condition-group-change: type: object title: Condition Group description: A group of expressions joined together by an and/or operator properties: expressions: type: array title: Expressions description: List of conditional expressions to be combined with the designated operator minItems: 1 items: anyOf: - $ref: "#/components/schemas/alert-condition-group-change" - $ref: "#/components/schemas/alert-expression-change" logical_operation: type: string enum: - AND - OR default: AND title: Logical Operation description: The logical operation (AND/OR) to be applied to the list of conditionals required: - expressions - logical_operation alert-evaluation: title: Evaluation description: Multiple events (Log or Metric) will be collected into a single event (via aggregation) over a specified time window. That event is what the alert conditions will evaluate against. This section specifies how that aggregation should work. For Log event types, a `custom` JavaScript operation is required because the event shape is not standardized. additionalProperties: type: string title: Operation description: Specifies the type of aggregation operation to use with the window type and duration. ui: display_type: list oneOf: - const: DEFAULT label: add description: Adds new values together if they are of the same type. For counters and gauges the values are added together. For sets, this is a union of the elements. For distribution, the samples are concatenated. For histograms, values in the same bucket will be added together. - const: SUM label: sum description: Values are added together. Works for the GAUGE and COUNT metric types. - const: MIN label: minimum description: Get the minimum value for a series of events. Works for the GAUGE and COUNT metric types. - const: MAX label: maximum description: Get the maximum value for a series of events. Works for the GAUGE and COUNT metric types. - const: AVG label: average description: Values are averaged together. Works for the GAUGE and COUNT metric types. - const: SET_INTERSECTION label: set intersection description: The aggregate produces a set of unique values. - const: DIST_CONCAT label: distribution concatenation description: The aggregate produces a list of values combined together. - const: CUSTOM label: custom description: Use a custom script to perform aggregation. allOf: - if: allOf: - type: object properties: event_type: enum: - log required: - event_type - type: object properties: alert_type: not: enum: - absence required: - alert_type then: properties: operation: title: Operation description: Specifies the type of aggregation operation to use with the window type and duration. ui: display_type: list oneOf: - label: custom description: Use a custom script to perform aggregation. enum: - CUSTOM default: CUSTOM group_by: type: array minItems: 1 maxItems: 50 uniqueItems: true title: Group By Field Paths description: When aggregating, group events based on matching values from each of these field paths. Supports nesting via dot-notation. items: type: string maxLength: 100 title: Field description: The dot-notated property path. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ ui: default_hint: - .name - .namespace - .tags event_timestamp: type: string title: Event Timestamp description: The path to a field on the event that contains a epoch timestamp value. If an event does not have a timestamp field, events will be associated to the wall clock value when the event is processed. maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: default_hint: timestamp pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ required: - operation - if: allOf: - type: object properties: event_type: enum: - metric required: - event_type - type: object properties: alert_type: not: enum: - absence required: - alert_type then: properties: operation: title: Operation description: Specifies the type of aggregation operation to use with the window type and duration. ui: display_type: list oneOf: - label: add description: Adds new values together if they are of the same type. For counters and gauges the values are added together. For sets, this is a union of the elements. For distribution, the samples are concatenated. For histograms, values in the same bucket will be added together. enum: - DEFAULT - label: sum description: Values are added together. Works for the GAUGE and COUNT metric types. enum: - SUM - label: minimum description: Get the minimum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MIN - label: maximum description: Get the maximum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MAX - label: average description: Values are averaged together. Works for the GAUGE and COUNT metric types. enum: - AVG - label: set intersection description: The aggregate produces a set of unique values. enum: - SET_INTERSECTION - label: distribution concatenation description: The aggregate produces a list of values combined together. enum: - DIST_CONCAT - label: custom description: Use a custom script to perform aggregation. enum: - CUSTOM default: SUM event_timestamp: not: type: string title: Event Timestamp description: The path to a field on the event that contains a epoch timestamp value. If an event does not have a timestamp field, events will be associated to the wall clock value when the event is processed. maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ required: - operation - if: properties: operation: enum: - CUSTOM required: - operation then: properties: script: type: string title: Script description: A custom JavaScript function that will control the aggregation. At the time of flushing, this aggregation will become the emitted event. ui: display_type: code-editor options: language: js minLength: 1 maxLength: 5000 default: > // Receives the current event, metadata and an accumulator object. Logic can be performed // on the event properties and added to the accumulator object for later analysis. // The accumulator is persisted, and will become the emitted event when // conditions are true, OR if the time window naturally expires. Those same conditions // will also be evaluated against the emitted event to determine if the alert should // be triggered. function alertAggregation(accum, event, metadata) { // This function *must* return the accumulated event return accum } required: - script - if: properties: alert_type: enum: - threshold required: - alert_type then: properties: conditional: title: Alert Conditions description: Evaluate these conditions against the aggregated event, and trigger an alert if they are met. If the conditions are true, it will also flush the current aggregated event and end the time window. allOf: - $ref: "#/components/schemas/alert-condition-group-threshold" ui: display_type: rules-engine options: operators: alert-condition-group-threshold - if: properties: alert_type: enum: - change required: - alert_type then: properties: conditional: title: Alert Conditions description: Evaluate these conditions against the aggregated event, and trigger an alert if they are met. If the conditions are true, it will also flush the current aggregated event and end the time window. allOf: - $ref: "#/components/schemas/alert-condition-group-change" ui: display_type: rules-engine options: operators: alert-condition-group-change - if: properties: alert_type: enum: - absence required: - alert_type then: properties: conditional: title: Alert Conditions description: No conditions are necessary for alerting on the absence of events. not: anyOf: - $ref: "#/components/schemas/alert-condition-group-threshold" - $ref: "#/components/schemas/alert-condition-group-change" ui: display_type: rules-engine options: hidden: true operation: not: type: string title: Operation description: Specifies the type of aggregation operation to use with the window type and duration. ui: display_type: list oneOf: - label: add description: Adds new values together if they are of the same type. For counters and gauges the values are added together. For sets, this is a union of the elements. For distribution, the samples are concatenated. For histograms, values in the same bucket will be added together. enum: - DEFAULT - label: sum description: Values are added together. Works for the GAUGE and COUNT metric types. enum: - SUM - label: minimum description: Get the minimum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MIN - label: maximum description: Get the maximum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MAX - label: average description: Values are averaged together. Works for the GAUGE and COUNT metric types. enum: - AVG - label: set intersection description: The aggregate produces a set of unique values. enum: - SET_INTERSECTION - label: distribution concatenation description: The aggregate produces a list of values combined together. enum: - DIST_CONCAT - label: custom description: Use a custom script to perform aggregation. enum: - CUSTOM script: not: type: string title: Script description: A custom JavaScript function that will control the aggregation. At the time of flushing, this aggregation will become the emitted event. ui: display_type: code-editor options: language: js minLength: 1 maxLength: 5000 properties: alert_type: title: Alert Type description: Specifies what kind of alert this should be. ui: display_type: list errorMessage: must be one of the allowed alert types oneOf: - label: Threshold Alert description: Trigger an alert when the evaluation is higher or lower than a set value. enum: - threshold - label: Change Alert description: Trigger an alert when there is a change in percentage or value thresholds. enum: - change - label: Absence Alert description: Trigger an alert when there is an absence of an event. enum: - absence event_type: title: Event Type description: The type of event is either a Log event or a Metric event. ui: display_type: list errorMessage: must be one of the allowed event types oneOf: - label: Metric description: Alert configuration based on a Metric event enum: - metric - label: Log description: Alert configuration based on a Log event enum: - log group_by: type: array minItems: 1 maxItems: 50 uniqueItems: true title: Group By Field Paths description: When aggregating, group events based on matching values from each of these field paths. Supports nesting via dot-notation. items: type: string maxLength: 100 title: Field description: The dot-notated property path. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ window_type: type: string title: Window Type description: Sliding windows can overlap, whereas tumbling windows are disjoint. For example, a tumbling window has a fixed time span and any events that fall within the "window duration" will be used in the aggregate. In a sliding window, the aggregation occurs every "window duration" seconds after an event is encountered. enum: - tumbling - sliding default: tumbling window_duration_minutes: type: integer title: Window Duration (minutes) description: The duration of the aggregation window in minutes. default: 5 minimum: 1 maximum: 1440 conditional: type: object additionalProperties: false properties: expressions: type: array items: type: object logical_operation: type: string enum: - AND - OR required: - alert_type - event_type - window_type - window_duration_minutes notification-channel-sink: type: object title: Alert Payload description: Determine the content of the alert and where to send it. additionalProperties: false properties: service: title: Service description: Configure a service to receive the payload. Currently supports Slack, PagerDuty, Webhook or Mezmo (Log Analysis). Where allowed, messagetemplating can be used. To use a template, surround field names with double curly braces, e.g., {{.my_field}} or {{."my-object".thing}}. See [documentation](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) for more examples. additionalProperties: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 allOf: - if: properties: name: enum: - slack required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The text to be sent as a Slack message. required: - uri - message_text - if: properties: name: enum: - pager_duty required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 summary: type: string title: Summary description: Summarize the alert details for PagerDuty. minLength: 1 ui: display_type: data-template severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. source: type: string title: Source description: The source of the alert. minLength: 1 ui: display_type: data-template routing_key: type: string title: Routing Key description: The service integration routing key. minLength: 1 ui: display_type: secret event_action: type: string enum: - trigger - acknowledge - resolve title: Event Action description: The event action associated with the alert. required: - uri - summary - severity - source - routing_key - event_action - if: properties: name: enum: - webhook required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 method: type: string title: HTTP Method description: The HTTP method to use for the request. enum: - post - put - patch - delete - get - head - options - trace ui: default_hint: post message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The Webhook payload to send. This can be a text string or stringified JSON. If the message can be parsed as JSON, it will be sent as such. headers: type: array title: HTTP Headers description: A list of key/value pairs to be included in the HTTP request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 maxLength: 1024 description: The value of the header. required: - header_name - header_value maxItems: 20 auth: title: Authentication Options description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 description: The basic authentication user password: type: string title: Password minLength: 1 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: Choose basic or token-based authentication. default: none required: - uri - message_text - if: properties: name: enum: - log_analysis required: - name then: properties: severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. default: INFO subject: type: string title: Subject description: A string to be used as the subject of the alert. minLength: 1 maxLength: 200 ui: display_type: data-template body: type: string title: Body description: A string to be used as the alert message body. minLength: 1 maxLength: 1024 ui: display_type: data-template ingestion_key: type: string title: Ingestion Key description: The key required to communicate to Log Analysis for sending the alert. ui: display_type: mezmo-ingestion-key minLength: 1 required: - severity - subject - body - ingestion_key properties: name: title: Name description: The name of the service that will receive the notification. ui: display_type: list errorMessage: must be a valid service name oneOf: - label: Slack enum: - slack - label: PagerDuty enum: - pager_duty - label: Webhook enum: - webhook - label: Mezmo Log Analysis enum: - log_analysis required: - name throttling: type: object title: Throttling description: Only allow a certain number of notifications to be sent within a given time window. properties: window_secs: type: integer title: Window Seconds description: The time frame during which the number of notifications (set by the Threshold) is permitted. default: 60 minimum: 1 threshold: type: integer title: Threshold description: 'The maximum number of notifications allowed over the given time window set by "Window Seconds" (default: 1).' default: 1 minimum: 1 required: - service - throttling paths: /pipeline/{pipeline_id}: delete: summary: Deletes a pipeline tags: - Pipeline Management description: Removes a pipeline from the account parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response get: summary: Fetch pipeline confg tags: - Pipeline Management description: Retrieve a pipeline configuration parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response put: summary: Update pipeline tags: - Pipeline Management description: Update pipeline properties, like title requestBody: content: application/json: schema: type: object properties: title: type: string pattern: ^[^<>"]+$ errorMessage: invalid title name maxLength: 512 deploy_type: type: string enum: - saas - local default: saas deployment_groups: type: array maxItems: 10 items: type: string pattern: ^[a-zA-Z0-9-_]{1,256}$ required: - title required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline: get: summary: List all pipelines tags: - Pipeline Management description: List all pipelines for the account parameters: - schema: type: string enum: - ui - system in: query name: origin required: false description: Include Pipelines with a specific `origin` responses: "200": description: Default Response post: summary: Create pipeline tags: - Pipeline Management description: Creates a new pipeline (non published) requestBody: content: application/json: schema: type: object properties: title: type: string pattern: ^[^<>"]+$ errorMessage: invalid title name maxLength: 512 deploy_type: type: string enum: - saas - local default: saas deployment_groups: type: array maxItems: 10 items: type: string pattern: ^[a-zA-Z0-9-_]{1,256}$ required: - title required: true responses: "200": description: Default Response /pipeline/account: get: summary: Get account details tags: - Account description: Get list of settings applied to an account responses: "200": description: Default Response patch: summary: Update account details tags: - Account description: Update account settings requestBody: content: application/json: schema: type: object properties: local_deploy_config: type: object additionalProperties: false ui: feature_flag: pipeline-pulse properties: start_port: type: integer title: Start port minimum: 1 maximum: 65535 description: The lower bound of the port range end_port: type: integer exclusiveMinimum: $data: 1/start_port maximum: 65535 title: End port description: The upper bound of the port range required: - start_port - end_port responses: "200": description: Default Response /pipeline/{pipeline_id}/{component_kind}/{component_id}/alert/{alert_id}: delete: summary: Delete component alert tags: - Alerting description: Removes an alert attached to a pipeline component parameters: - schema: type: string enum: - node_alert - surge_detection_alert default: node_alert in: query name: component_alert_type required: false - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string enum: - source - transform in: path name: component_kind required: true - schema: type: string format: uuid in: path name: component_id required: true - schema: type: string format: uuid in: path name: alert_id required: true responses: "200": description: Default Response put: summary: Update component alert tags: - Alerting description: Update an alert attached to a pipeline component requestBody: content: application/json: schema: type: object additionalProperties: false properties: inputs: type: array items: type: string minItems: 1 alert_config: type: object properties: general: type: object evaluation: type: object alert_payload: type: object required: - general - evaluation - alert_payload active: type: boolean component_alert_type: type: string enum: - node_alert - surge_detection_alert default: node_alert required: - inputs required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string enum: - source - transform in: path name: component_kind required: true - schema: type: string format: uuid in: path name: component_id required: true - schema: type: string format: uuid in: path name: alert_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/alert/{alert_id}: get: summary: Fetch component alert tags: - Alerting description: Retrieves an alert attached to a pipeline component parameters: - schema: type: string enum: - node_alert - surge_detection_alert default: node_alert in: query name: component_alert_type required: false - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: alert_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/alert: get: summary: List component alerts tags: - Alerting description: List of alerts attached to a pipeline's components parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/{component_kind}/{component_id}/alert: post: summary: Create component alert tags: - Alerting description: Create an alert attached to a pipeline component requestBody: content: application/json: schema: type: object additionalProperties: false properties: inputs: type: array items: type: string minItems: 1 alert_config: type: object properties: general: type: object evaluation: type: object alert_payload: type: object required: - general - evaluation - alert_payload component_alert_type: type: string enum: - node_alert - surge_detection_alert default: node_alert required: - inputs - component_alert_type required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string enum: - source - transform in: path name: component_kind required: true - schema: type: string format: uuid in: path name: component_id required: true responses: "200": description: Default Response /pipeline/audit-event: get: summary: Fetch audit events tags: - Pipeline Management description: Retrieve a paginated list of pipeline audit events for an account parameters: - schema: type: string format: date-time in: query name: from required: true - schema: type: string format: date-time default: 2022-01-01T00:00:00Z in: query name: to required: true - schema: type: integer minimum: 1 maximum: 200 default: 20 in: query name: page_size required: true - schema: type: integer minimum: 0 default: 0 in: query name: page_index required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/event_metrics: get: summary: Get event metrics for a pipeline tags: - Metrics description: Query and fetch event metrics for a pipeline parameters: - schema: type: number minimum: 300 maximum: 1296000 title: Bucket size bucket_within_range: unit: seconds from_field: from to_field: to in: query name: bucket required: true description: The interval, in seconds, over which events will be grouped - schema: enum: - app - host - label - level default: app title: The field to group by in: query name: field required: false description: The field in the event to group events by - schema: type: string format: date-time title: Events from in: query name: from required: true description: The event metrics start date - schema: type: string format: date-time title: Events to date_time_greater_than: field: from in: query name: to required: true description: The event metrics end date - schema: type: boolean title: With metric changes default: false in: query name: with_changes required: false description: Include metric changes compared to the previous duration - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/export: post: summary: Trigger pipeline export tags: - Pipeline Export description: Prepare a pipeline for export into a terraform workspace. parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/to-manifest/{pipeline_id}: get: summary: Export pipeline to manifest tags: - Pipeline Export description: Export a SaaS pipeline to a portable YAML or JSON manifest. Requires pipeline:read. parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string enum: - yaml - json default: yaml in: query name: format required: false description: The manifest output format responses: "200": description: Default Response /pipeline/from-manifest/prepare: post: summary: Inspect a manifest's required secrets tags: - Pipeline Management description: Inspect a manifest and report the secrets that must be supplied on import. Creates nothing. Requires pipeline:write. requestBody: content: application/json: schema: type: object additionalProperties: false properties: manifest: type: string minLength: 1 maxLength: 1048576 required: - manifest required: true responses: "200": description: Default Response /pipeline/from-manifest: post: summary: Import a manifest as a new pipeline tags: - Pipeline Management description: Import a manifest as a new unpublished draft pipeline. Requires pipeline:write. requestBody: content: application/json: schema: type: object additionalProperties: false properties: manifest: type: string minLength: 1 maxLength: 1048576 required_secrets: type: array maxItems: 100 items: type: object additionalProperties: false properties: name: type: string pattern: ^[A-Z0-9_]{1,128}$ value: type: string minLength: 1 maxLength: 65536 required: - name - value required: - manifest required: true responses: "200": description: Default Response /pipeline/gateway-route/{gateway_route_id}: delete: summary: Delete gateway route tags: - Shared Sources description: Delete an unused gateway route. If the route is used by any active pipeline, an error will be returned. parameters: - schema: type: string in: path name: gateway_route_id required: true responses: "200": description: Default Response get: summary: Retrieve gateway route tags: - Shared Sources description: Retreive a gateway route parameters: - schema: type: string in: path name: gateway_route_id required: true responses: "200": description: Default Response patch: summary: Update gateway route property tags: - Shared Sources description: Patches a gateway route requestBody: content: application/json: schema: type: object additionalProperties: false properties: title: type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description origin: type: string enum: - ui - system description: The route origin legacy_routing: type: boolean required: - title required: true parameters: - schema: type: string in: path name: gateway_route_id required: true responses: "200": description: Default Response put: summary: Update gateway route tags: - Shared Sources description: Updates a gateway route requestBody: content: application/json: schema: type: object additionalProperties: false properties: title: type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ errorMessage: invalid title name description: oneOf: - type: string minLength: 1 - type: "null" errorMessage: invalid description type: type: string enum: - http - splunk-hec - kinesis-firehose - fluent - logstash - mezmo-agent - mezmo-datadog-source - webhook - prometheus-remote-write - open-telemetry-metrics - open-telemetry-logs - open-telemetry-traces legacy_routing: type: boolean required: - title - type required: true parameters: - schema: type: string in: path name: gateway_route_id required: true responses: "200": description: Default Response /pipeline/gateway-route: get: summary: List all gateway routes tags: - Shared Sources description: List all of the shareable gateway routes for an account parameters: - schema: type: string format: uuid in: query name: id required: false description: Optional gateway route id to retrieve. - schema: type: string enum: - ui - system in: query name: origin required: false description: Filters the list of routes by origin. responses: "200": description: Default Response post: summary: Create gateway route tags: - Shared Sources description: Create a gateway route requestBody: content: application/json: schema: type: object additionalProperties: false properties: title: type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description type: type: string enum: - http - splunk-hec - kinesis-firehose - fluent - logstash - mezmo-agent - mezmo-datadog-source - webhook - prometheus-remote-write - open-telemetry-metrics - open-telemetry-logs - open-telemetry-traces start_classification: type: boolean default: true legacy_routing: type: boolean default: true required: - title - type required: true responses: "200": description: Default Response /pipeline/grok-pattern: get: summary: Retrieve grok pattern tags: - Sampling and Simulation description: Retrieves a list of grok patterns available to use responses: "200": description: Default Response /pipeline/grok-pattern/simulate: post: summary: Simulate grok pattern tags: - Sampling and Simulation description: Simulate data flowing through a grok pattern requestBody: content: application/json: schema: type: object properties: pattern: type: string errorMessage: must be a valid grok expression composed only by community patterns pattern: ^(?:%\{[A-Z_0-9]+(?::\w+){0,2}\}[\s,;:-]*)+$ values: type: array items: type: string minItems: 1 required: - pattern - values required: true responses: "200": description: Default Response /pipeline/health: get: summary: Summarize pipeline health for one or more pipelines tags: - Pipeline Management description: Returns overall ingestion health as well as pipeline-specific health, such as lag in a pipeline's sources. parameters: - schema: type: integer minimum: 1 maximum: 60 default: 15 in: query name: interval_minutes required: false description: The window interval for measuring ingestion volume - schema: type: array items: type: string format: uuid description: A pipeline id in: query name: pipeline_ids required: false description: Return only pipelines with the given IDs responses: "200": description: Default Response content: application/json: schema: type: object properties: meta: type: object additionalProperties: true data: type: object description: Overall health of data ingestion and pipeline sources properties: interval_minutes: type: integer description: The window interval for measuring ingestion volume ingestion_health: type: object properties: ingestion_metrics_status: type: string description: A success or failure message for the ability to fetch metrics related to ingestion routes routes: type: array description: Ingestion status code health metrics per route items: type: object properties: route: type: string description: The route path health: type: array description: The health of a specific route measured by changes in status code items: type: object properties: status_code: type: integer description: The HTTP status code change_percent: type: number description: "The percentage change for the status code over time. The result is a comparison of the current interval and the same interval offset by `interval_minutes` ago. For example, when `{interval_minutes: 5, offset_minutes: 5}`, that calculates the percentage change in a 5-minute window from 5 minutes ago to a 5-minute window 10 minutes ago." pipeline_health: type: object properties: lag_metrics_status: type: string description: A success or failure message for the ability to fetch lag metrics pipelines: type: array description: All active and published pipelines. Inactive or unpublished pipelines are not included. items: type: object properties: id: type: string format: uuid description: The pipeline id title: type: string description: The pipeline title published_at: type: string format: date-time description: The date/time of the pipeline's last publish (if at all) sources: type: array description: Sources associated with this pipeline items: type: object properties: id: type: string description: The source id, which can also be the consumer group id type: type: string description: The type of source component title: type: string description: The title of the source component is_consumer_group: type: boolean description: Whether this source is a consumer group that could have lag lag_event_count: anyOf: - type: number description: A relative measure of how far this source is falling behind in processing. A higher value indicates the source is further behind, and a value of `0` indicates it is caught up. Use this as a directional signal, not an exact count of individual events. If this value is `null`, then it could mean the source has never had consumers, or that the lag metrics were unavailable (check `lag_metrics_status`). - type: "null" /pipeline/{pipeline_id}/log-volume-reduction: post: summary: Log Volume Reduction tags: - Log Volume Reduction description: Post log volume reduction actions to a pipeline requestBody: content: application/json: schema: type: object properties: place_before_transform_id: type: string format: uuid processor_title: type: string actions: type: array minItems: 1 items: type: object oneOf: - type: object required: - drain - type: object required: - field_summary properties: type: type: string enum: - dedupe - filter - sample - throttle - drop-field title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name drain: type: string field_summary: type: object properties: field: type: string minLength: 1 value: oneOf: - type: string minLength: 1 - type: "null" required: - field description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description user_config: type: object if: properties: type: enum: - drop-field then: allOf: - type: object required: - field_summary - type: object properties: field_summary: type: object properties: value: type: "null" else: properties: type: not: enum: - drop-field required: - processor_title - actions required: true parameters: - schema: type: boolean default: false in: query name: create_new_classifier_analysis required: false - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/node: get: summary: List available pipeline nodes tags: - Pipeline Management description: A list of all available nodes for use when creating pipelines responses: "200": description: Default Response /pipeline/state-variable/{state_variable_id}: get: summary: Fetch state variable tags: - Incident Mode description: Fetches the value of a state variable parameters: - schema: type: string format: uuid in: path name: state_variable_id required: true responses: "200": description: Default Response put: summary: Update state variable tags: - Incident Mode description: Edits an existing state variable requestBody: content: application/json: schema: type: object properties: pipeline_id: type: string format: uuid state: type: object required: - pipeline_id - state required: true parameters: - schema: type: string format: uuid in: path name: state_variable_id required: true responses: "200": description: Default Response /pipeline/state-variable: get: summary: List all state variables tags: - Incident Mode description: List all state variables defined in the account parameters: - schema: type: string format: uuid in: query name: pipeline_id required: false responses: "200": description: Default Response patch: summary: Partially update state variable tags: - Incident Mode description: Update one or many state variables requestBody: content: application/json: schema: type: object properties: pipeline_id: type: string format: uuid state: type: object required: - pipeline_id - state required: true responses: "200": description: Default Response post: summary: Create state variable tags: - Incident Mode description: Add a pipeline state variable to the account requestBody: content: application/json: schema: type: object properties: pipeline_id: type: string format: uuid state: type: object required: - pipeline_id - state required: true responses: "200": description: Default Response /pipeline/pre-built: post: summary: Add prebuilt pipeline to account tags: - Pipeline Management description: Creates a new pipeline in the account that is prebuilt requestBody: content: application/json: schema: type: object properties: origin: type: string enum: - ui - system default: system requested_types: type: array items: type: string enum: - DEMO - OTEL minItems: 1 metadata: type: object properties: userFullName: type: string minLength: 1 maxLength: 200 baseUrl: type: string minLength: 1 maxLength: 200 required: - userFullName - baseUrl required: - requested_types - metadata required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/processing-status: put: summary: Update pipeline processing status tags: - Account description: Change a pipeline to enabled or disabled requestBody: content: application/json: schema: type: object properties: processing_status: type: string enum: - enabled - disabled required: - processing_status required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/regex-pattern/simulate: post: summary: Simulate regex pattern tags: - Sampling and Simulation description: View results of sending data using a regular expression requestBody: content: application/json: schema: type: object properties: pattern: type: string values: type: array items: type: string minItems: 1 required: - pattern - values required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/revision/{pipeline_revision_lookup}: get: summary: Fetch pipeline revision tags: - Pipeline Management description: Returns a pipeline configuration from a particular version parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: oneOf: - type: string format: uuid - enum: - latest in: path name: pipeline_revision_lookup required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/revision: get: summary: List all pipeline revisions tags: - Pipeline Management description: Returns a list of revisions that exist in the pipeline parameters: - schema: type: integer default: 200 minimum: 1 maximum: 1000 in: query name: limit required: false description: The maximum number of revisions to return - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/publish: post: summary: Publish pipeline tags: - Pipeline Management description: Creates a new pipeline in the account parameters: - schema: type: boolean default: false in: query name: allow_unconnected_edges required: false - schema: type: boolean default: false in: query name: allow_empty_access_keys required: false - schema: type: boolean default: false in: query name: allow_disabled required: false - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/revert: post: summary: Revert pipeline to version tags: - Pipeline Management description: Updates a pipeline configuration to a previous version requestBody: content: application/json: schema: type: object properties: pipeline_revision_id: type: string format: uuid required: - pipeline_revision_id required: true parameters: - schema: type: boolean default: false in: query name: allow_invalid_pipeline_state required: false - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/simulate/{pipeline_source_id}: post: summary: Start simulation of pipeline tags: - Sampling and Simulation description: Sends data to a vector instance with a simulated pipeline requestBody: content: application/json: schema: type: object properties: pipeline_sample_id: type: string format: uuid required: - pipeline_sample_id required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_source_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/sink/{pipeline_sink_id}: delete: summary: Delete sink tags: - Pipeline Nodes description: Remove sink from a pipeline parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_sink_id required: true responses: "200": description: Default Response get: summary: Fetch sink tags: - Pipeline Nodes description: Fetch a sink configuration of a pipeline parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_sink_id required: true responses: "200": description: Default Response put: summary: Update sink tags: - Pipeline Nodes description: Updates a sink of a pipeline requestBody: content: application/json: schema: type: object allOf: - if: properties: type: enum: - mezmo deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Log Analysis description: Send log data to Mezmo Log Analysis additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true mezmo_host: type: string title: Mezmo Host description: The host for your Log Analysis environment default: null minLength: 1 ingestion_key: type: string title: Ingestion Key description: Ingestion key ui: display_type: mezmo-ingestion-key minLength: 1 query: type: object title: Query Parameters default: hostname: mezmo properties: hostname: type: string title: Hostname description: Hostname string or template to extract event values to form a hostname. Depending on the type of ingestion, this value is usually set from other places so the default value is often sufficient. If the value in the event is null, empty, or doesn't exist, when using a template then it will be set to "mezmo" default: mezmo minLength: 1 ui: display_type: data-template tags: type: array title: Tags description: List of tag strings or templates to attach to logs. Template must resolve to either an array of strings, or a string. Commas are treated as tag delimiters. items: type: string title: Tag description: Tag string or template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template ip: type: string title: IP description: IP address template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template mac: type: string title: MAC description: MAC address template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template message: title: Log Construction Scheme default: scheme: Explicit field selection allOf: - if: properties: scheme: enum: - Explicit field selection then: properties: line: type: string title: Line description: Template or field reference to use as the log line. Field reference can point to an object. minLength: 1 maxLength: 1024 meta_field: type: string title: Meta Field description: Field containing the meta object for the log pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference maxLength: 512 app: type: string title: App description: App name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template file: type: string title: File description: File name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template timestamp_field: type: string title: Timestamp description: Field containing the timestamp for the log pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference maxLength: 512 env: type: string title: Env description: Environment name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template properties: scheme: type: string title: Scheme description: How to construct the log message. enum: - Explicit field selection - Message pass-through default: Explicit field selection required: - scheme required: - mezmo_host - ingestion_key - if: properties: type: enum: - mezmo-archive deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Archive Destination description: Store logs for future imports into mezmo pipeline or log analysis. additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true destination: title: Destination description: "" allOf: - if: properties: archive_provider: enum: - Azure then: properties: container_name: type: string title: Container Name minLength: 1 description: The name of the container for blob storage. connection_string: type: string title: Connection String minLength: 1 description: A connection string for the account that contains an access key required: - container_name - connection_string - if: properties: archive_provider: enum: - S3 then: properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret bucket: type: string title: Bucket pattern: (?!(^xn--|.+-s3alias$))^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ description: The S3 bucket name. Do not include a leading s3:// or a trailing / region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - access_key_id - secret_access_key - bucket - region properties: archive_provider: type: string title: Archive Provider description: The storage destination for your data. enum: - Azure - S3 default: Azure required: - archive_provider required: - destination - if: properties: type: enum: - http deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: HTTP description: Send data to a HTTP endpoint additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI to make HTTP requests to. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none auth: title: Authentication description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 description: The basic authentication user password: type: string title: Password minLength: 1 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none headers: type: array title: HTTP Headers description: An array of key/value objects for the headers to be sent with the request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 description: The value of the header. required: - header_name - header_value advanced_options: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-http-advanced-options title: Advanced Options description: Settings available for fine-tuning the destination behavior. additionalProperties: false dependencies: payload_prefix: - payload_suffix payload_suffix: - payload_prefix properties: max_bytes: type: integer title: Payload Size description: The maximum number of uncompressed bytes when batching data to the destination. minimum: 1 maximum: 2097152 timeout_secs: type: integer title: Timeout Seconds description: The number of seconds before a destination write timeout. minimum: 5 method: type: string enum: - post - put - patch - delete - get - head - options - trace title: HTTP Method description: The HTTP method to use for the destination. payload_prefix: type: string title: Payload Prefix description: Add a prefix to the payload. Only used for serialized JSON chunks. This option also requires "Payload Suffix" to form a valid JSON string. examples: - '{"event":"' - '{"extra_prop": true, "our_thing":"' minLength: 1 payload_suffix: type: string title: Payload Suffix description: Used in combination with "Payload Prefix" to form valid JSON from the payload. examples: - "}" - '"extra_prop": true }' minLength: 1 proxy: type: object additionalProperties: false title: Proxy Settings description: Configure proxy settings for the destination. properties: enabled: type: boolean title: Enabled description: Turns proxying on/off default: false endpoint: type: string minLength: 1 title: HTTP(S) Endpoint description: HTTP or HTTPS Endpoint to use for traffic. format: uri errorMessage: format: Must be a valid URI hosts_bypass_proxy: type: array minItems: 1 title: Hosts to Bypass description: A list of hosts to bypass proxying. Can be specified as a domain name, IP address, or CIDR block. Wildcards are supported as a dot (.) in domain names, or as a star (*) to match all hosts. examples: - 93.184.216.34 - ignorehost.example.com - .ignoreall.example.com - 93.184.216.0/24 items: type: string title: Host description: Domain name, IP address, or CIDR block. minLength: 1 tls_protocols: type: array title: TLS Protocols description: A list of ALPN protocols to use during TLS negotiation. They are attempted in the order they appear. items: type: string title: Protocol description: An ALPN protocol name minLength: 1 minItems: 1 rate_limiting: type: object additionalProperties: false title: Rate Limiting description: Settings for controlling rate limiting to the destination. properties: request_limit: type: integer minimum: 1 title: Request Limit description: The max number of requests allowed within the specified "Duration Seconds" setting. duration_secs: type: integer default: 1 title: Duration Seconds description: The window of time used to apply "Request Limit." minimum: 1 required: - uri - encoding - compression - if: properties: type: enum: - cloudwatch-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Cloudwatch Logs description: Publish log events to AWS Cloudwatch Logs additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none group_name: type: string title: Group Name minLength: 1 description: The name of the log group for the targeted log stream. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the targeted log stream. required: - auth - encoding - compression - group_name - region - stream_name - if: properties: type: enum: - cloudwatch-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Cloudwatch Metrics description: Send metric events to AWS Cloudwatch Metrics additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none namespace: type: string title: Namespace minLength: 1 description: Name for the container that will isolate metrics from one another. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - auth - compression - namespace - region - if: properties: type: enum: - sqs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Simple Queue Service (SQS) description: Publish log events to AWS Simple Queue Service additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text sqs_queue_url: type: string title: SQS Queue URL format: url errorMessage: format: Must be a valid URL description: The URL of the SQS queue that the data is publishing to. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - auth - encoding - sqs_queue_url - region - if: properties: type: enum: - notification-channel deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Notification Channel description: Send a Message to a 3rd Party Service. additionalProperties: false properties: service: title: Service description: Configure a service to receive the payload. Currently supports Slack, PagerDuty, Webhook or Mezmo (Log Analysis). Where allowed, messagetemplating can be used. To use a template, surround field names with double curly braces, e.g., {{.my_field}} or {{."my-object".thing}}. See [documentation](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) for more examples. additionalProperties: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 allOf: - if: properties: name: enum: - slack required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The text to be sent as a Slack message. required: - uri - message_text - if: properties: name: enum: - pager_duty required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 summary: type: string title: Summary description: Summarize the alert details for PagerDuty. minLength: 1 ui: display_type: data-template severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. source: type: string title: Source description: The source of the alert. minLength: 1 ui: display_type: data-template routing_key: type: string title: Routing Key description: The service integration routing key. minLength: 1 ui: display_type: secret event_action: type: string enum: - trigger - acknowledge - resolve title: Event Action description: The event action associated with the alert. required: - uri - summary - severity - source - routing_key - event_action - if: properties: name: enum: - webhook required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 method: type: string title: HTTP Method description: The HTTP method to use for the request. enum: - post - put - patch - delete - get - head - options - trace ui: default_hint: post message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The Webhook payload to send. This can be a text string or stringified JSON. If the message can be parsed as JSON, it will be sent as such. headers: type: array title: HTTP Headers description: A list of key/value pairs to be included in the HTTP request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 maxLength: 1024 description: The value of the header. required: - header_name - header_value maxItems: 20 auth: title: Authentication Options description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 description: The basic authentication user password: type: string title: Password minLength: 1 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: Choose basic or token-based authentication. default: none required: - uri - message_text - if: properties: name: enum: - log_analysis required: - name then: properties: severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. default: INFO subject: type: string title: Subject description: A string to be used as the subject of the alert. minLength: 1 maxLength: 200 ui: display_type: data-template body: type: string title: Body description: A string to be used as the alert message body. minLength: 1 maxLength: 1024 ui: display_type: data-template ingestion_key: type: string title: Ingestion Key description: The key required to communicate to Log Analysis for sending the alert. ui: display_type: mezmo-ingestion-key minLength: 1 required: - severity - subject - body - ingestion_key properties: name: title: Name description: The name of the service that will receive the notification. ui: display_type: list errorMessage: must be a valid service name oneOf: - label: Slack enum: - slack - label: PagerDuty enum: - pager_duty - label: Webhook enum: - webhook - label: Mezmo Log Analysis enum: - log_analysis required: - name throttling: type: object title: Throttling description: Only allow a certain number of notifications to be sent within a given time window. properties: window_secs: type: integer title: Window Seconds description: The time frame during which the number of notifications (set by the Threshold) is permitted. default: 60 minimum: 1 threshold: type: integer title: Threshold description: 'The maximum number of notifications allowed over the given time window set by "Window Seconds" (default: 1).' default: 1 minimum: 1 required: - service - throttling - if: properties: type: enum: - azure-blob-storage deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Azure Blob Storage description: Publish log events to Azure Blob Storage additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none container_name: type: string title: Container Name minLength: 1 description: The name of the container for blob storage. connection_string: type: string title: Connection String minLength: 1 description: A connection string for the account that contains an access key prefix: type: string title: Prefix minLength: 1 description: A prefix to be applied to all object keys ui: display_type: data-template file_consolidation: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-azure-blob-file-consolidation title: File Consolidation description: This sink writes many small files out to azure blob storage. Enabling this process will allow the automatic consolidation of these small files into larger files of your choosing. This process will enable upon deployment and run on the chosen interval from `Processing Interval` creating files named `merged_[timestamp].log` where `timestamp` is the time since epoch when the actual file was created. The process will recursively access all files under the `Base Path` to handle merging sub-directory logging structures. additionalProperties: false properties: enabled: type: boolean default: false title: Enabled description: Toggles whether the process is enabled. process_every_seconds: type: integer title: Processing Interval (seconds) description: How often to run the consolidation process. default: 600 minimum: 300 maximum: 3600 requested_size_bytes: type: integer title: Requested File Size (bytes) description: The requested size of the consolidated files. default: 500000000 minimum: 50000000 maximum: 10000000000 base_path: type: string title: Base Path default: "" maxLength: 512 description: The path from the container to begin recursivelylooking for files to merge. A blank value indicates root.Merged files will only contain data from the respectivefolder. required: - encoding - compression - container_name - connection_string - if: properties: type: enum: - kinesis-firehose deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Kinesis Data Firehose description: Publish log events to AWS Kinesis Data Firehose additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the Kinesis Firehose stream required: - auth - encoding - compression - region - stream_name - if: properties: type: enum: - kinesis-streams deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Kinesis Streams description: Publish log events to AWS Kinesis Streams additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none partition_key_field: type: string title: Partition Key Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ description: The field in the log whose value is used as Kinesis' partition key. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the targeted log stream. required: - auth - encoding - compression - region - stream_name - if: properties: type: enum: - elasticsearch deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Elasticsearch description: Publish log events to Elasticsearch additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none auth: title: ElasticSearch Auth Strategy description: ElasticSearch authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - aws then: properties: access_key_id: type: string title: Access Key ID minLength: 1 maxLength: 1024 description: AWS access key ID ui: display_type: secret secret_access_key: type: string title: Secrete Access Key minLength: 1 maxLength: 1024 description: AWS secrete access key ui: display_type: secret region: type: string title: Region minLength: 1 maxLength: 128 description: The name of the AWS region that is targeted. required: - access_key_id - secret_access_key - region properties: strategy: type: string title: Strategy description: The ES authentication strategy to use enum: - basic - aws required: - strategy endpoints: type: array title: Endpoints uniqueItems: true minItems: 1 description: An array of ElasticSearch endpoints items: type: string title: URL format: url errorMessage: format: Must be a valid URL description: The full Elasticsearch URL to send data to. minLength: 1 maxLength: 1024 pipeline: type: string title: Pipeline minLength: 1 description: Name of an ES ingest pipeline to include. index: type: string minLength: 1 maxLength: 512 title: Custom Index description: Index to use when writing ES events (default = mezmo-%Y.%m.%d) default: mezmo-%Y.%m.%d required: - compression - auth - endpoints - if: properties: type: enum: - gcp-cloud-monitoring deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud Monitoring description: Publish metrics events to GCP Cloud Monitoring additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. resource_type: type: string title: Resource Type minLength: 1 description: The `monitored-resource` type as defined in Monitoring. resource_labels: type: array title: Resource Labels description: An array of key/val objects describing labels of the Monitoring resource. items: type: object title: Label/Value description: A key/val object describing a label name and its value. additionalProperties: false properties: label_name: type: string title: Label Name minLength: 1 description: The name of the label. label_value: type: string title: Label Value minLength: 1 description: The value of the label. required: - label_name - label_value required: - credentials_json - project_id - resource_type - if: properties: type: enum: - gcp-cloud-storage deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud Storage description: Publish log events to GCP Cloud Storage additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file encoding: type: string description: Dictates how the data will be serialized before storing. enum: - json - ndjson - text title: Encoding default: text bucket: type: string title: Bucket minLength: 1 description: The name of the bucket in GCP where the data will be stored. compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none bucket_prefix: type: string title: Bucket Prefix minLength: 1 description: The prefix applied to the bucket name, giving the appearance of having directories. required: - credentials_json - encoding - bucket - compression - if: properties: type: enum: - gcp-cloud-operations deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud Operations description: Publish log events to GCP Cloud Operations additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file log_id: type: string title: Log ID minLength: 1 description: Concise reference for the log stream name. project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. resource_type: type: string title: Resource Type minLength: 1 description: The `monitored-resource` type as defined in Monitoring. resource_labels: type: array title: Resource Labels description: An array of key/val objects describing labels of the Monitoring resource. items: type: object title: Label/Value description: A key/val object describing a label name and its value. additionalProperties: false properties: label_name: type: string title: Label Name minLength: 1 description: The name of the label. label_value: type: string title: Label Value minLength: 1 description: The value of the label. required: - label_name - label_value required: - credentials_json - log_id - project_id - resource_type - if: properties: type: enum: - gcp-cloud-pubsub deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud PubSub description: Publish log events to GCP Cloud PubSub additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file encoding: type: string description: Dictates how the data will be serialized before storing. enum: - json - text title: Encoding default: text project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. topic: type: string title: Topic minLength: 1 description: The name of the topic to publish messages to. required: - credentials_json - encoding - project_id - topic - if: properties: type: enum: - honeycomb-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Honeycomb Logs description: Send log data to Honeycomb additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Honeycomb API key description: Your Honeycomb API key minLength: 1 maxLength: 512 ui: display_type: secret dataset: type: string title: Dataset description: The name of the targeted dataset. minLength: 1 maxLength: 512 required: - api_key - dataset - if: properties: type: enum: - indexed-search deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Indexed Search description: Publish log events to Mezmo's data store for Indexed Search capabilities additionalProperties: true properties: search_field: type: string title: Full Text Search Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: .line maxLength: 256 description: If your data reaches this stage as a plain-text string, the entire payload will be indexed for full-text searching. If your data reaches this stage as an object, the field you enter here will be used as the full-text search field. Nested fields are allowed. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference timestamp_field: type: string title: Timestamp Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: timestamp maxLength: 256 description: If your data reaches this stage as an object, this field represents the timestamp value to be cataloged for searching. If the field does not exists the current time will be used. We accept UNIX timestamps (in milliseconds), ISO8601/RFC3389, and RFC2822 formats. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - search_field - timestamp_field - if: properties: type: enum: - kafka deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Kafka description: Publish log events to Kafka additionalProperties: type: string allOf: - if: properties: sasl_enabled: enum: - true then: properties: sasl_mechanism: type: string enum: - SCRAM-SHA-512 - SCRAM-SHA-256 - PLAIN title: SASL description: The Kafka SASL/SCRAM mechanism default: SCRAM-SHA-512 sasl_username: type: string title: Username description: The SASL/SCRAM username minLength: 1 sasl_password: type: string title: Password description: The SASL/SCRAM username ui: display_type: secret minLength: 1 required: - sasl_mechanism - sasl_username - sasl_password - if: properties: tls_enabled: enum: - true then: properties: tls_ca_certificate: type: string title: TLS CA Certificate Chain minLength: 1 description: The CA certificate chain in the PEM format ui: display_type: secret-file format: x509_certificate errorMessage: format: Must be a valid PEM or DER certificate tls_verify_certificate: type: boolean title: TLS Verify Certificate description: Verify the TLS certificates default: true required: - tls_verify_certificate properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - lz4 - snappy - zstd - none title: Compression default: none event_key_field: type: string title: Event Key Field minLength: 1 description: The field in the log whose value is used as Kafka's event key. brokers: type: array title: Brokers uniqueItems: true minItems: 1 description: An array of key/val objects for each host/port of a Kafka broker. items: type: object title: Host/Port description: The host and port for the Kafka broker. additionalProperties: false properties: host: title: Host description: The hostname or IP address of the broker. anyOf: - type: string format: ipv4 errorMessage: format: Must be a valid IPv4 address - type: string format: hostname errorMessage: format: Must be a valid hostname port: type: number title: Port description: The port that the broker listens on. required: - host - port topic: type: string title: Topic minLength: 1 description: The name of the topic to publish to. group_id: type: string title: Group ID minLength: 1 description: The consumer group identifier sasl_enabled: type: boolean title: SASL/SCRAM Enabled description: Enable SASL/SCRAM authentication with Kafka default: false tls_enabled: type: boolean title: Use TLS description: Enable TLS during connections to the remote default: false required: - encoding - compression - brokers - topic - if: properties: type: enum: - loki deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Loki description: Publish log events to Loki additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: title: Loki Auth Strategy description: Loki authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token (passed as-is) ui: display_type: secret required: - token properties: strategy: type: string title: Strategy description: The authentication strategy to use (only basic supported) enum: - basic - bearer required: - strategy encoding: type: object title: Encoding description: Configures how event are encoded properties: codec: type: string title: Codec enum: - json - text required: - codec endpoint: type: string title: Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Loki base URL path: type: string title: Path minLength: 1 maxLength: 512 description: The path appended to the Loki base URL(defaults to /loki/api/v1/push) default: /loki/api/v1/push labels: type: array title: Loki Labels description: Key/Value pair used to describe Loki data items: type: object title: Labels description: Label name and values additionalProperties: false properties: label_name: type: string title: Label Name description: The name of the label minLength: 1 maxLength: 512 label_value: type: string title: Label Value description: The value of the label minLength: 1 maxLength: 512 required: - label_name - label_value required: - auth - encoding - endpoint - labels - if: properties: type: enum: - new-relic deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: New Relic description: Publish to New Relic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true account_id: type: string title: New Relic Account ID description: Account ID for your New Relic account minLength: 1 maxLength: 1024 ui: display_type: secret api: type: string title: API description: New Relic API endpoint (events not supported) enum: - logs - metrics default: logs license_key: type: string title: New Relic License Key description: License key for your New Relic account minLength: 1 maxLength: 1024 ui: display_type: secret required: - account_id - api - license_key - if: properties: type: enum: - prometheus-remote-write deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Prometheus Remote Write description: Publish metrics to Prometheus Remote Write endpoint additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Endpoint format: uri errorMessage: format: Must be a valid URI description: "The full URI to make HTTP requests to. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. Example: http://example.org:8080/api/v1/push" minLength: 1 maxLength: 2000 auth: title: Authentication description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none required: - strategy required: - endpoint - if: properties: type: enum: - pulsar deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Pulsar description: Publish log events to Apache Pulsar additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text url: type: string title: URL format: url errorMessage: format: Must be a valid URL description: The full Pulsar URL to send data to. topic: type: string title: Topic minLength: 1 description: The name of the Pulsar topic to publish to. required: - encoding - url - topic - if: properties: type: enum: - redis deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Redis description: Publish log events to Redis additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text url: type: string title: URL format: uri errorMessage: format: Must be a valid URL description: The full Redis URL to send data to. key: type: string title: Key minLength: 1 description: The name of the Redis key to write data to. required: - encoding - url - key - if: properties: type: enum: - splunk-hec-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Splunk HTTP Event Collector description: Publish log events to Splunk HTTP Event Collector additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none url: type: string title: Endpoint format: url errorMessage: format: Must be a valid URL description: The base URL for the Splunk instance. The collector path, such as `/services/collector/events`, will be automatically inferred from the destination's configuration. token: type: string title: Token minLength: 1 ui: display_type: secret description: The default token to authenticate to Splunk HEC. tls_verify_certificate: type: boolean title: Verify TLS Certificate description: If unchecked, the TLS certificate of the remote host will not be verified. **This is insecure**, but may be useful for testing or when evaluating Splunk in a trial instance. default: true host_field: type: string title: Host Field description: The field that contains the hostname to include in the event. default: metadata.host timestamp_field: type: string title: Timestamp Field description: The field that contains the timestamp to include in the event. default: metadata.time pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: feature_flag: pipeline-splunk-sink-v2 display_type: field-reference source: title: Source description: The source of events sent to this sink. This is typically the filename the logs originated from. Use the field path ".metadata.source" to use the upstream source value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - "null" required: - value_type sourcetype: title: Source Type description: The sourcetype of events sent to this sink. Use the field path ".metadata.sourcetype" to use the upstream sourcetype value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - "null" required: - value_type index: title: Index description: The name of the index to send events to. Use the field path ".metadata.index" to use the upstream index value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - "null" required: - value_type required: - compression - url - token - source - sourcetype - index - if: properties: type: enum: - sumo-logic-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sumo Logic Logs description: Publish log data to Sumo Logic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Sumo Logic Collector URL that receives the data ui: display_type: secret compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none category: type: string minLength: 1 maxLength: 512 title: Category description: Category to use when publishing data to Sumo Logic (default = mezmo_default_category) default: mezmo_default_category required: - endpoint - compression - if: properties: type: enum: - sumo-logic-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sumo Logic Metrics description: Publish metrics data to Sumo Logic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Sumo Logic Collector URL that receives the data ui: display_type: secret compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none category: type: string minLength: 1 maxLength: 512 title: Category description: Category to use when publishing data to Sumo Logic (default = mezmo_default_category) default: mezmo_default_category required: - endpoint - compression - if: properties: type: enum: - vector deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Vector description: Publish log events to Vector additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none vector_ip: type: string title: Vector IP format: ipv4 description: The IP portion of the Vector host vector_port: type: number title: Vector Port description: The port that the Vector host listens on. version: type: string enum: - "1" - "2" title: Version default: "2" description: The version of the Vector sink which must match the source version. required: - compression - vector_ip - vector_port - version - if: properties: type: enum: - s3 deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS S3 description: Publish events as objects in AWS S3 additionalProperties: false errorMessage: properties: bucket: must be a valid bucket name tags: must be a valid tag identifier properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key bucket: type: string title: Bucket pattern: (?!(^xn--|.+-s3alias$))^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ description: The S3 bucket name. Do not include a leading s3:// or a trailing / prefix: type: string title: Prefix description: A prefix to apply to all object key names. default: / ui: display_type: data-template tags: type: array maxItems: 10 title: Tags description: A set of tags (key/value pairs) to apply to the object in S3. items: type: object properties: key: type: string title: Name description: The tag name pattern: ^[A-Za-z0-9_.:=+@/\-\s]*$ minLength: 1 maxLength: 128 value: type: string title: Value description: The tag value default: "" pattern: ^[A-Za-z0-9_.:=+@/\-\s]*$ minLength: 0 maxLength: 256 required: - key - value encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. file_consolidation: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-s3-file-consolidation title: File Consolidation description: "This sink writes many small files out to S3. Enabling this process will allow the automatic consolidation of these small files into larger files of your choosing. This process will enable upon deployment and run on the chosen interval from `Processing Interval` creating files named `merged_[timestamp].log` where `timestamp` is the time since epoch. *NOTE*: only `text` and `ndjson` files written by the sink are consolidated." additionalProperties: false properties: enabled: type: boolean default: false title: Enabled description: Toggles whether the process is enabled. process_every_seconds: type: integer title: Processing Interval (seconds) description: How often to run the consolidation process. default: 600 minimum: 300 maximum: 3600 requested_size_bytes: type: integer title: Requested File Size (bytes) description: The size of the requested files. default: 500000000 minimum: 50000000 maximum: 10000000000 base_path: type: string title: Base Path default: "" maxLength: 512 description: The path from the container to begin recursivelylooking for files to merge. A blank value indicates root.Merged files will only contain data from the respectivefolder. required: - auth - bucket - prefix - encoding - compression - region - if: properties: type: enum: - blackhole deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Blackhole description: Drop data arriving at this destination additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true - if: properties: type: enum: - datadog-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Datadog Logs description: Publish log events to Datadog additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Datadog API Key description: Datadog log application API key minLength: 1 ui: display_type: secret compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: gzip site: type: string description: The Datadog site (region) to send logs to. enum: - us1 - us3 - us5 - eu1 title: Datadog Site default: us1 required: - api_key - compression - site - if: properties: type: enum: - datadog-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Datadog Metrics description: Publish metrics events to Datadog. Sending custom metrics to Datadog may result in extra charges on your Datadog bill. Please go to https://docs.datadoghq.com/metrics/custom_metrics/ for more information on Datadog custom metrics pricing. additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Datadog API Key description: Datadog metrics application API key minLength: 1 maxLength: 512 ui: display_type: secret site: type: string description: The Datadog site (region) to send metrics to. enum: - us1 - us3 - us5 - eu1 title: Datadog Site default: us1 required: - api_key - site - if: properties: type: enum: - clickhouse deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Clickhouse description: Deliver log data to the ClickHouse database additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string minLength: 1 maxLength: 2048 title: Endpoint description: Endpoint of the Clickhouse server format: uri errorMessage: format: Must be a valid URI auth: title: Auth Strategy description: Authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 2048 description: The bearer token to use for authentication ui: display_type: secret required: - token properties: strategy: type: string title: Strategy description: The ES authentication strategy to use enum: - basic - bearer required: - strategy table: type: string title: Table description: The table that data is inserted into pattern: ^[a-zA-Z0-9_]+$ errorMessage: Invalid table name ui: display_type: data-template database: type: string title: Database description: The database that contains the table that data is inserted into pattern: ^[a-zA-Z0-9_]+$ errorMessage: Invalid database name ui: display_type: data-template compression: type: string enum: - gzip - none - snappy - zlib - zstd title: Compression description: The compression algorithm to use default: gzip skip_unknown_fields: type: boolean title: Skip Unknown Fields description: Allows ClickHouse to discard fields not present in the table schema default: false required: - endpoint - auth - table - if: properties: type: enum: - opentelemetry deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry description: Publish data to OpenTelemetry additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The OpenTelemetry Collector URL that receives the data auth: title: Authentication description: Configures HTTP authentication additionalProperties: type: string allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none headers: type: array title: HTTP Headers description: An array of key/value objects for the headers to be sent with the request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 description: The value of the header. required: - header_name - header_value compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none required: - endpoint - compression properties: title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description type: type: string enum: - mezmo - mezmo-archive - http - cloudwatch-logs - cloudwatch-metrics - sqs - notification-channel - azure-blob-storage - kinesis-firehose - kinesis-streams - elasticsearch - gcp-cloud-monitoring - gcp-cloud-storage - gcp-cloud-operations - gcp-cloud-pubsub - honeycomb-logs - indexed-search - kafka - loki - new-relic - prometheus-remote-write - pulsar - redis - splunk-hec-logs - sumo-logic-logs - sumo-logic-metrics - vector - s3 - blackhole - datadog-logs - datadog-metrics - clickhouse - opentelemetry deploy_type: type: string enum: - saas - local default: saas user_config: type: object generation_id: type: integer inputs: type: array items: type: string required: - type - user_config - generation_id - inputs required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_sink_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/sink: post: summary: Add a sink to a pipeline tags: - Pipeline Nodes requestBody: content: application/json: schema: type: object allOf: - if: properties: type: enum: - mezmo deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Log Analysis description: Send log data to Mezmo Log Analysis additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true mezmo_host: type: string title: Mezmo Host description: The host for your Log Analysis environment default: null minLength: 1 ingestion_key: type: string title: Ingestion Key description: Ingestion key ui: display_type: mezmo-ingestion-key minLength: 1 query: type: object title: Query Parameters default: hostname: mezmo properties: hostname: type: string title: Hostname description: Hostname string or template to extract event values to form a hostname. Depending on the type of ingestion, this value is usually set from other places so the default value is often sufficient. If the value in the event is null, empty, or doesn't exist, when using a template then it will be set to "mezmo" default: mezmo minLength: 1 ui: display_type: data-template tags: type: array title: Tags description: List of tag strings or templates to attach to logs. Template must resolve to either an array of strings, or a string. Commas are treated as tag delimiters. items: type: string title: Tag description: Tag string or template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template ip: type: string title: IP description: IP address template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template mac: type: string title: MAC description: MAC address template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template message: title: Log Construction Scheme default: scheme: Explicit field selection allOf: - if: properties: scheme: enum: - Explicit field selection then: properties: line: type: string title: Line description: Template or field reference to use as the log line. Field reference can point to an object. minLength: 1 maxLength: 1024 meta_field: type: string title: Meta Field description: Field containing the meta object for the log pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference maxLength: 512 app: type: string title: App description: App name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template file: type: string title: File description: File name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template timestamp_field: type: string title: Timestamp description: Field containing the timestamp for the log pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference maxLength: 512 env: type: string title: Env description: Environment name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template properties: scheme: type: string title: Scheme description: How to construct the log message. enum: - Explicit field selection - Message pass-through default: Explicit field selection required: - scheme required: - mezmo_host - ingestion_key - if: properties: type: enum: - mezmo-archive deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Archive Destination description: Store logs for future imports into mezmo pipeline or log analysis. additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true destination: title: Destination description: "" allOf: - if: properties: archive_provider: enum: - Azure then: properties: container_name: type: string title: Container Name minLength: 1 description: The name of the container for blob storage. connection_string: type: string title: Connection String minLength: 1 description: A connection string for the account that contains an access key required: - container_name - connection_string - if: properties: archive_provider: enum: - S3 then: properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret bucket: type: string title: Bucket pattern: (?!(^xn--|.+-s3alias$))^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ description: The S3 bucket name. Do not include a leading s3:// or a trailing / region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - access_key_id - secret_access_key - bucket - region properties: archive_provider: type: string title: Archive Provider description: The storage destination for your data. enum: - Azure - S3 default: Azure required: - archive_provider required: - destination - if: properties: type: enum: - http deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: HTTP description: Send data to a HTTP endpoint additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI to make HTTP requests to. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none auth: title: Authentication description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 description: The basic authentication user password: type: string title: Password minLength: 1 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none headers: type: array title: HTTP Headers description: An array of key/value objects for the headers to be sent with the request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 description: The value of the header. required: - header_name - header_value advanced_options: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-http-advanced-options title: Advanced Options description: Settings available for fine-tuning the destination behavior. additionalProperties: false dependencies: payload_prefix: - payload_suffix payload_suffix: - payload_prefix properties: max_bytes: type: integer title: Payload Size description: The maximum number of uncompressed bytes when batching data to the destination. minimum: 1 maximum: 2097152 timeout_secs: type: integer title: Timeout Seconds description: The number of seconds before a destination write timeout. minimum: 5 method: type: string enum: - post - put - patch - delete - get - head - options - trace title: HTTP Method description: The HTTP method to use for the destination. payload_prefix: type: string title: Payload Prefix description: Add a prefix to the payload. Only used for serialized JSON chunks. This option also requires "Payload Suffix" to form a valid JSON string. examples: - '{"event":"' - '{"extra_prop": true, "our_thing":"' minLength: 1 payload_suffix: type: string title: Payload Suffix description: Used in combination with "Payload Prefix" to form valid JSON from the payload. examples: - "}" - '"extra_prop": true }' minLength: 1 proxy: type: object additionalProperties: false title: Proxy Settings description: Configure proxy settings for the destination. properties: enabled: type: boolean title: Enabled description: Turns proxying on/off default: false endpoint: type: string minLength: 1 title: HTTP(S) Endpoint description: HTTP or HTTPS Endpoint to use for traffic. format: uri errorMessage: format: Must be a valid URI hosts_bypass_proxy: type: array minItems: 1 title: Hosts to Bypass description: A list of hosts to bypass proxying. Can be specified as a domain name, IP address, or CIDR block. Wildcards are supported as a dot (.) in domain names, or as a star (*) to match all hosts. examples: - 93.184.216.34 - ignorehost.example.com - .ignoreall.example.com - 93.184.216.0/24 items: type: string title: Host description: Domain name, IP address, or CIDR block. minLength: 1 tls_protocols: type: array title: TLS Protocols description: A list of ALPN protocols to use during TLS negotiation. They are attempted in the order they appear. items: type: string title: Protocol description: An ALPN protocol name minLength: 1 minItems: 1 rate_limiting: type: object additionalProperties: false title: Rate Limiting description: Settings for controlling rate limiting to the destination. properties: request_limit: type: integer minimum: 1 title: Request Limit description: The max number of requests allowed within the specified "Duration Seconds" setting. duration_secs: type: integer default: 1 title: Duration Seconds description: The window of time used to apply "Request Limit." minimum: 1 required: - uri - encoding - compression - if: properties: type: enum: - cloudwatch-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Cloudwatch Logs description: Publish log events to AWS Cloudwatch Logs additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none group_name: type: string title: Group Name minLength: 1 description: The name of the log group for the targeted log stream. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the targeted log stream. required: - auth - encoding - compression - group_name - region - stream_name - if: properties: type: enum: - cloudwatch-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Cloudwatch Metrics description: Send metric events to AWS Cloudwatch Metrics additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none namespace: type: string title: Namespace minLength: 1 description: Name for the container that will isolate metrics from one another. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - auth - compression - namespace - region - if: properties: type: enum: - sqs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Simple Queue Service (SQS) description: Publish log events to AWS Simple Queue Service additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text sqs_queue_url: type: string title: SQS Queue URL format: url errorMessage: format: Must be a valid URL description: The URL of the SQS queue that the data is publishing to. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - auth - encoding - sqs_queue_url - region - if: properties: type: enum: - notification-channel deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Notification Channel description: Send a Message to a 3rd Party Service. additionalProperties: false properties: service: title: Service description: Configure a service to receive the payload. Currently supports Slack, PagerDuty, Webhook or Mezmo (Log Analysis). Where allowed, messagetemplating can be used. To use a template, surround field names with double curly braces, e.g., {{.my_field}} or {{."my-object".thing}}. See [documentation](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) for more examples. additionalProperties: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 allOf: - if: properties: name: enum: - slack required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The text to be sent as a Slack message. required: - uri - message_text - if: properties: name: enum: - pager_duty required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 summary: type: string title: Summary description: Summarize the alert details for PagerDuty. minLength: 1 ui: display_type: data-template severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. source: type: string title: Source description: The source of the alert. minLength: 1 ui: display_type: data-template routing_key: type: string title: Routing Key description: The service integration routing key. minLength: 1 ui: display_type: secret event_action: type: string enum: - trigger - acknowledge - resolve title: Event Action description: The event action associated with the alert. required: - uri - summary - severity - source - routing_key - event_action - if: properties: name: enum: - webhook required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 method: type: string title: HTTP Method description: The HTTP method to use for the request. enum: - post - put - patch - delete - get - head - options - trace ui: default_hint: post message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The Webhook payload to send. This can be a text string or stringified JSON. If the message can be parsed as JSON, it will be sent as such. headers: type: array title: HTTP Headers description: A list of key/value pairs to be included in the HTTP request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 maxLength: 1024 description: The value of the header. required: - header_name - header_value maxItems: 20 auth: title: Authentication Options description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 description: The basic authentication user password: type: string title: Password minLength: 1 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: Choose basic or token-based authentication. default: none required: - uri - message_text - if: properties: name: enum: - log_analysis required: - name then: properties: severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. default: INFO subject: type: string title: Subject description: A string to be used as the subject of the alert. minLength: 1 maxLength: 200 ui: display_type: data-template body: type: string title: Body description: A string to be used as the alert message body. minLength: 1 maxLength: 1024 ui: display_type: data-template ingestion_key: type: string title: Ingestion Key description: The key required to communicate to Log Analysis for sending the alert. ui: display_type: mezmo-ingestion-key minLength: 1 required: - severity - subject - body - ingestion_key properties: name: title: Name description: The name of the service that will receive the notification. ui: display_type: list errorMessage: must be a valid service name oneOf: - label: Slack enum: - slack - label: PagerDuty enum: - pager_duty - label: Webhook enum: - webhook - label: Mezmo Log Analysis enum: - log_analysis required: - name throttling: type: object title: Throttling description: Only allow a certain number of notifications to be sent within a given time window. properties: window_secs: type: integer title: Window Seconds description: The time frame during which the number of notifications (set by the Threshold) is permitted. default: 60 minimum: 1 threshold: type: integer title: Threshold description: 'The maximum number of notifications allowed over the given time window set by "Window Seconds" (default: 1).' default: 1 minimum: 1 required: - service - throttling - if: properties: type: enum: - azure-blob-storage deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Azure Blob Storage description: Publish log events to Azure Blob Storage additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none container_name: type: string title: Container Name minLength: 1 description: The name of the container for blob storage. connection_string: type: string title: Connection String minLength: 1 description: A connection string for the account that contains an access key prefix: type: string title: Prefix minLength: 1 description: A prefix to be applied to all object keys ui: display_type: data-template file_consolidation: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-azure-blob-file-consolidation title: File Consolidation description: This sink writes many small files out to azure blob storage. Enabling this process will allow the automatic consolidation of these small files into larger files of your choosing. This process will enable upon deployment and run on the chosen interval from `Processing Interval` creating files named `merged_[timestamp].log` where `timestamp` is the time since epoch when the actual file was created. The process will recursively access all files under the `Base Path` to handle merging sub-directory logging structures. additionalProperties: false properties: enabled: type: boolean default: false title: Enabled description: Toggles whether the process is enabled. process_every_seconds: type: integer title: Processing Interval (seconds) description: How often to run the consolidation process. default: 600 minimum: 300 maximum: 3600 requested_size_bytes: type: integer title: Requested File Size (bytes) description: The requested size of the consolidated files. default: 500000000 minimum: 50000000 maximum: 10000000000 base_path: type: string title: Base Path default: "" maxLength: 512 description: The path from the container to begin recursivelylooking for files to merge. A blank value indicates root.Merged files will only contain data from the respectivefolder. required: - encoding - compression - container_name - connection_string - if: properties: type: enum: - kinesis-firehose deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Kinesis Data Firehose description: Publish log events to AWS Kinesis Data Firehose additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the Kinesis Firehose stream required: - auth - encoding - compression - region - stream_name - if: properties: type: enum: - kinesis-streams deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Kinesis Streams description: Publish log events to AWS Kinesis Streams additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none partition_key_field: type: string title: Partition Key Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ description: The field in the log whose value is used as Kinesis' partition key. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the targeted log stream. required: - auth - encoding - compression - region - stream_name - if: properties: type: enum: - elasticsearch deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Elasticsearch description: Publish log events to Elasticsearch additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none auth: title: ElasticSearch Auth Strategy description: ElasticSearch authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - aws then: properties: access_key_id: type: string title: Access Key ID minLength: 1 maxLength: 1024 description: AWS access key ID ui: display_type: secret secret_access_key: type: string title: Secrete Access Key minLength: 1 maxLength: 1024 description: AWS secrete access key ui: display_type: secret region: type: string title: Region minLength: 1 maxLength: 128 description: The name of the AWS region that is targeted. required: - access_key_id - secret_access_key - region properties: strategy: type: string title: Strategy description: The ES authentication strategy to use enum: - basic - aws required: - strategy endpoints: type: array title: Endpoints uniqueItems: true minItems: 1 description: An array of ElasticSearch endpoints items: type: string title: URL format: url errorMessage: format: Must be a valid URL description: The full Elasticsearch URL to send data to. minLength: 1 maxLength: 1024 pipeline: type: string title: Pipeline minLength: 1 description: Name of an ES ingest pipeline to include. index: type: string minLength: 1 maxLength: 512 title: Custom Index description: Index to use when writing ES events (default = mezmo-%Y.%m.%d) default: mezmo-%Y.%m.%d required: - compression - auth - endpoints - if: properties: type: enum: - gcp-cloud-monitoring deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud Monitoring description: Publish metrics events to GCP Cloud Monitoring additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. resource_type: type: string title: Resource Type minLength: 1 description: The `monitored-resource` type as defined in Monitoring. resource_labels: type: array title: Resource Labels description: An array of key/val objects describing labels of the Monitoring resource. items: type: object title: Label/Value description: A key/val object describing a label name and its value. additionalProperties: false properties: label_name: type: string title: Label Name minLength: 1 description: The name of the label. label_value: type: string title: Label Value minLength: 1 description: The value of the label. required: - label_name - label_value required: - credentials_json - project_id - resource_type - if: properties: type: enum: - gcp-cloud-storage deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud Storage description: Publish log events to GCP Cloud Storage additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file encoding: type: string description: Dictates how the data will be serialized before storing. enum: - json - ndjson - text title: Encoding default: text bucket: type: string title: Bucket minLength: 1 description: The name of the bucket in GCP where the data will be stored. compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none bucket_prefix: type: string title: Bucket Prefix minLength: 1 description: The prefix applied to the bucket name, giving the appearance of having directories. required: - credentials_json - encoding - bucket - compression - if: properties: type: enum: - gcp-cloud-operations deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud Operations description: Publish log events to GCP Cloud Operations additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file log_id: type: string title: Log ID minLength: 1 description: Concise reference for the log stream name. project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. resource_type: type: string title: Resource Type minLength: 1 description: The `monitored-resource` type as defined in Monitoring. resource_labels: type: array title: Resource Labels description: An array of key/val objects describing labels of the Monitoring resource. items: type: object title: Label/Value description: A key/val object describing a label name and its value. additionalProperties: false properties: label_name: type: string title: Label Name minLength: 1 description: The name of the label. label_value: type: string title: Label Value minLength: 1 description: The value of the label. required: - label_name - label_value required: - credentials_json - log_id - project_id - resource_type - if: properties: type: enum: - gcp-cloud-pubsub deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud PubSub description: Publish log events to GCP Cloud PubSub additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file encoding: type: string description: Dictates how the data will be serialized before storing. enum: - json - text title: Encoding default: text project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. topic: type: string title: Topic minLength: 1 description: The name of the topic to publish messages to. required: - credentials_json - encoding - project_id - topic - if: properties: type: enum: - honeycomb-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Honeycomb Logs description: Send log data to Honeycomb additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Honeycomb API key description: Your Honeycomb API key minLength: 1 maxLength: 512 ui: display_type: secret dataset: type: string title: Dataset description: The name of the targeted dataset. minLength: 1 maxLength: 512 required: - api_key - dataset - if: properties: type: enum: - indexed-search deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Indexed Search description: Publish log events to Mezmo's data store for Indexed Search capabilities additionalProperties: true properties: search_field: type: string title: Full Text Search Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: .line maxLength: 256 description: If your data reaches this stage as a plain-text string, the entire payload will be indexed for full-text searching. If your data reaches this stage as an object, the field you enter here will be used as the full-text search field. Nested fields are allowed. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference timestamp_field: type: string title: Timestamp Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: timestamp maxLength: 256 description: If your data reaches this stage as an object, this field represents the timestamp value to be cataloged for searching. If the field does not exists the current time will be used. We accept UNIX timestamps (in milliseconds), ISO8601/RFC3389, and RFC2822 formats. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - search_field - timestamp_field - if: properties: type: enum: - kafka deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Kafka description: Publish log events to Kafka additionalProperties: type: string allOf: - if: properties: sasl_enabled: enum: - true then: properties: sasl_mechanism: type: string enum: - SCRAM-SHA-512 - SCRAM-SHA-256 - PLAIN title: SASL description: The Kafka SASL/SCRAM mechanism default: SCRAM-SHA-512 sasl_username: type: string title: Username description: The SASL/SCRAM username minLength: 1 sasl_password: type: string title: Password description: The SASL/SCRAM username ui: display_type: secret minLength: 1 required: - sasl_mechanism - sasl_username - sasl_password - if: properties: tls_enabled: enum: - true then: properties: tls_ca_certificate: type: string title: TLS CA Certificate Chain minLength: 1 description: The CA certificate chain in the PEM format ui: display_type: secret-file format: x509_certificate errorMessage: format: Must be a valid PEM or DER certificate tls_verify_certificate: type: boolean title: TLS Verify Certificate description: Verify the TLS certificates default: true required: - tls_verify_certificate properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - lz4 - snappy - zstd - none title: Compression default: none event_key_field: type: string title: Event Key Field minLength: 1 description: The field in the log whose value is used as Kafka's event key. brokers: type: array title: Brokers uniqueItems: true minItems: 1 description: An array of key/val objects for each host/port of a Kafka broker. items: type: object title: Host/Port description: The host and port for the Kafka broker. additionalProperties: false properties: host: title: Host description: The hostname or IP address of the broker. anyOf: - type: string format: ipv4 errorMessage: format: Must be a valid IPv4 address - type: string format: hostname errorMessage: format: Must be a valid hostname port: type: number title: Port description: The port that the broker listens on. required: - host - port topic: type: string title: Topic minLength: 1 description: The name of the topic to publish to. group_id: type: string title: Group ID minLength: 1 description: The consumer group identifier sasl_enabled: type: boolean title: SASL/SCRAM Enabled description: Enable SASL/SCRAM authentication with Kafka default: false tls_enabled: type: boolean title: Use TLS description: Enable TLS during connections to the remote default: false required: - encoding - compression - brokers - topic - if: properties: type: enum: - loki deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Loki description: Publish log events to Loki additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: title: Loki Auth Strategy description: Loki authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token (passed as-is) ui: display_type: secret required: - token properties: strategy: type: string title: Strategy description: The authentication strategy to use (only basic supported) enum: - basic - bearer required: - strategy encoding: type: object title: Encoding description: Configures how event are encoded properties: codec: type: string title: Codec enum: - json - text required: - codec endpoint: type: string title: Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Loki base URL path: type: string title: Path minLength: 1 maxLength: 512 description: The path appended to the Loki base URL(defaults to /loki/api/v1/push) default: /loki/api/v1/push labels: type: array title: Loki Labels description: Key/Value pair used to describe Loki data items: type: object title: Labels description: Label name and values additionalProperties: false properties: label_name: type: string title: Label Name description: The name of the label minLength: 1 maxLength: 512 label_value: type: string title: Label Value description: The value of the label minLength: 1 maxLength: 512 required: - label_name - label_value required: - auth - encoding - endpoint - labels - if: properties: type: enum: - new-relic deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: New Relic description: Publish to New Relic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true account_id: type: string title: New Relic Account ID description: Account ID for your New Relic account minLength: 1 maxLength: 1024 ui: display_type: secret api: type: string title: API description: New Relic API endpoint (events not supported) enum: - logs - metrics default: logs license_key: type: string title: New Relic License Key description: License key for your New Relic account minLength: 1 maxLength: 1024 ui: display_type: secret required: - account_id - api - license_key - if: properties: type: enum: - prometheus-remote-write deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Prometheus Remote Write description: Publish metrics to Prometheus Remote Write endpoint additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Endpoint format: uri errorMessage: format: Must be a valid URI description: "The full URI to make HTTP requests to. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. Example: http://example.org:8080/api/v1/push" minLength: 1 maxLength: 2000 auth: title: Authentication description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none required: - strategy required: - endpoint - if: properties: type: enum: - pulsar deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Pulsar description: Publish log events to Apache Pulsar additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text url: type: string title: URL format: url errorMessage: format: Must be a valid URL description: The full Pulsar URL to send data to. topic: type: string title: Topic minLength: 1 description: The name of the Pulsar topic to publish to. required: - encoding - url - topic - if: properties: type: enum: - redis deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Redis description: Publish log events to Redis additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text url: type: string title: URL format: uri errorMessage: format: Must be a valid URL description: The full Redis URL to send data to. key: type: string title: Key minLength: 1 description: The name of the Redis key to write data to. required: - encoding - url - key - if: properties: type: enum: - splunk-hec-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Splunk HTTP Event Collector description: Publish log events to Splunk HTTP Event Collector additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none url: type: string title: Endpoint format: url errorMessage: format: Must be a valid URL description: The base URL for the Splunk instance. The collector path, such as `/services/collector/events`, will be automatically inferred from the destination's configuration. token: type: string title: Token minLength: 1 ui: display_type: secret description: The default token to authenticate to Splunk HEC. tls_verify_certificate: type: boolean title: Verify TLS Certificate description: If unchecked, the TLS certificate of the remote host will not be verified. **This is insecure**, but may be useful for testing or when evaluating Splunk in a trial instance. default: true host_field: type: string title: Host Field description: The field that contains the hostname to include in the event. default: metadata.host timestamp_field: type: string title: Timestamp Field description: The field that contains the timestamp to include in the event. default: metadata.time pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: feature_flag: pipeline-splunk-sink-v2 display_type: field-reference source: title: Source description: The source of events sent to this sink. This is typically the filename the logs originated from. Use the field path ".metadata.source" to use the upstream source value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - "null" required: - value_type sourcetype: title: Source Type description: The sourcetype of events sent to this sink. Use the field path ".metadata.sourcetype" to use the upstream sourcetype value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - "null" required: - value_type index: title: Index description: The name of the index to send events to. Use the field path ".metadata.index" to use the upstream index value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - "null" required: - value_type required: - compression - url - token - source - sourcetype - index - if: properties: type: enum: - sumo-logic-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sumo Logic Logs description: Publish log data to Sumo Logic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Sumo Logic Collector URL that receives the data ui: display_type: secret compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none category: type: string minLength: 1 maxLength: 512 title: Category description: Category to use when publishing data to Sumo Logic (default = mezmo_default_category) default: mezmo_default_category required: - endpoint - compression - if: properties: type: enum: - sumo-logic-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sumo Logic Metrics description: Publish metrics data to Sumo Logic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Sumo Logic Collector URL that receives the data ui: display_type: secret compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none category: type: string minLength: 1 maxLength: 512 title: Category description: Category to use when publishing data to Sumo Logic (default = mezmo_default_category) default: mezmo_default_category required: - endpoint - compression - if: properties: type: enum: - vector deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Vector description: Publish log events to Vector additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none vector_ip: type: string title: Vector IP format: ipv4 description: The IP portion of the Vector host vector_port: type: number title: Vector Port description: The port that the Vector host listens on. version: type: string enum: - "1" - "2" title: Version default: "2" description: The version of the Vector sink which must match the source version. required: - compression - vector_ip - vector_port - version - if: properties: type: enum: - s3 deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS S3 description: Publish events as objects in AWS S3 additionalProperties: false errorMessage: properties: bucket: must be a valid bucket name tags: must be a valid tag identifier properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key bucket: type: string title: Bucket pattern: (?!(^xn--|.+-s3alias$))^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ description: The S3 bucket name. Do not include a leading s3:// or a trailing / prefix: type: string title: Prefix description: A prefix to apply to all object key names. default: / ui: display_type: data-template tags: type: array maxItems: 10 title: Tags description: A set of tags (key/value pairs) to apply to the object in S3. items: type: object properties: key: type: string title: Name description: The tag name pattern: ^[A-Za-z0-9_.:=+@/\-\s]*$ minLength: 1 maxLength: 128 value: type: string title: Value description: The tag value default: "" pattern: ^[A-Za-z0-9_.:=+@/\-\s]*$ minLength: 0 maxLength: 256 required: - key - value encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. file_consolidation: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-s3-file-consolidation title: File Consolidation description: "This sink writes many small files out to S3. Enabling this process will allow the automatic consolidation of these small files into larger files of your choosing. This process will enable upon deployment and run on the chosen interval from `Processing Interval` creating files named `merged_[timestamp].log` where `timestamp` is the time since epoch. *NOTE*: only `text` and `ndjson` files written by the sink are consolidated." additionalProperties: false properties: enabled: type: boolean default: false title: Enabled description: Toggles whether the process is enabled. process_every_seconds: type: integer title: Processing Interval (seconds) description: How often to run the consolidation process. default: 600 minimum: 300 maximum: 3600 requested_size_bytes: type: integer title: Requested File Size (bytes) description: The size of the requested files. default: 500000000 minimum: 50000000 maximum: 10000000000 base_path: type: string title: Base Path default: "" maxLength: 512 description: The path from the container to begin recursivelylooking for files to merge. A blank value indicates root.Merged files will only contain data from the respectivefolder. required: - auth - bucket - prefix - encoding - compression - region - if: properties: type: enum: - blackhole deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Blackhole description: Drop data arriving at this destination additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true - if: properties: type: enum: - datadog-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Datadog Logs description: Publish log events to Datadog additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Datadog API Key description: Datadog log application API key minLength: 1 ui: display_type: secret compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: gzip site: type: string description: The Datadog site (region) to send logs to. enum: - us1 - us3 - us5 - eu1 title: Datadog Site default: us1 required: - api_key - compression - site - if: properties: type: enum: - datadog-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Datadog Metrics description: Publish metrics events to Datadog. Sending custom metrics to Datadog may result in extra charges on your Datadog bill. Please go to https://docs.datadoghq.com/metrics/custom_metrics/ for more information on Datadog custom metrics pricing. additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Datadog API Key description: Datadog metrics application API key minLength: 1 maxLength: 512 ui: display_type: secret site: type: string description: The Datadog site (region) to send metrics to. enum: - us1 - us3 - us5 - eu1 title: Datadog Site default: us1 required: - api_key - site - if: properties: type: enum: - clickhouse deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Clickhouse description: Deliver log data to the ClickHouse database additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string minLength: 1 maxLength: 2048 title: Endpoint description: Endpoint of the Clickhouse server format: uri errorMessage: format: Must be a valid URI auth: title: Auth Strategy description: Authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 2048 description: The bearer token to use for authentication ui: display_type: secret required: - token properties: strategy: type: string title: Strategy description: The ES authentication strategy to use enum: - basic - bearer required: - strategy table: type: string title: Table description: The table that data is inserted into pattern: ^[a-zA-Z0-9_]+$ errorMessage: Invalid table name ui: display_type: data-template database: type: string title: Database description: The database that contains the table that data is inserted into pattern: ^[a-zA-Z0-9_]+$ errorMessage: Invalid database name ui: display_type: data-template compression: type: string enum: - gzip - none - snappy - zlib - zstd title: Compression description: The compression algorithm to use default: gzip skip_unknown_fields: type: boolean title: Skip Unknown Fields description: Allows ClickHouse to discard fields not present in the table schema default: false required: - endpoint - auth - table - if: properties: type: enum: - opentelemetry deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry description: Publish data to OpenTelemetry additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The OpenTelemetry Collector URL that receives the data auth: title: Authentication description: Configures HTTP authentication additionalProperties: type: string allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none headers: type: array title: HTTP Headers description: An array of key/value objects for the headers to be sent with the request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 description: The value of the header. required: - header_name - header_value compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none required: - endpoint - compression properties: type: type: string enum: - mezmo - mezmo-archive - http - cloudwatch-logs - cloudwatch-metrics - sqs - notification-channel - azure-blob-storage - kinesis-firehose - kinesis-streams - elasticsearch - gcp-cloud-monitoring - gcp-cloud-storage - gcp-cloud-operations - gcp-cloud-pubsub - honeycomb-logs - indexed-search - kafka - loki - new-relic - prometheus-remote-write - pulsar - redis - splunk-hec-logs - sumo-logic-logs - sumo-logic-metrics - vector - s3 - blackhole - datadog-logs - datadog-metrics - clickhouse - opentelemetry deploy_type: type: string enum: - saas - local default: saas title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description user_config: type: object inputs: type: array items: type: string description: A list of inputs for the given sink. Typically this is empty when creating a new sink via the UI, but is leveraged by the terraform provider when managing pipeline state. required: - type - user_config required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/tap/{component_id}: post: summary: Start pipeline tap tags: - Sampling and Simulation description: Start a tap operation to view data as it flows through the pipeline. requestBody: content: application/json: schema: type: object additionalProperties: false properties: limit: type: integer default: 10 description: The maximum number of events to return minimum: 1 maximum: 100 timeout: type: integer default: 5 description: The amount of time, in seconds, to wait for `limit` events to return before timing out minimum: 1 maximum: 20 edge_id: type: string minLength: 1 description: Specific edge_id instance to tap filter: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string pattern: ^([^\.]+)(?:\.(.+))?$ in: path name: component_id required: true description: The component whose output is to be tapped responses: "200": description: Default Response /pipeline/{pipeline_id}/source/{pipeline_source_id}: delete: summary: Remove source tags: - Pipeline Nodes description: Removes a source from a pipeline parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_source_id required: true responses: "200": description: Default Response get: summary: Retrieve source tags: - Pipeline Nodes description: Fetches configuration of a source parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_source_id required: true responses: "200": description: Default Response put: summary: Update source tags: - Pipeline Nodes description: Updates a source in a pipeline requestBody: content: application/json: schema: type: object allOf: - if: properties: type: enum: - http deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: HTTP description: Receive data from incoming HTTP requests additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 decoding: oneOf: - enum: - bytes - enum: - json - enum: - ndjson - enum: - auto ui: display_type: list errorMessage: must be equal to one of the allowed values description: The decoding method for converting frames into data events. title: Decoding Method default: auto required: - decoding - if: properties: type: enum: - splunk-hec deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Splunk HEC description: Receive Splunk logs additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - s3 deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS S3 description: Collect data from object contents in AWS S3 additionalProperties: false properties: sqs_queue_url: type: string title: SQS Queue URL description: The URL of a AWS SQS queue configured to receive S3 bucket notifications for the desired S3 buckets format: uri errorMessage: format: Must be a valid URI auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key compression: type: string title: Compression description: The compression format of the S3 objects enum: - auto - gzip - none - zstd default: auto region: type: string title: Region minLength: 1 description: The name of the source AWS region. required: - sqs_queue_url - auth - compression - region - if: properties: type: enum: - sqs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS SQS description: Collect messages from AWS SQS additionalProperties: false properties: queue_url: type: string title: Queue URL description: The URL of an AWS SQS queue format: uri errorMessage: format: Must be a valid URI maxLength: 128 auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id maxLength: 128 secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret maxLength: 128 required: - access_key_id - secret_access_key region: type: string title: Region minLength: 1 description: The name of the source AWS region. maxLength: 128 required: - queue_url - auth - region - if: properties: type: enum: - demo-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Demo Logs description: Generate log messages for testing and demonstration. additionalProperties: type: array title: Log Lines description: A list of user-provided log lines that will be sent through the pipeline infinitely. A maximum of 25 lines is allowed. The lines can be either plain text or ndjson. If ndjson is detected, the encoding for the source will be set to JSON, and lines that are not valid will show up as a visible component error. Otherwise, text encoding will be used. minItems: 1 maxItems: 25 items: type: string minLength: 1 maxLength: 4096 title: Line description: A single line of log data, either stringified JSON or plain text. ui: display_type: user-provided-lines allOf: - if: properties: format: enum: - user_provided required: - format then: properties: lines: type: array title: Log Lines description: A list of user-provided log lines that will be sent through the pipeline infinitely. A maximum of 25 lines is allowed. The lines can be either plain text or ndjson. If ndjson is detected, the encoding for the source will be set to JSON, and lines that are not valid will show up as a visible component error. Otherwise, text encoding will be used. minItems: 1 maxItems: 25 items: type: string minLength: 1 maxLength: 4096 title: Line description: A single line of log data, either stringified JSON or plain text. ui: display_type: user-provided-lines required: - lines - allOf: - errorMessage: "`lines` is only valid when `format` is `user_provided`" if: not: type: object properties: format: enum: - user_provided required: - format then: not: type: object properties: lines: type: array title: Log Lines description: A list of user-provided log lines that will be sent through the pipeline infinitely. A maximum of 25 lines is allowed. The lines can be either plain text or ndjson. If ndjson is detected, the encoding for the source will be set to JSON, and lines that are not valid will show up as a visible component error. Otherwise, text encoding will be used. minItems: 1 maxItems: 25 items: type: string minLength: 1 maxLength: 4096 title: Line description: A single line of log data, either stringified JSON or plain text. ui: display_type: user-provided-lines required: - lines properties: interval_ms: type: number title: Interval description: The amount of time to pause between each demo log. Fractions of a second are accepted to increase throughput. e.g. 500 ms == 120 lines per minute default: 1000 minimum: 500 maximum: 3600000 ui: display_type: milliseconds-to-human-readable format: title: Format description: The format of the log messages. All of the data is generated. ui: display_type: list errorMessage: must be one of the available types of demo data oneOf: - label: HTTP JSON description: HTTP request data in JSON format. enum: - json - label: Sensor JSON description: Data from environment sensors in JSON format. enum: - env_sensor - label: Financial JSON description: Financial data in JSON format. enum: - financial - label: Nginx Requests description: Nginx logs in text format. enum: - nginx - label: Apache Requests description: Apache request logs in text format. enum: - apache_common - label: Apache Errors description: Apache error logs in text format. enum: - apache_error - label: BSD Syslog description: Mock BSD syslog entries in rfc3164 text format. enum: - bsd_syslog - label: Syslog description: Mock syslog entries in rfc5424 text format. enum: - syslog - label: HTTP Metrics description: HTTP-based metrics in the Mezmo metrics JSON format. enum: - http_metrics - label: Generic Metrics description: Generic metrics in the Mezmo metrics JSON format enum: - generic_metrics - label: User-Provided Logs description: Custom logs provided by the user. These can either be plain text or serialized JSON. enum: - user_provided - label: Infrastructure Logs description: Infrastructure logs consisting of sysdig, nginx, HDFS and spark logs enum: - infrastructure - label: Kubernetes Logs description: Kubernetes logs in JSON format enum: - kubernetes default: json required: - format - if: properties: type: enum: - kinesis-firehose deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Kinesis Firehose description: Receive Kinesis Firehose data additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 format: type: string enum: - json - text default: json title: Data Format description: This specifies what the data format will be after it is base64 decoded. If it is JSON, it will be automatically parsed. required: - format - if: properties: type: enum: - fluent deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Fluent (Fluentd, Fluent Bit) description: Receive data from Fluentd or Fluent Bit additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 decoding: type: string description: The decoding method for converting frames into data events. enum: - bytes - json - ndjson title: Decoding Method default: json required: - decoding - if: properties: type: enum: - log-analysis deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Log Analysis description: Receive data directly from your Mezmo Log Analysis account additionalProperties: false - if: properties: type: enum: - log-analysis-ingestion deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Log Analysis Ingestion description: Receive unparsed source data directly from your Mezmo Log Analysis account. additionalProperties: type: string - if: properties: type: enum: - logstash deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Logstash description: Receive Logstash data additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 format: type: string enum: - json - text default: json title: Data Format description: The format of the logstash data required: - format - if: properties: type: enum: - mezmo-agent deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Agent description: Receive logs from the Mezmo Agent additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - mezmo-datadog-source deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Datadog Agent description: Send logs and metrics data directly from an installed datadog agent additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - pipeline-data-restoration deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Pipeline Data Restoration description: Receive data directly from your Mezmo Pipline Data Restoration additionalProperties: false - if: properties: type: enum: - webhook deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Webhook (WebSub) description: Receive data from incoming webhooks using the WebSub protocol. additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - kafka deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Kafka Source description: Collect events from Kafka compatible brokers additionalProperties: type: string properties: brokers: type: array title: Brokers uniqueItems: true minItems: 1 description: An array of key/val objects for each host/port of a Kafka broker. items: type: object title: Host/Port description: The host and port for the Kafka broker. additionalProperties: false properties: host: title: Host description: The hostname or IP address of the broker. anyOf: - type: string format: ipv4 errorMessage: format: Must be a valid IPv4 address - type: string format: hostname maxLength: 512 errorMessage: format: Must be a valid hostname port: type: number minimum: 1 maximum: 65535 title: Port description: The port that the broker listens on. required: - host - port topics: type: array title: Topics items: type: string title: Topic minLength: 1 maxLength: 256 description: The Kafka topic name. minItems: 1 description: List of Kafka topics names to read events from. group_id: type: string title: Consumer Group minLength: 1 description: The consumer group name to be used to consume events from Kafka. tls_enabled: type: boolean title: TLS enabled description: Whether or not to require TLS for outgoing connections. default: true sasl_enabled: type: boolean title: SASL/SCRAM Enabled description: Enable SASL/SCRAM authentication with Kafka default: true decoding_codec: type: string enum: - bytes - json title: Decoding description: Configures how events are decoded from raw bytes default: bytes if: properties: sasl_enabled: enum: - true then: properties: sasl_mechanism: type: string enum: - SCRAM-SHA-512 - SCRAM-SHA-256 - PLAIN title: SASL description: The Kafka SASL/SCRAM mechanism default: PLAIN sasl_username: type: string title: Username description: The SASL/SCRAM username minLength: 1 sasl_password: type: string title: Password description: The SASL/SCRAM username ui: display_type: secret minLength: 1 required: - sasl_mechanism - sasl_username - sasl_password required: - brokers - topics - group_id - decoding_codec - if: properties: type: enum: - azure-event-hub deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Azure Event Hub Source description: Collect events from Azure Event Hub using Azure Kafka Endpoint additionalProperties: false properties: connection_string: type: string title: Connection String minLength: 1 maxLength: 512 description: The Connection String as it appears in hub consumer SAS Policy ui: display_type: secret namespace: type: string title: Namespace minLength: 1 maxLength: 256 description: The Event Hub Namespace topics: type: array title: Event Hub Names items: type: string title: Name minLength: 1 maxLength: 256 description: Azure Event Hub instance name. minItems: 1 description: The list of Azure Event Hub name(s) to read events from. group_id: type: string title: Consumer Group minLength: 1 maxLength: 256 description: The consumer group name that this consumer belongs to. decoding_codec: type: string enum: - bytes - json title: Decoding description: Configures how events are decoded from raw bytes default: bytes required: - connection_string - namespace - topics - group_id - decoding_codec - if: properties: type: enum: - prometheus-remote-write deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Prometheus Remote Write description: Receive prometheus remote write protobuf data from HTTP requests additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - open-telemetry-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry Metrics description: Receive metrics from OpenTelemetry over HTTP additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - open-telemetry-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry Logs description: Receive logs from OpenTelemetry over HTTP additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - open-telemetry-traces deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry Traces description: Receive traces from OpenTelemetry over HTTP additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 properties: title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description type: type: string enum: - http - splunk-hec - s3 - sqs - demo-logs - kinesis-firehose - fluent - log-analysis - log-analysis-ingestion - logstash - mezmo-agent - mezmo-datadog-source - pipeline-data-restoration - webhook - kafka - azure-event-hub - prometheus-remote-write - open-telemetry-metrics - open-telemetry-logs - open-telemetry-traces deploy_type: type: string enum: - saas - local default: saas user_config: type: object generation_id: type: integer required: - type - user_config - generation_id required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_source_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/source: post: summary: Add source tags: - Pipeline Nodes description: Adds a source to a pipeline requestBody: content: application/json: schema: type: object additionalProperties: false allOf: - if: properties: type: enum: - http deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: HTTP description: Receive data from incoming HTTP requests additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 decoding: oneOf: - enum: - bytes - enum: - json - enum: - ndjson - enum: - auto ui: display_type: list errorMessage: must be equal to one of the allowed values description: The decoding method for converting frames into data events. title: Decoding Method default: auto required: - decoding - if: properties: type: enum: - splunk-hec deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Splunk HEC description: Receive Splunk logs additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - s3 deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS S3 description: Collect data from object contents in AWS S3 additionalProperties: false properties: sqs_queue_url: type: string title: SQS Queue URL description: The URL of a AWS SQS queue configured to receive S3 bucket notifications for the desired S3 buckets format: uri errorMessage: format: Must be a valid URI auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key compression: type: string title: Compression description: The compression format of the S3 objects enum: - auto - gzip - none - zstd default: auto region: type: string title: Region minLength: 1 description: The name of the source AWS region. required: - sqs_queue_url - auth - compression - region - if: properties: type: enum: - sqs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS SQS description: Collect messages from AWS SQS additionalProperties: false properties: queue_url: type: string title: Queue URL description: The URL of an AWS SQS queue format: uri errorMessage: format: Must be a valid URI maxLength: 128 auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id maxLength: 128 secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret maxLength: 128 required: - access_key_id - secret_access_key region: type: string title: Region minLength: 1 description: The name of the source AWS region. maxLength: 128 required: - queue_url - auth - region - if: properties: type: enum: - demo-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Demo Logs description: Generate log messages for testing and demonstration. additionalProperties: type: array title: Log Lines description: A list of user-provided log lines that will be sent through the pipeline infinitely. A maximum of 25 lines is allowed. The lines can be either plain text or ndjson. If ndjson is detected, the encoding for the source will be set to JSON, and lines that are not valid will show up as a visible component error. Otherwise, text encoding will be used. minItems: 1 maxItems: 25 items: type: string minLength: 1 maxLength: 4096 title: Line description: A single line of log data, either stringified JSON or plain text. ui: display_type: user-provided-lines allOf: - if: properties: format: enum: - user_provided required: - format then: properties: lines: type: array title: Log Lines description: A list of user-provided log lines that will be sent through the pipeline infinitely. A maximum of 25 lines is allowed. The lines can be either plain text or ndjson. If ndjson is detected, the encoding for the source will be set to JSON, and lines that are not valid will show up as a visible component error. Otherwise, text encoding will be used. minItems: 1 maxItems: 25 items: type: string minLength: 1 maxLength: 4096 title: Line description: A single line of log data, either stringified JSON or plain text. ui: display_type: user-provided-lines required: - lines - allOf: - errorMessage: "`lines` is only valid when `format` is `user_provided`" if: not: type: object properties: format: enum: - user_provided required: - format then: not: type: object properties: lines: type: array title: Log Lines description: A list of user-provided log lines that will be sent through the pipeline infinitely. A maximum of 25 lines is allowed. The lines can be either plain text or ndjson. If ndjson is detected, the encoding for the source will be set to JSON, and lines that are not valid will show up as a visible component error. Otherwise, text encoding will be used. minItems: 1 maxItems: 25 items: type: string minLength: 1 maxLength: 4096 title: Line description: A single line of log data, either stringified JSON or plain text. ui: display_type: user-provided-lines required: - lines properties: interval_ms: type: number title: Interval description: The amount of time to pause between each demo log. Fractions of a second are accepted to increase throughput. e.g. 500 ms == 120 lines per minute default: 1000 minimum: 500 maximum: 3600000 ui: display_type: milliseconds-to-human-readable format: title: Format description: The format of the log messages. All of the data is generated. ui: display_type: list errorMessage: must be one of the available types of demo data oneOf: - label: HTTP JSON description: HTTP request data in JSON format. enum: - json - label: Sensor JSON description: Data from environment sensors in JSON format. enum: - env_sensor - label: Financial JSON description: Financial data in JSON format. enum: - financial - label: Nginx Requests description: Nginx logs in text format. enum: - nginx - label: Apache Requests description: Apache request logs in text format. enum: - apache_common - label: Apache Errors description: Apache error logs in text format. enum: - apache_error - label: BSD Syslog description: Mock BSD syslog entries in rfc3164 text format. enum: - bsd_syslog - label: Syslog description: Mock syslog entries in rfc5424 text format. enum: - syslog - label: HTTP Metrics description: HTTP-based metrics in the Mezmo metrics JSON format. enum: - http_metrics - label: Generic Metrics description: Generic metrics in the Mezmo metrics JSON format enum: - generic_metrics - label: User-Provided Logs description: Custom logs provided by the user. These can either be plain text or serialized JSON. enum: - user_provided - label: Infrastructure Logs description: Infrastructure logs consisting of sysdig, nginx, HDFS and spark logs enum: - infrastructure - label: Kubernetes Logs description: Kubernetes logs in JSON format enum: - kubernetes default: json required: - format - if: properties: type: enum: - kinesis-firehose deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Kinesis Firehose description: Receive Kinesis Firehose data additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 format: type: string enum: - json - text default: json title: Data Format description: This specifies what the data format will be after it is base64 decoded. If it is JSON, it will be automatically parsed. required: - format - if: properties: type: enum: - fluent deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Fluent (Fluentd, Fluent Bit) description: Receive data from Fluentd or Fluent Bit additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 decoding: type: string description: The decoding method for converting frames into data events. enum: - bytes - json - ndjson title: Decoding Method default: json required: - decoding - if: properties: type: enum: - log-analysis deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Log Analysis description: Receive data directly from your Mezmo Log Analysis account additionalProperties: false - if: properties: type: enum: - log-analysis-ingestion deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Log Analysis Ingestion description: Receive unparsed source data directly from your Mezmo Log Analysis account. additionalProperties: type: string - if: properties: type: enum: - logstash deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Logstash description: Receive Logstash data additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 format: type: string enum: - json - text default: json title: Data Format description: The format of the logstash data required: - format - if: properties: type: enum: - mezmo-agent deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Agent description: Receive logs from the Mezmo Agent additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - mezmo-datadog-source deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Datadog Agent description: Send logs and metrics data directly from an installed datadog agent additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - pipeline-data-restoration deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Pipeline Data Restoration description: Receive data directly from your Mezmo Pipline Data Restoration additionalProperties: false - if: properties: type: enum: - webhook deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Webhook (WebSub) description: Receive data from incoming webhooks using the WebSub protocol. additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - kafka deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Kafka Source description: Collect events from Kafka compatible brokers additionalProperties: type: string properties: brokers: type: array title: Brokers uniqueItems: true minItems: 1 description: An array of key/val objects for each host/port of a Kafka broker. items: type: object title: Host/Port description: The host and port for the Kafka broker. additionalProperties: false properties: host: title: Host description: The hostname or IP address of the broker. anyOf: - type: string format: ipv4 errorMessage: format: Must be a valid IPv4 address - type: string format: hostname maxLength: 512 errorMessage: format: Must be a valid hostname port: type: number minimum: 1 maximum: 65535 title: Port description: The port that the broker listens on. required: - host - port topics: type: array title: Topics items: type: string title: Topic minLength: 1 maxLength: 256 description: The Kafka topic name. minItems: 1 description: List of Kafka topics names to read events from. group_id: type: string title: Consumer Group minLength: 1 description: The consumer group name to be used to consume events from Kafka. tls_enabled: type: boolean title: TLS enabled description: Whether or not to require TLS for outgoing connections. default: true sasl_enabled: type: boolean title: SASL/SCRAM Enabled description: Enable SASL/SCRAM authentication with Kafka default: true decoding_codec: type: string enum: - bytes - json title: Decoding description: Configures how events are decoded from raw bytes default: bytes if: properties: sasl_enabled: enum: - true then: properties: sasl_mechanism: type: string enum: - SCRAM-SHA-512 - SCRAM-SHA-256 - PLAIN title: SASL description: The Kafka SASL/SCRAM mechanism default: PLAIN sasl_username: type: string title: Username description: The SASL/SCRAM username minLength: 1 sasl_password: type: string title: Password description: The SASL/SCRAM username ui: display_type: secret minLength: 1 required: - sasl_mechanism - sasl_username - sasl_password required: - brokers - topics - group_id - decoding_codec - if: properties: type: enum: - azure-event-hub deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Azure Event Hub Source description: Collect events from Azure Event Hub using Azure Kafka Endpoint additionalProperties: false properties: connection_string: type: string title: Connection String minLength: 1 maxLength: 512 description: The Connection String as it appears in hub consumer SAS Policy ui: display_type: secret namespace: type: string title: Namespace minLength: 1 maxLength: 256 description: The Event Hub Namespace topics: type: array title: Event Hub Names items: type: string title: Name minLength: 1 maxLength: 256 description: Azure Event Hub instance name. minItems: 1 description: The list of Azure Event Hub name(s) to read events from. group_id: type: string title: Consumer Group minLength: 1 maxLength: 256 description: The consumer group name that this consumer belongs to. decoding_codec: type: string enum: - bytes - json title: Decoding description: Configures how events are decoded from raw bytes default: bytes required: - connection_string - namespace - topics - group_id - decoding_codec - if: properties: type: enum: - prometheus-remote-write deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Prometheus Remote Write description: Receive prometheus remote write protobuf data from HTTP requests additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - open-telemetry-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry Metrics description: Receive metrics from OpenTelemetry over HTTP additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - open-telemetry-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry Logs description: Receive logs from OpenTelemetry over HTTP additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - open-telemetry-traces deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry Traces description: Receive traces from OpenTelemetry over HTTP additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 properties: type: type: string enum: - http - splunk-hec - s3 - sqs - demo-logs - kinesis-firehose - fluent - log-analysis - log-analysis-ingestion - logstash - mezmo-agent - mezmo-datadog-source - pipeline-data-restoration - webhook - kafka - azure-event-hub - prometheus-remote-write - open-telemetry-metrics - open-telemetry-logs - open-telemetry-traces deploy_type: type: string enum: - saas - local default: saas title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description user_config: type: object gateway_route_id: type: string format: uuid required: - type - user_config required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/timestamp-format/simulate: post: summary: Simulate timestamp pattern tags: - Sampling and Simulation description: Simulate data with a timestamp format requestBody: content: application/json: schema: type: object properties: format: type: string format: strftime_datetime errorMessage: must be a valid strftime datetime format minLength: 1 values: type: array items: type: string minItems: 1 required: - format - values required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/transform/{pipeline_transform_id}: delete: summary: Remove transform tags: - Pipeline Nodes description: Removes transform from pipeline parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_transform_id required: true responses: "200": description: Default Response get: summary: Fetch transform tags: - Pipeline Nodes description: Fetch transform configuration parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_transform_id required: true responses: "200": description: Default Response put: summary: Update transform tags: - Pipeline Nodes description: Updates a transform of a pipeline requestBody: content: application/json: schema: type: object allOf: - if: properties: type: enum: - filter deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Filter description: Define condition(s) to include or exclude events from the pipeline additionalProperties: false properties: action: title: Action description: How to handle events matching this criteria oneOf: - label: Allow events matching this criteria enum: - allow - label: Drop events matching this criteria enum: - drop default: allow ui: display_type: radio-group conditional: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine required: - action - conditional - if: properties: type: enum: - sample deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sample description: Sample data at a given rate, retaining only a subset of data events for further processing. additionalProperties: false properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 events will be forwarded and the rest will be dropped minimum: 2 maximum: 10000 default: 10 always_include: title: Always Include description: Certain events may always want to be included (not sampled), regardless of the sample rate. Events matching this criteria will always show up in the results. additionalProperties: type: - string - number dependencies: field: - str_operator str_operator: - field allOf: - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: errorMessage: pattern: Value must be a number type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 pattern: ^\s*-?(\d*\.)?\d+\s*$ required: - value - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix - percent_change_greater - percent_change_greater_or_equal - percent_change_less - percent_change_less_or_equal - value_change_greater - value_change_greater_or_equal - value_change_less - value_change_less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 required: - value - if: title: Operator properties: str_operator: type: string enum: - contains - equal - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix ui: display_type: list-operators title: Operator required: - str_operator then: title: Case sensitive properties: case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive comparison? default: true properties: field: type: string title: Field description: The JSON field to use in a condition to always include in sampling pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - is_metric - is_array - is_boolean - is_empty - is_null - is_number - is_object - is_string - exists - ends_with - starts_with - contains_token_term - contains_token_prefix - regex_match ui: display_type: list-operators title: Operator description: The comparison operator negate: type: boolean default: false title: Negate description: Negate the comparison expression required: - rate - if: properties: type: enum: - drop-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Remove Fields description: Remove fields additionalProperties: false properties: fields: type: array title: Fields description: A list of fields to be removed items: type: string title: Field pattern: ^(?:metadata|\.|(?:metadata|message)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields). Note that dropping an event's message or timestamp is not allowed minItems: 1 required: - fields - if: properties: type: enum: - parse deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Parse description: Parse a specified field using the chosen parser additionalProperties: type: object errorMessage: properties: target_field: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) allOf: - if: properties: parser: enum: - parse_key_value then: properties: options: $ref: "#key-value-parser-options" - if: properties: parser: enum: - parse_timestamp then: properties: options: $ref: "#timestamp-parser-options" required: - options - if: properties: parser: enum: - parse_grok then: properties: options: $ref: "#grok-pattern-parser-options" required: - options - if: properties: parser: enum: - parse_regex then: properties: options: allOf: - $ref: "#regex-parser-options" ui: display_type: regex-pattern required: - options - if: properties: parser: enum: - parse_apache_log then: properties: options: $ref: "#apache-parser-options" required: - options - if: properties: parser: enum: - parse_nginx_log then: properties: options: $ref: "#nginx-parser-options" required: - options - if: properties: parser: enum: - parse_cef then: properties: options: $ref: "#cef-parser-options" - if: properties: parser: enum: - parse_csv then: properties: options: $ref: "#csv-parser-options" properties: field: type: string title: Field description: The JSON field whose value should be parsed. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: oneOf: - type: "null" - type: string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference title: Target field description: The field into which the parsed value should be inserted. Leave blank to insert the parsed data into the original field. default: null parser: oneOf: - label: AWS Cloudfront Access Log group: Line Parsers ui: {} enum: - aws_cloudfront_access_log - label: AWS Cloudwatch Log group: Line Parsers ui: {} enum: - parse_aws_cloudwatch_log_subscription_message - label: AWS Load Balancer Log group: Line Parsers ui: {} enum: - parse_aws_alb_log - label: AWS S3 Access Log group: Line Parsers ui: {} enum: - aws_s3_access_log - label: Apache Log group: Line Parsers ui: {} enum: - parse_apache_log - label: CSV group: Line Parsers ui: {} enum: - parse_csv - label: Common Event Format (CEF) Log group: Line Parsers ui: {} enum: - parse_cef - label: Common Log group: Line Parsers ui: {} enum: - parse_common_log - label: Elasticsearch Log group: Line Parsers ui: {} enum: - elasticsearch_log - label: Golang Log group: Line Parsers ui: {} enum: - golang_log - label: Google Log group: Line Parsers ui: {} enum: - parse_glog - label: Grok Pattern group: Line Parsers ui: {} enum: - parse_grok - label: Integer group: Field Parsers ui: {} enum: - parse_int - label: JSON group: Line Parsers ui: {} enum: - parse_json - label: Key/Value group: Line Parsers ui: {} enum: - parse_key_value - label: Kubernetes Log group: Line Parsers ui: {} enum: - parse_klog - label: Linux Authorization group: Line Parsers ui: {} enum: - parse_linux_authorization - label: Mongo Log group: Line Parsers ui: {} enum: - mongo_log - label: Mongo3 Log group: Line Parsers ui: {} enum: - mongo3_log - label: Nginx Log group: Line Parsers ui: {} enum: - parse_nginx_log - label: PSQL Log group: Line Parsers ui: {} enum: - psql_log - label: Query String group: Field Parsers ui: {} enum: - parse_query_string - label: Regular Expression group: Line Parsers ui: feature_flag: pipeline-parser-regex enum: - parse_regex - label: Syslog group: Line Parsers ui: {} enum: - parse_syslog - label: Timestamp group: Field Parsers ui: {} enum: - parse_timestamp - label: Tokens group: Field Parsers ui: {} enum: - parse_tokens - label: URL group: Field Parsers ui: {} enum: - parse_url - label: User Agent group: Field Parsers ui: {} enum: - parse_user_agent - label: VPC Flow Log group: Line Parsers ui: {} description: Parse VPC logs using default format for fields from v2 to v4 enum: - parse_aws_vpc_flow_log ui: display_type: list errorMessage: must be one of allowed parsers title: Parser description: The kind of parser to use against the input value from "field". required: - field - parser - if: properties: type: enum: - parse-sequentially deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Parse Sequentially description: Parse fields using one or more parsers in order, sending out the event on the first successful match. additionalProperties: false errorMessage: properties: target_field: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) properties: field: type: string title: Field description: The field whose value should be parsed. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: oneOf: - type: "null" - type: string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference title: Target field description: The field into which the parsed value should be inserted. Leave blank to insert the parsed data into the original field. default: null parsers: type: array title: List of parsers description: The list of parsers to use in order against the input valuefrom "field", short-circuiting on the first successful match. items: type: object allOf: - if: properties: parser: enum: - parse_key_value then: properties: options: $ref: "#key-value-parser-options" - if: properties: parser: enum: - parse_timestamp then: properties: options: $ref: "#timestamp-parser-options" required: - options - if: properties: parser: enum: - parse_grok then: properties: options: $ref: "#grok-pattern-parser-options" required: - options - if: properties: parser: enum: - parse_regex then: properties: options: allOf: - $ref: "#regex-parser-options" ui: display_type: regex-pattern required: - options - if: properties: parser: enum: - parse_apache_log then: properties: options: $ref: "#apache-parser-options" required: - options - if: properties: parser: enum: - parse_nginx_log then: properties: options: $ref: "#nginx-parser-options" required: - options - if: properties: parser: enum: - parse_cef then: properties: options: $ref: "#cef-parser-options" - if: properties: parser: enum: - parse_csv then: properties: options: $ref: "#csv-parser-options" properties: label: oneOf: - type: "null" - type: string maxLength: 20 title: Title description: An arbitrary name you choose to identify the results of this processor for use when connecting them to other components. parser: oneOf: - label: AWS Cloudfront Access Log group: Line Parsers ui: {} enum: - aws_cloudfront_access_log - label: AWS Cloudwatch Log group: Line Parsers ui: {} enum: - parse_aws_cloudwatch_log_subscription_message - label: AWS Load Balancer Log group: Line Parsers ui: {} enum: - parse_aws_alb_log - label: AWS S3 Access Log group: Line Parsers ui: {} enum: - aws_s3_access_log - label: Apache Log group: Line Parsers ui: {} enum: - parse_apache_log - label: CSV group: Line Parsers ui: {} enum: - parse_csv - label: Common Event Format (CEF) Log group: Line Parsers ui: {} enum: - parse_cef - label: Common Log group: Line Parsers ui: {} enum: - parse_common_log - label: Elasticsearch Log group: Line Parsers ui: {} enum: - elasticsearch_log - label: Golang Log group: Line Parsers ui: {} enum: - golang_log - label: Google Log group: Line Parsers ui: {} enum: - parse_glog - label: Grok Pattern group: Line Parsers ui: {} enum: - parse_grok - label: Integer group: Field Parsers ui: {} enum: - parse_int - label: JSON group: Line Parsers ui: {} enum: - parse_json - label: Key/Value group: Line Parsers ui: {} enum: - parse_key_value - label: Kubernetes Log group: Line Parsers ui: {} enum: - parse_klog - label: Linux Authorization group: Line Parsers ui: {} enum: - parse_linux_authorization - label: Mongo Log group: Line Parsers ui: {} enum: - mongo_log - label: Mongo3 Log group: Line Parsers ui: {} enum: - mongo3_log - label: Nginx Log group: Line Parsers ui: {} enum: - parse_nginx_log - label: PSQL Log group: Line Parsers ui: {} enum: - psql_log - label: Query String group: Field Parsers ui: {} enum: - parse_query_string - label: Regular Expression group: Line Parsers ui: feature_flag: pipeline-parser-regex enum: - parse_regex - label: Syslog group: Line Parsers ui: {} enum: - parse_syslog - label: Timestamp group: Field Parsers ui: {} enum: - parse_timestamp - label: Tokens group: Field Parsers ui: {} enum: - parse_tokens - label: URL group: Field Parsers ui: {} enum: - parse_url - label: User Agent group: Field Parsers ui: {} enum: - parse_user_agent - label: VPC Flow Log group: Line Parsers ui: {} description: Parse VPC logs using default format for fields from v2 to v4 enum: - parse_aws_vpc_flow_log ui: display_type: list errorMessage: must be one of allowed parsers title: Parser description: The kind of parser to use against the input value from "field". _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. default: "" required: - parser - _output_name additionalProperties: type: object minItems: 1 ui: display_type: multi-output-map unique_parser_and_options: true errorMessage: uniqueItemProperties: parser can not be defined more than once required: - field - parsers - if: properties: type: enum: - route deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Route description: Route data based on whether it matches (or doesn't match) logical comparisons additionalProperties: false properties: conditionals: type: array minItems: 1 title: Conditionals description: A list of complete conditionals, including expressions and label items: type: object additionalProperties: false title: Conditional description: A list of conditions and a label properties: conditional: $ref: "#condition-group" label: type: string minLength: 1 maxLength: 255 title: Label description: An arbitrary name you choose to identify the results of this processor for use when connecting them to other components. _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. default: "" required: - conditional - label - _output_name required: - conditionals - if: properties: type: enum: - stringify deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Stringify description: Stringify messages as JSON additionalProperties: false - if: properties: type: enum: - compact-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Compact Fields description: Remove empty values from a list of fields additionalProperties: false properties: fields: type: array title: Fields description: A list of fields to remove empty values from items: type: string title: Field description: Field to remove empty value from pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference minItems: 1 options: type: object additionalProperties: false title: Compact Options description: Additional options for removing empty values from fields properties: compact_array: type: boolean title: Compact Array description: Remove empty arrays from a field default: true compact_object: type: boolean title: Compact Object description: Remove empty objects from a field default: true required: - fields - if: properties: type: enum: - flatten-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Flatten Fields description: Flattens the object or array value of a field into a single level representation additionalProperties: false properties: fields: type: array title: Fields description: Define specific nested objects you wish to flatten. This will not flatten the entire object, just the specified fields. Leave blank to flatten the entire object items: type: string title: Field description: Field to flatten pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference options: type: object additionalProperties: false title: Flatten Options description: Additional options for flattening fields properties: delimiter: type: string default: _ title: Delimiter description: When flattening, the nested field labels will be appended to the higher level field names along with a delimiter to allow understanding of the original structure. The default is an underscore (_) to ensure no collisions with the . notation used for field identifiers minLength: 1 maxLength: 1 - if: properties: type: enum: - dedupe deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Dedupe description: Dedupe (remove duplicates) from the data stream additionalProperties: false properties: number_of_events: type: number description: Number of events to compare across title: Number of Events minimum: 2 maximum: 5000 default: 5000 comparison_type: type: string enum: - Ignore - Match description: "Match: Only compares across the fields which are specified by the user.Ignore: Compares everything but the fields specified by the user." title: Comparison Type default: Match fields: type: array title: Fields description: A list of fields on which to base deduping items: type: string title: Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference minItems: 1 required: - number_of_events - comparison_type - fields - if: properties: type: enum: - encrypt-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Encrypt Field description: Encrypts the value of the provided field additionalProperties: false allOf: - if: properties: algorithm: enum: - AES-256-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key properties: field: type: string title: Field description: Field to encrypt. The value of the field must be a primitive (string, number, boolean) pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference algorithm: oneOf: - label: AES-256-CFB (key = 32 characters, iv = 16 characters) description: AES-256-CFB (key = 32 characters, iv = 16 characters) enum: - AES-256-CFB - label: AES-192-CFB (key = 24 characters, iv = 16 characters) description: AES-192-CFB (key = 24 characters, iv = 16 characters) enum: - AES-192-CFB - label: AES-128-CFB (key = 16 characters, iv = 16 characters) description: AES-128-CFB (key = 16 characters, iv = 16 characters) enum: - AES-128-CFB - label: AES-256-OFB (key = 32 characters, iv = 16 characters) description: AES-256-OFB (key = 32 characters, iv = 16 characters) enum: - AES-256-OFB - label: AES-192-OFB (key = 24 characters, iv = 16 characters) description: AES-192-OFB (key = 24 characters, iv = 16 characters) enum: - AES-192-OFB - label: AES-128-OFB (key = 16 characters, iv = 16 characters) description: AES-128-OFB (key = 16 characters, iv = 16 characters) enum: - AES-128-OFB - label: AES-256-CTR (key = 32 characters, iv = 16 characters) description: AES-256-CTR (key = 32 characters, iv = 16 characters) enum: - AES-256-CTR - label: AES-192-CTR (key = 24 characters, iv = 16 characters) description: AES-192-CTR (key = 24 characters, iv = 16 characters) enum: - AES-192-CTR - label: AES-128-CTR (key = 16 characters, iv = 16 characters) description: AES-128-CTR (key = 16 characters, iv = 16 characters) enum: - AES-128-CTR - label: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-PKCS7 - label: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-PKCS7 - label: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-PKCS7 - label: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ANSIX923 - label: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ANSIX923 - label: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ANSIX923 - label: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO7816 - label: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO7816 - label: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO7816 - label: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO10126 - label: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO10126 - label: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO10126 ui: display_type: list title: Encryption algorithm description: The encryption algorithm to use key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 32 iv_field: type: string title: Initialization vector (IV) field description: The field in which to store the generated initialization vector, IV. Each encrypted value will have a unique IV pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference encode_raw_bytes: type: boolean title: Encode encrypted field and IV as Base 64 text description: Encode the encrypted value and generated initialization vector as Base64 text default: true required: - field - algorithm - key - iv_field - if: properties: type: enum: - decrypt-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Decrypt Field additionalProperties: false description: Decrypts the value of the provided field allOf: - if: properties: algorithm: enum: - AES-256-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key properties: field: type: string title: Field description: Field to decrypt. The value of the field must be a string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference algorithm: oneOf: - label: AES-256-CFB (key = 32 characters, iv = 16 characters) description: AES-256-CFB (key = 32 characters, iv = 16 characters) enum: - AES-256-CFB - label: AES-192-CFB (key = 24 characters, iv = 16 characters) description: AES-192-CFB (key = 24 characters, iv = 16 characters) enum: - AES-192-CFB - label: AES-128-CFB (key = 16 characters, iv = 16 characters) description: AES-128-CFB (key = 16 characters, iv = 16 characters) enum: - AES-128-CFB - label: AES-256-OFB (key = 32 characters, iv = 16 characters) description: AES-256-OFB (key = 32 characters, iv = 16 characters) enum: - AES-256-OFB - label: AES-192-OFB (key = 24 characters, iv = 16 characters) description: AES-192-OFB (key = 24 characters, iv = 16 characters) enum: - AES-192-OFB - label: AES-128-OFB (key = 16 characters, iv = 16 characters) description: AES-128-OFB (key = 16 characters, iv = 16 characters) enum: - AES-128-OFB - label: AES-256-CTR (key = 32 characters, iv = 16 characters) description: AES-256-CTR (key = 32 characters, iv = 16 characters) enum: - AES-256-CTR - label: AES-192-CTR (key = 24 characters, iv = 16 characters) description: AES-192-CTR (key = 24 characters, iv = 16 characters) enum: - AES-192-CTR - label: AES-128-CTR (key = 16 characters, iv = 16 characters) description: AES-128-CTR (key = 16 characters, iv = 16 characters) enum: - AES-128-CTR - label: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-PKCS7 - label: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-PKCS7 - label: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-PKCS7 - label: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ANSIX923 - label: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ANSIX923 - label: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ANSIX923 - label: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO7816 - label: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO7816 - label: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO7816 - label: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO10126 - label: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO10126 - label: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO10126 ui: display_type: list title: Decryption algorithm description: The algorithm with which the data was encrypted key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 32 iv_field: type: string title: Initialization vector (IV) field description: The field from which to read the initialization vector, IV pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference decode_raw_bytes: type: boolean title: Decode the encrypted value and IV as Base 64 text description: Decode the encrypted value and initialization vector as Base64 text default: true required: - field - algorithm - key - iv_field - if: properties: type: enum: - js-script deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Script Execution description: Use JavaScript to reshape and transform your data. You can combine multiple actions like filtering, dropping, mapping, and casting inside of a single js script. additionalProperties: true properties: script: type: string title: Script ui: display_type: code-editor options: language: js description: The JavaScript representation of the event transformation. minLength: 1 maxLength: 10000 default: > // Modify the event using a subset of the JavaScript language. // The function must return the modified event function processEvent(message, metadata, timestamp, annotations) { return message } - if: properties: type: enum: - unroll deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Unroll description: Takes an array of events and emits them all as individual events. additionalProperties: false properties: field: type: string title: Field description: The JSON field name that contains an array of events pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference values_only: type: boolean title: Values only? description: When enabled, the values from the specified array field will be emitted as new events. Otherwise, the original event will be duplicated for each value in the array field, with the unrolled value present in the field specified. default: true required: - field - if: properties: type: enum: - vrl deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: VRL description: Modify your observability data using Vector Remap Language (VRL) additionalProperties: false properties: source: type: string title: VRL Source description: VRL code to execute for each event ui: display_type: code-editor options: language: vrl minLength: 1 default: .message = .message required: - source - if: properties: type: enum: - reduce deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Reduce description: Combine multiple events over time into one based on a set of criteria additionalProperties: false properties: duration_ms: type: number maximum: 7200000 title: Duration description: The amount of time (in milliseconds) to allow streaming events to accumulate into a single "reduced" event. The process repeats indefinitely, or until an "ends when" condition is satisfied. ui: display_type: milliseconds-to-human-readable default: 30000 max_events: type: number minimum: 1 maximum: 100000 title: Max Events description: The maximum number of events that can be included in a time window (specified by Duration above). The reduce operation will stop once it has reached this number of events, regardless of whether the Duration seconds have elapsed. group_by: type: array minItems: 1 title: Group By Field Path items: type: string title: Field description: The dot-notated property path errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ description: Before reducing, group events based on matching data from each of these field paths. Supports nesting via dot-notation. date_formats: type: array title: Date Formats description: Describes which root-level properties are dates, and their expected format. Dot-notation is supported, but nested field lookup paths will be an error. minItems: 1 items: type: object additionalProperties: false properties: field: type: string title: Date Field Path description: Specifies a root-level path property that contains a date value. maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ format: type: string title: Date Format description: The template describing the date format format: strftime_datetime errorMessage: must be a valid strftime datetime format maxLength: 200 required: - field - format merge_strategies: type: array title: Merge Strategy per Field minItems: 1 description: Specify merge strategies for individual root-level properties. Dot-notation is supported, but nested field lookup paths will be an error. items: type: object additionalProperties: false properties: field: type: string title: Field Path description: This is a root-level path property to apply a merge strategy to its value maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ strategy: oneOf: - label: Array description: Append each value onto an array enum: - array - label: Smallest array description: Keep the smallest array within each reduce time window enum: - shortest_array - label: Largest array description: Keep the largest array within each reduce time window enum: - longest_array - label: Unique Array description: All unique values go into a flattened array enum: - flat_unique - label: Join with space description: Concatenate into a single space-delimited string enum: - concat - label: Join with newline description: Concatenate into a single newline-delimited string enum: - concat_newline - label: Join description: Concatenate into a single string without using a delimiter enum: - concat_raw - label: Keep first description: Keep only the first value and discard the rest enum: - discard - label: Keep last description: Keep only the last non-null value enum: - retain - label: Min value description: Keep the minimum numeric value within each reduce time window enum: - min - label: Max value description: Keep the maximum numeric value within each reduce time window enum: - max - label: Sum description: Add numeric values together enum: - sum ui: display_type: list errorMessage: must be one of the allowed merge strategies title: Merge Strategy description: The merge strategy to be used for the specified property required: - field - strategy flush_condition: title: Event Flush Condition description: Force accumulated event reduction to flush the result when a conditional expression evaluates to true on an inbound event. allOf: - type: object properties: when: type: string enum: - none - starts_when - ends_when default: none ui: display_type: list errorMessage: "" title: Flush When required: - when - if: properties: when: not: enum: - none then: properties: conditional: allOf: - $ref: "#condition-group" ui: display_type: rules-engine required: - conditional - if: properties: type: enum: - map-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Map Fields description: Maps data from one field to another, either by moving or copying. additionalProperties: false properties: mappings: type: array title: Mappings description: A list of field mappings. Mappings are applied in the order they are defined. minItems: 1 ui: display_type: sortable items: type: object properties: source_field: type: string title: Source field description: The field to copy data from pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: type: string title: Target field description: The field to copy data into pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference drop_source: type: boolean title: Drop source? description: When enabled, the source field is dropped after the data is copied to the target field. Otherwise, it is preserved. default: false overwrite_target: type: boolean title: Overwrite target? description: When enabled, any existing data in the target field is overwritten. Otherwise, the target field will be preserved and this mapping will have no effect. default: false required: - source_field - target_field - drop_source - overwrite_target if: properties: drop_source: enum: - true required: - drop_source then: properties: source_field: type: string title: Source field description: The field to copy data from pattern: ^(?:metadata|\.|(?:metadata|message)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields). Note that dropping an event's message or timestamp is not allowed required: - source_field required: - mappings - if: properties: type: enum: - metrics-tag-cardinality-limit deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Tag Cardinality Limit (Metrics) description: Limits the cardinality of metric events by either dropping events or tags that exceed a specified value limit additionalProperties: false allOf: - if: title: Mode properties: mode: type: string enum: - exact then: title: Value Limit properties: value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags minimum: 1 maximum: 500 required: - value_limit - if: title: Mode properties: mode: type: string enum: - probabilistic then: title: Value Limit properties: value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags minimum: 1 maximum: 5000 required: - value_limit properties: tags: type: array title: Tags description: A list of tags to apply cardinality limits. If none are provided, all tags will be considered. items: type: string title: Tag minLength: 1 maxLength: 100 minItems: 0 maxItems: 10 exclude_tags: type: array title: Exclude Tags description: A list of tags to explicitly exclude from cardinality limits. items: type: string title: Tag minLength: 1 maxLength: 100 minItems: 0 maxItems: 10 action: type: string title: Action description: The action to take when a tag's cardinality exceeds the value limit enum: - drop_tag - drop_event default: drop_event value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags mode: type: string title: Mode description: The method to used to reduce tag value cardinality enum: - exact - probabilistic default: exact - if: properties: type: enum: - aggregate deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Aggregate description: Aggregates multiple events into a single event using either a tumbling interval window or a sliding interval window. additionalProperties: false properties: group_by: type: array minItems: 1 maxItems: 100 title: Group By Field Path description: When aggregating, group events based on matching values from each of these field paths. Supports nesting via dot-notation. items: type: string title: Field description: The dot-notated property path errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ ui: default_hint: - .name - .namespace - .tags evaluate: additionalProperties: type: string title: Evaluate description: How events should be aggregated within an interval. allOf: - if: properties: operation: enum: - CUSTOM then: properties: script: type: string title: Script ui: display_type: code-editor options: language: js description: The JavaScript representation of an event aggregation. minLength: 1 maxLength: 2048 default: > // Perform custom aggregation of the current event into the accumulated event. // The accumulated event is persisted across the entire aggregation window. // The function must return the accumulated event, which will be emitted when // the aggregation window elapses. function aggregateEvent(accum, event, metadata, timestamp, annotations) { return accum } required: - script properties: operation: title: Operation oneOf: - label: add description: Adds new values together if they are of the same type. For counters and gauges the values are added together. For sets, this is a union of the elements. For distribution, the samples are concatenated. For histograms, values in the same bucket will be added together. enum: - DEFAULT - label: sum description: Values are added together. Works for the GAUGE and COUNT metric types. enum: - SUM - label: minimum description: Get the minimum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MIN - label: maximum description: Get the maximum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MAX - label: average description: Values are averaged together. Works for the GAUGE and COUNT metric types. enum: - AVG - label: set intersection description: The aggregate produces a set of unique values. enum: - SET_INTERSECTION - label: distribution concatenation description: The aggregate produces a list of values combined together. enum: - DIST_CONCAT - label: custom description: Use a custom script to perform aggregation. enum: - CUSTOM ui: display_type: list required: - operation window: default: {} additionalProperties: type: integer title: Window Type description: Sliding windows can overlap, whereas tumbling windows are disjoint. For example, a tumbling window has a fixed time span and any events that fall within the "interval" will be used in the aggregate. Whereas in a sliding window, the aggregation occurs every "window duration" seconds after an event is encountered. allOf: - if: properties: type: enum: - sliding required: - type then: properties: window_min: type: integer title: Slide Duration (seconds) description: The amount of time to wait before creating a new sliding window for future events. minimum: 1 maximum: 200 default: 5 required: - window_min properties: type: type: string title: Type enum: - tumbling - sliding default: tumbling interval: type: integer title: Interval (seconds) description: The interval over which events are aggregated in seconds. default: 5 minimum: 1 maximum: 90000 required: - type - interval event_timestamp: type: string title: Event Timestamp description: The path to a field on the event that contains a epoch timestamp value. If an event does not have a timestamp field, events will be associated to the wall clock value when the event is processed. default: timestamp maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ conditional: ui: display_type: rules-engine title: Event Flush Condition description: Force accumulated event reduction to flush the result when a conditional expression evaluates to true on an inbound event. allOf: - $ref: "#condition-group" required: - evaluate - window - if: properties: type: enum: - event-to-metric deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Event to Metric description: Allows conversion between arbitrary events and a Metric. additionalProperties: false properties: metric_name: type: string title: Metric Name description: The machine name of the metric to emit. minLength: 1 maxLength: 128 pattern: ^[a-zA-Z][a-zA-Z0-9_:]*$ errorMessage: must be a valid metric name metric_kind: title: Kind description: The kind of metric to emit, Absolute or Incremental. Absolute metrics represent a complete value, and will generally replace an existing value for the metric in the target destination. Incremental metrics represent an additive value which is aggregated in the target destination to produce a new value. oneOf: - label: Incremental enum: - incremental - label: Absolute enum: - absolute default: incremental ui: display_type: radio-group metric_type: title: Type oneOf: - label: Counter enum: - counter - label: Sum enum: - sum - label: Gauge enum: - gauge default: counter ui: display_type: list value: title: Value description: The value of this metric. additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value then: title: Value properties: value: type: number title: Value description: The value. required: - value properties: value_type: title: Value type description: Either a value from an event field, or a new value input. oneOf: - label: Value from event field enum: - field - label: New value enum: - value default: field ui: display_type: radio-group value: type: - number - string required: - value_type namespace: title: Namespace description: The namespace for this metric. additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 128 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit the namespace. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: type: - string - "null" required: - value_type tags: type: array maxItems: 10 title: Tags description: A set of tags (also called labels) to apply to the metric event. items: type: object title: Tag description: A tag to apply to the metric event. allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value of this tag. Note that fields with highly-variable values will result in high-cardinality metrics, which may impact storage or cost in downstream destinations. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference fallback: type: string title: Fallback value description: The value to use for this tag when the field value is missing. minLength: 1 maxLength: 512 required: - value - if: properties: value_type: enum: - value then: title: Value properties: value: type: string title: Value description: The value for this tag. minLength: 1 maxLength: 128 required: - value properties: name: type: string title: Name description: The tag name pattern: ^[a-zA-Z][a-zA-Z0-9_]*$ minLength: 1 maxLength: 128 errorMessage: must be a valid metric tag name value_type: title: Value type description: Either a value from an event field, or a new value input. oneOf: - label: Value from event field enum: - field - label: New value enum: - value default: field ui: display_type: radio-group value: type: string required: - name - value_type - value additionalProperties: type: string required: - metric_name - metric_kind - metric_type - value - namespace - if: properties: type: enum: - clustering deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Clustering description: Finds clusters of similar values additionalProperties: false properties: cluster_field: type: string title: Field description: Field to cluster values on pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference default: .message similarity_threshold: type: number title: Similarity threshold description: A threshold from 0.0 (not similar) to 1.0 (exactly similar) that determines how close two values need to be to be consider as being part of the same cluster default: 0.4 minimum: 0 maximum: 1 max_clusters: type: integer title: Maximum cluster description: Maximum number of clusters kept cached by the transform default: 1000 minimum: 1 maximum: 2000 max_node_depth: type: integer title: Maximum tree depth description: Maximum depth of the prefix tree default: 4 minimum: 2 maximum: 8 max_children: type: integer title: Maximum children default: 100 minimum: 2 maximum: 200 description: Maximum number of children per node of the prefix tree required: - cluster_field - if: properties: type: enum: - redact deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Redact description: Detect and anonymize sensitive data in logs and metrics additionalProperties: false properties: field: type: string title: Field description: Detect patterns in the specified field. If the field is empty, the processor will look for patterns across all the fields. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 256 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference mask_patterns: type: array title: Redact Patterns description: List of PII patterns to detect and mask minItems: 1 maxItems: 20 items: type: object additionalProperties: type: string title: Redact Pattern description: A PII pattern to detect and/or anonymize allOf: - if: properties: action: enum: - Replace required: - action then: properties: replacement: type: string title: Replacement description: "The text to replace the detected pattern. Example: 000-00-0000" minLength: 1 maxLength: 512 required: - replacement - if: properties: action: enum: - Hash required: - action then: properties: algorithm: title: Algorithm description: Hash and encode (base16 or hex) matched PII data using the specified algorithm oneOf: - label: sha2 (sha-512_256) enum: - sha2 - label: sha3 (sha3-512) enum: - sha3 ui: display_type: list required: - algorithm - if: properties: name: enum: - custom required: - name then: properties: custom_options: allOf: - type: object additionalProperties: false properties: pattern: type: string title: Pattern description: The custom pattern minLength: 1 maxLength: 4096 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching default: true multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively default: false match_newline: type: boolean title: Dot all mode description: Dot matches newline character default: false crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" default: false ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines default: false required: - pattern title: Replace Expression description: The regular expression used to find replacements. Named capture groups are supported, e.g. `(?P<key>\w+):(?P<value>.+)`. ui: display_type: regex-pattern required: - custom_options properties: name: title: Pattern description: The PII pattern to match oneOf: - label: Credit card number enum: - credit_card - label: Email address enum: - email_address - label: IPv4 address enum: - ipv4_address - label: US or Canada phone number enum: - us_canada_phone_number - label: US social security number enum: - us_social_security_number - label: Custom pattern ui: feature_flag: pipeline-parser-regex enum: - custom ui: display_type: list action: type: string title: Action description: Action to take on the detected PII data enum: - Hash - Replace - Detect Only default: Replace required: - name - action required: - mask_patterns - if: properties: type: enum: - throttle deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Throttle description: Throttle (rate-limit) events passing through this component additionalProperties: false properties: threshold: type: number minimum: 1 maximum: 10000000 title: Limit description: The number of events to allow over the time window. If a Key Field is specified, the limit will be applied to events for each unique value of that field. default: 1 window_ms: type: number minimum: 1000 maximum: 2592000000 title: Window description: The time window over which the configured limit is applied ui: display_type: milliseconds-to-human-readable default: 5000 key_field: type: string title: Key Field description: When configured, the value from this field will be used to group events into separate buckets for throttling. If left blank, or if the event does not contain a value in this field, the event will be throttled as part of the default bucket. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference exclude: allOf: - $ref: "#condition-group" title: Exclude description: When configured, events matching this condition will not be throttled. ui: display_type: rules-engine required: - threshold - window_ms - if: properties: type: enum: - set-timestamp deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Set Timestamp description: Parse a list of fields using a chosen time format. First match will override the default timestamp of the event. additionalProperties: false properties: parsers: type: array title: List of Fields and Parsers description: The list of fields and parsers to use in order of priority, short-circuiting on the first successful match. ui: display_type: multi-output-map options: sortable: false items: type: object properties: field: type: string title: Field description: The field whose value will be parsed by the chosen date pattern pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference options: $ref: "#timestamp-parser-options" required: - field - options minItems: 1 required: - parsers - if: properties: type: enum: - data-profiler deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Data Profiler description: Profile the data sent through your pipeline. The system will generate annotations such as event type and total bytes to be used downstream. It will also normalize the app/host/level/line fields before sending the event downstream. A profile of the data including total, events, event size, log composition, and message templates will be generated as well. additionalProperties: false properties: app_fields: type: array title: App Path(s) description: A list of paths to look for the app value minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: App description: The path to the app maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .app - .application - .container host_fields: type: array title: Host Path(s) description: A list of paths to look for the host value minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Host description: The path to the host maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .host - .hostname level_fields: type: array title: Level Path(s) description: A list of paths to look for the log level minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Level description: The path to the level maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .level - .log_level line_fields: type: array title: Line Path(s) description: A list of paths to look for the event line minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Line description: The path to the line maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .line - .message label_fields: type: array title: Label Path(s) description: A list of paths to look for the event labels maxItems: 10 uniqueItems: true items: type: string title: Label description: The path to the labels maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .labels field_summary: type: boolean default: false title: Field Summary description: If enabled the profiler will attempt to summarize the values in fields of events passing through the node. ui: display_type: hidden ff_default_value: pipeline-node-transform-field-summary required: - app_fields - host_fields - level_fields - line_fields - if: properties: type: enum: - replace deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Replace description: Replace parts of a string using a regular expression or static string additionalProperties: type: object allOf: - if: properties: replacement_type: enum: - regex required: - replacement_type then: properties: options: allOf: - type: object additionalProperties: false properties: pattern: type: string minLength: 1 maxLength: 4096 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching default: true multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively default: false match_newline: type: boolean title: Dot all mode description: Dot matches newline character default: false crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" default: false ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines default: false required: - pattern title: Replace Expression description: The regular expression used to find replacements. Named capture groups are supported, e.g. `(?P<key>\w+):(?P<value>.+)`. ui: display_type: regex-pattern replace_with: type: string minLength: 1 maxLength: 4096 title: Replacement String description: The replacement string can be static or optionally contain capture group references. Capture groups can be positional or named. For example, to switch the position of two strings in the source string `onetwo`, the pattern could be `(?Pone)(?P.+)`, and the replacement would be `$two$one`. To do the same thing with positional capture groups, a pattern of `(\w{3})(\w{3})`, and a replacement of `$2$1` would have the same result.

To render a literal dollar sign `$` in the replacement, escape it with `$$`. So, if you had a capture group named `price`, to prepend a dollar sign to the replacement, you would use `$$$price`. required: - options - replace_with - if: properties: replacement_type: enum: - string required: - replacement_type then: properties: string_pattern: type: string title: String Pattern minLength: 1 maxLength: 4096 description: A static, case-sensitive string, which will be searched and replaced. replace_with: type: string minLength: 1 maxLength: 4096 title: Replacement String description: The static string used in each replacement. required: - string_pattern - replace_with properties: source_field: type: string title: Source Field description: The field that contains a string for replacement to operate on. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference max_times: type: number title: Maximum Replacements description: The maximum number of replacements to be made. If not specified, all occurrences will be replaced. minimum: 1 replacement_type: title: Replacement Type description: Specifies how the source string will be searched for replacement. ui: display_type: list errorMessage: must be one of the available pattern types oneOf: - label: String description: Replace a specified string one or more times. enum: - string - label: Regular Expression description: Replace using a regular expression. enum: - regex required: - source_field - replacement_type - if: properties: type: enum: - trace-sampling deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Trace Sampling description: Performs sampling of traces per a given rate using the Trace ID as the unique identifier. If a trace is rate selected, all spans of the trace are forwarded. If the Trace ID cannot be found, the event is dropped. allOf: - if: properties: sample_type: enum: - head then: properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 unique traces will be forwarded and all others will be dropped default: 10 minimum: 2 maximum: 100000 parent_span_id_field: type: "null" conditionals: type: "null" required: - rate - if: properties: sample_type: enum: - tail then: properties: parent_span_id_field: type: string default: .parent_span_id title: Parent Span ID description: The field to use as the parent span identifier pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference conditionals: type: array minItems: 1 title: Conditionals description: A list of complete conditionals, including expressions and a sample rate to maintain items: type: object additionalProperties: false title: Conditional description: A list of conditions and a sample rate to maintain properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 unique traces will be forwarded and all others will be dropped. A `rate = 1` means keep all traces that match the condition. minimum: 1 maximum: 100000 default: 10 conditional: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. required: - rate - conditional required: - parent_span_id_field - conditionals properties: sample_type: type: string description: The type of trace sampling to apply. enum: - head title: Sample Type default: head trace_id_field: type: string default: .trace_id title: Trace ID description: The field to use as the identifier of a trace. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - sample_type - trace_id_field properties: title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description type: type: string enum: - filter - sample - drop-fields - parse - parse-sequentially - route - stringify - compact-fields - flatten-fields - dedupe - encrypt-fields - decrypt-fields - js-script - unroll - vrl - reduce - map-fields - metrics-tag-cardinality-limit - aggregate - event-to-metric - clustering - redact - throttle - set-timestamp - data-profiler - replace - trace-sampling deploy_type: type: string enum: - saas - local default: saas user_config: type: object generation_id: type: integer inputs: type: array items: type: string required: - type - user_config - generation_id - inputs required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_transform_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/transform: post: summary: Add transform tags: - Pipeline Nodes description: Add a transform to a pipeline requestBody: content: application/json: schema: type: object allOf: - if: properties: type: enum: - filter deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Filter description: Define condition(s) to include or exclude events from the pipeline additionalProperties: false properties: action: title: Action description: How to handle events matching this criteria oneOf: - label: Allow events matching this criteria enum: - allow - label: Drop events matching this criteria enum: - drop default: allow ui: display_type: radio-group conditional: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine required: - action - conditional - if: properties: type: enum: - sample deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sample description: Sample data at a given rate, retaining only a subset of data events for further processing. additionalProperties: false properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 events will be forwarded and the rest will be dropped minimum: 2 maximum: 10000 default: 10 always_include: title: Always Include description: Certain events may always want to be included (not sampled), regardless of the sample rate. Events matching this criteria will always show up in the results. additionalProperties: type: - string - number dependencies: field: - str_operator str_operator: - field allOf: - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: errorMessage: pattern: Value must be a number type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 pattern: ^\s*-?(\d*\.)?\d+\s*$ required: - value - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix - percent_change_greater - percent_change_greater_or_equal - percent_change_less - percent_change_less_or_equal - value_change_greater - value_change_greater_or_equal - value_change_less - value_change_less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 required: - value - if: title: Operator properties: str_operator: type: string enum: - contains - equal - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix ui: display_type: list-operators title: Operator required: - str_operator then: title: Case sensitive properties: case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive comparison? default: true properties: field: type: string title: Field description: The JSON field to use in a condition to always include in sampling pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - is_metric - is_array - is_boolean - is_empty - is_null - is_number - is_object - is_string - exists - ends_with - starts_with - contains_token_term - contains_token_prefix - regex_match ui: display_type: list-operators title: Operator description: The comparison operator negate: type: boolean default: false title: Negate description: Negate the comparison expression required: - rate - if: properties: type: enum: - drop-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Remove Fields description: Remove fields additionalProperties: false properties: fields: type: array title: Fields description: A list of fields to be removed items: type: string title: Field pattern: ^(?:metadata|\.|(?:metadata|message)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields). Note that dropping an event's message or timestamp is not allowed minItems: 1 required: - fields - if: properties: type: enum: - parse deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Parse description: Parse a specified field using the chosen parser additionalProperties: type: object errorMessage: properties: target_field: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) allOf: - if: properties: parser: enum: - parse_key_value then: properties: options: $ref: "#key-value-parser-options" - if: properties: parser: enum: - parse_timestamp then: properties: options: $ref: "#timestamp-parser-options" required: - options - if: properties: parser: enum: - parse_grok then: properties: options: $ref: "#grok-pattern-parser-options" required: - options - if: properties: parser: enum: - parse_regex then: properties: options: allOf: - $ref: "#regex-parser-options" ui: display_type: regex-pattern required: - options - if: properties: parser: enum: - parse_apache_log then: properties: options: $ref: "#apache-parser-options" required: - options - if: properties: parser: enum: - parse_nginx_log then: properties: options: $ref: "#nginx-parser-options" required: - options - if: properties: parser: enum: - parse_cef then: properties: options: $ref: "#cef-parser-options" - if: properties: parser: enum: - parse_csv then: properties: options: $ref: "#csv-parser-options" properties: field: type: string title: Field description: The JSON field whose value should be parsed. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: oneOf: - type: "null" - type: string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference title: Target field description: The field into which the parsed value should be inserted. Leave blank to insert the parsed data into the original field. default: null parser: oneOf: - label: AWS Cloudfront Access Log group: Line Parsers ui: {} enum: - aws_cloudfront_access_log - label: AWS Cloudwatch Log group: Line Parsers ui: {} enum: - parse_aws_cloudwatch_log_subscription_message - label: AWS Load Balancer Log group: Line Parsers ui: {} enum: - parse_aws_alb_log - label: AWS S3 Access Log group: Line Parsers ui: {} enum: - aws_s3_access_log - label: Apache Log group: Line Parsers ui: {} enum: - parse_apache_log - label: CSV group: Line Parsers ui: {} enum: - parse_csv - label: Common Event Format (CEF) Log group: Line Parsers ui: {} enum: - parse_cef - label: Common Log group: Line Parsers ui: {} enum: - parse_common_log - label: Elasticsearch Log group: Line Parsers ui: {} enum: - elasticsearch_log - label: Golang Log group: Line Parsers ui: {} enum: - golang_log - label: Google Log group: Line Parsers ui: {} enum: - parse_glog - label: Grok Pattern group: Line Parsers ui: {} enum: - parse_grok - label: Integer group: Field Parsers ui: {} enum: - parse_int - label: JSON group: Line Parsers ui: {} enum: - parse_json - label: Key/Value group: Line Parsers ui: {} enum: - parse_key_value - label: Kubernetes Log group: Line Parsers ui: {} enum: - parse_klog - label: Linux Authorization group: Line Parsers ui: {} enum: - parse_linux_authorization - label: Mongo Log group: Line Parsers ui: {} enum: - mongo_log - label: Mongo3 Log group: Line Parsers ui: {} enum: - mongo3_log - label: Nginx Log group: Line Parsers ui: {} enum: - parse_nginx_log - label: PSQL Log group: Line Parsers ui: {} enum: - psql_log - label: Query String group: Field Parsers ui: {} enum: - parse_query_string - label: Regular Expression group: Line Parsers ui: feature_flag: pipeline-parser-regex enum: - parse_regex - label: Syslog group: Line Parsers ui: {} enum: - parse_syslog - label: Timestamp group: Field Parsers ui: {} enum: - parse_timestamp - label: Tokens group: Field Parsers ui: {} enum: - parse_tokens - label: URL group: Field Parsers ui: {} enum: - parse_url - label: User Agent group: Field Parsers ui: {} enum: - parse_user_agent - label: VPC Flow Log group: Line Parsers ui: {} description: Parse VPC logs using default format for fields from v2 to v4 enum: - parse_aws_vpc_flow_log ui: display_type: list errorMessage: must be one of allowed parsers title: Parser description: The kind of parser to use against the input value from "field". required: - field - parser - if: properties: type: enum: - parse-sequentially deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Parse Sequentially description: Parse fields using one or more parsers in order, sending out the event on the first successful match. additionalProperties: false errorMessage: properties: target_field: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) properties: field: type: string title: Field description: The field whose value should be parsed. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: oneOf: - type: "null" - type: string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference title: Target field description: The field into which the parsed value should be inserted. Leave blank to insert the parsed data into the original field. default: null parsers: type: array title: List of parsers description: The list of parsers to use in order against the input valuefrom "field", short-circuiting on the first successful match. items: type: object allOf: - if: properties: parser: enum: - parse_key_value then: properties: options: $ref: "#key-value-parser-options" - if: properties: parser: enum: - parse_timestamp then: properties: options: $ref: "#timestamp-parser-options" required: - options - if: properties: parser: enum: - parse_grok then: properties: options: $ref: "#grok-pattern-parser-options" required: - options - if: properties: parser: enum: - parse_regex then: properties: options: allOf: - $ref: "#regex-parser-options" ui: display_type: regex-pattern required: - options - if: properties: parser: enum: - parse_apache_log then: properties: options: $ref: "#apache-parser-options" required: - options - if: properties: parser: enum: - parse_nginx_log then: properties: options: $ref: "#nginx-parser-options" required: - options - if: properties: parser: enum: - parse_cef then: properties: options: $ref: "#cef-parser-options" - if: properties: parser: enum: - parse_csv then: properties: options: $ref: "#csv-parser-options" properties: label: oneOf: - type: "null" - type: string maxLength: 20 title: Title description: An arbitrary name you choose to identify the results of this processor for use when connecting them to other components. parser: oneOf: - label: AWS Cloudfront Access Log group: Line Parsers ui: {} enum: - aws_cloudfront_access_log - label: AWS Cloudwatch Log group: Line Parsers ui: {} enum: - parse_aws_cloudwatch_log_subscription_message - label: AWS Load Balancer Log group: Line Parsers ui: {} enum: - parse_aws_alb_log - label: AWS S3 Access Log group: Line Parsers ui: {} enum: - aws_s3_access_log - label: Apache Log group: Line Parsers ui: {} enum: - parse_apache_log - label: CSV group: Line Parsers ui: {} enum: - parse_csv - label: Common Event Format (CEF) Log group: Line Parsers ui: {} enum: - parse_cef - label: Common Log group: Line Parsers ui: {} enum: - parse_common_log - label: Elasticsearch Log group: Line Parsers ui: {} enum: - elasticsearch_log - label: Golang Log group: Line Parsers ui: {} enum: - golang_log - label: Google Log group: Line Parsers ui: {} enum: - parse_glog - label: Grok Pattern group: Line Parsers ui: {} enum: - parse_grok - label: Integer group: Field Parsers ui: {} enum: - parse_int - label: JSON group: Line Parsers ui: {} enum: - parse_json - label: Key/Value group: Line Parsers ui: {} enum: - parse_key_value - label: Kubernetes Log group: Line Parsers ui: {} enum: - parse_klog - label: Linux Authorization group: Line Parsers ui: {} enum: - parse_linux_authorization - label: Mongo Log group: Line Parsers ui: {} enum: - mongo_log - label: Mongo3 Log group: Line Parsers ui: {} enum: - mongo3_log - label: Nginx Log group: Line Parsers ui: {} enum: - parse_nginx_log - label: PSQL Log group: Line Parsers ui: {} enum: - psql_log - label: Query String group: Field Parsers ui: {} enum: - parse_query_string - label: Regular Expression group: Line Parsers ui: feature_flag: pipeline-parser-regex enum: - parse_regex - label: Syslog group: Line Parsers ui: {} enum: - parse_syslog - label: Timestamp group: Field Parsers ui: {} enum: - parse_timestamp - label: Tokens group: Field Parsers ui: {} enum: - parse_tokens - label: URL group: Field Parsers ui: {} enum: - parse_url - label: User Agent group: Field Parsers ui: {} enum: - parse_user_agent - label: VPC Flow Log group: Line Parsers ui: {} description: Parse VPC logs using default format for fields from v2 to v4 enum: - parse_aws_vpc_flow_log ui: display_type: list errorMessage: must be one of allowed parsers title: Parser description: The kind of parser to use against the input value from "field". _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. default: "" required: - parser - _output_name additionalProperties: type: object minItems: 1 ui: display_type: multi-output-map unique_parser_and_options: true errorMessage: uniqueItemProperties: parser can not be defined more than once required: - field - parsers - if: properties: type: enum: - route deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Route description: Route data based on whether it matches (or doesn't match) logical comparisons additionalProperties: false properties: conditionals: type: array minItems: 1 title: Conditionals description: A list of complete conditionals, including expressions and label items: type: object additionalProperties: false title: Conditional description: A list of conditions and a label properties: conditional: $ref: "#condition-group" label: type: string minLength: 1 maxLength: 255 title: Label description: An arbitrary name you choose to identify the results of this processor for use when connecting them to other components. _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. default: "" required: - conditional - label - _output_name required: - conditionals - if: properties: type: enum: - stringify deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Stringify description: Stringify messages as JSON additionalProperties: false - if: properties: type: enum: - compact-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Compact Fields description: Remove empty values from a list of fields additionalProperties: false properties: fields: type: array title: Fields description: A list of fields to remove empty values from items: type: string title: Field description: Field to remove empty value from pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference minItems: 1 options: type: object additionalProperties: false title: Compact Options description: Additional options for removing empty values from fields properties: compact_array: type: boolean title: Compact Array description: Remove empty arrays from a field default: true compact_object: type: boolean title: Compact Object description: Remove empty objects from a field default: true required: - fields - if: properties: type: enum: - flatten-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Flatten Fields description: Flattens the object or array value of a field into a single level representation additionalProperties: false properties: fields: type: array title: Fields description: Define specific nested objects you wish to flatten. This will not flatten the entire object, just the specified fields. Leave blank to flatten the entire object items: type: string title: Field description: Field to flatten pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference options: type: object additionalProperties: false title: Flatten Options description: Additional options for flattening fields properties: delimiter: type: string default: _ title: Delimiter description: When flattening, the nested field labels will be appended to the higher level field names along with a delimiter to allow understanding of the original structure. The default is an underscore (_) to ensure no collisions with the . notation used for field identifiers minLength: 1 maxLength: 1 - if: properties: type: enum: - dedupe deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Dedupe description: Dedupe (remove duplicates) from the data stream additionalProperties: false properties: number_of_events: type: number description: Number of events to compare across title: Number of Events minimum: 2 maximum: 5000 default: 5000 comparison_type: type: string enum: - Ignore - Match description: "Match: Only compares across the fields which are specified by the user.Ignore: Compares everything but the fields specified by the user." title: Comparison Type default: Match fields: type: array title: Fields description: A list of fields on which to base deduping items: type: string title: Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference minItems: 1 required: - number_of_events - comparison_type - fields - if: properties: type: enum: - encrypt-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Encrypt Field description: Encrypts the value of the provided field additionalProperties: false allOf: - if: properties: algorithm: enum: - AES-256-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key properties: field: type: string title: Field description: Field to encrypt. The value of the field must be a primitive (string, number, boolean) pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference algorithm: oneOf: - label: AES-256-CFB (key = 32 characters, iv = 16 characters) description: AES-256-CFB (key = 32 characters, iv = 16 characters) enum: - AES-256-CFB - label: AES-192-CFB (key = 24 characters, iv = 16 characters) description: AES-192-CFB (key = 24 characters, iv = 16 characters) enum: - AES-192-CFB - label: AES-128-CFB (key = 16 characters, iv = 16 characters) description: AES-128-CFB (key = 16 characters, iv = 16 characters) enum: - AES-128-CFB - label: AES-256-OFB (key = 32 characters, iv = 16 characters) description: AES-256-OFB (key = 32 characters, iv = 16 characters) enum: - AES-256-OFB - label: AES-192-OFB (key = 24 characters, iv = 16 characters) description: AES-192-OFB (key = 24 characters, iv = 16 characters) enum: - AES-192-OFB - label: AES-128-OFB (key = 16 characters, iv = 16 characters) description: AES-128-OFB (key = 16 characters, iv = 16 characters) enum: - AES-128-OFB - label: AES-256-CTR (key = 32 characters, iv = 16 characters) description: AES-256-CTR (key = 32 characters, iv = 16 characters) enum: - AES-256-CTR - label: AES-192-CTR (key = 24 characters, iv = 16 characters) description: AES-192-CTR (key = 24 characters, iv = 16 characters) enum: - AES-192-CTR - label: AES-128-CTR (key = 16 characters, iv = 16 characters) description: AES-128-CTR (key = 16 characters, iv = 16 characters) enum: - AES-128-CTR - label: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-PKCS7 - label: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-PKCS7 - label: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-PKCS7 - label: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ANSIX923 - label: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ANSIX923 - label: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ANSIX923 - label: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO7816 - label: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO7816 - label: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO7816 - label: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO10126 - label: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO10126 - label: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO10126 ui: display_type: list title: Encryption algorithm description: The encryption algorithm to use key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 32 iv_field: type: string title: Initialization vector (IV) field description: The field in which to store the generated initialization vector, IV. Each encrypted value will have a unique IV pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference encode_raw_bytes: type: boolean title: Encode encrypted field and IV as Base 64 text description: Encode the encrypted value and generated initialization vector as Base64 text default: true required: - field - algorithm - key - iv_field - if: properties: type: enum: - decrypt-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Decrypt Field additionalProperties: false description: Decrypts the value of the provided field allOf: - if: properties: algorithm: enum: - AES-256-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key properties: field: type: string title: Field description: Field to decrypt. The value of the field must be a string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference algorithm: oneOf: - label: AES-256-CFB (key = 32 characters, iv = 16 characters) description: AES-256-CFB (key = 32 characters, iv = 16 characters) enum: - AES-256-CFB - label: AES-192-CFB (key = 24 characters, iv = 16 characters) description: AES-192-CFB (key = 24 characters, iv = 16 characters) enum: - AES-192-CFB - label: AES-128-CFB (key = 16 characters, iv = 16 characters) description: AES-128-CFB (key = 16 characters, iv = 16 characters) enum: - AES-128-CFB - label: AES-256-OFB (key = 32 characters, iv = 16 characters) description: AES-256-OFB (key = 32 characters, iv = 16 characters) enum: - AES-256-OFB - label: AES-192-OFB (key = 24 characters, iv = 16 characters) description: AES-192-OFB (key = 24 characters, iv = 16 characters) enum: - AES-192-OFB - label: AES-128-OFB (key = 16 characters, iv = 16 characters) description: AES-128-OFB (key = 16 characters, iv = 16 characters) enum: - AES-128-OFB - label: AES-256-CTR (key = 32 characters, iv = 16 characters) description: AES-256-CTR (key = 32 characters, iv = 16 characters) enum: - AES-256-CTR - label: AES-192-CTR (key = 24 characters, iv = 16 characters) description: AES-192-CTR (key = 24 characters, iv = 16 characters) enum: - AES-192-CTR - label: AES-128-CTR (key = 16 characters, iv = 16 characters) description: AES-128-CTR (key = 16 characters, iv = 16 characters) enum: - AES-128-CTR - label: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-PKCS7 - label: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-PKCS7 - label: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-PKCS7 - label: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ANSIX923 - label: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ANSIX923 - label: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ANSIX923 - label: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO7816 - label: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO7816 - label: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO7816 - label: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO10126 - label: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO10126 - label: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO10126 ui: display_type: list title: Decryption algorithm description: The algorithm with which the data was encrypted key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 32 iv_field: type: string title: Initialization vector (IV) field description: The field from which to read the initialization vector, IV pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference decode_raw_bytes: type: boolean title: Decode the encrypted value and IV as Base 64 text description: Decode the encrypted value and initialization vector as Base64 text default: true required: - field - algorithm - key - iv_field - if: properties: type: enum: - js-script deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Script Execution description: Use JavaScript to reshape and transform your data. You can combine multiple actions like filtering, dropping, mapping, and casting inside of a single js script. additionalProperties: true properties: script: type: string title: Script ui: display_type: code-editor options: language: js description: The JavaScript representation of the event transformation. minLength: 1 maxLength: 10000 default: > // Modify the event using a subset of the JavaScript language. // The function must return the modified event function processEvent(message, metadata, timestamp, annotations) { return message } - if: properties: type: enum: - unroll deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Unroll description: Takes an array of events and emits them all as individual events. additionalProperties: false properties: field: type: string title: Field description: The JSON field name that contains an array of events pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference values_only: type: boolean title: Values only? description: When enabled, the values from the specified array field will be emitted as new events. Otherwise, the original event will be duplicated for each value in the array field, with the unrolled value present in the field specified. default: true required: - field - if: properties: type: enum: - vrl deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: VRL description: Modify your observability data using Vector Remap Language (VRL) additionalProperties: false properties: source: type: string title: VRL Source description: VRL code to execute for each event ui: display_type: code-editor options: language: vrl minLength: 1 default: .message = .message required: - source - if: properties: type: enum: - reduce deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Reduce description: Combine multiple events over time into one based on a set of criteria additionalProperties: false properties: duration_ms: type: number maximum: 7200000 title: Duration description: The amount of time (in milliseconds) to allow streaming events to accumulate into a single "reduced" event. The process repeats indefinitely, or until an "ends when" condition is satisfied. ui: display_type: milliseconds-to-human-readable default: 30000 max_events: type: number minimum: 1 maximum: 100000 title: Max Events description: The maximum number of events that can be included in a time window (specified by Duration above). The reduce operation will stop once it has reached this number of events, regardless of whether the Duration seconds have elapsed. group_by: type: array minItems: 1 title: Group By Field Path items: type: string title: Field description: The dot-notated property path errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ description: Before reducing, group events based on matching data from each of these field paths. Supports nesting via dot-notation. date_formats: type: array title: Date Formats description: Describes which root-level properties are dates, and their expected format. Dot-notation is supported, but nested field lookup paths will be an error. minItems: 1 items: type: object additionalProperties: false properties: field: type: string title: Date Field Path description: Specifies a root-level path property that contains a date value. maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ format: type: string title: Date Format description: The template describing the date format format: strftime_datetime errorMessage: must be a valid strftime datetime format maxLength: 200 required: - field - format merge_strategies: type: array title: Merge Strategy per Field minItems: 1 description: Specify merge strategies for individual root-level properties. Dot-notation is supported, but nested field lookup paths will be an error. items: type: object additionalProperties: false properties: field: type: string title: Field Path description: This is a root-level path property to apply a merge strategy to its value maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ strategy: oneOf: - label: Array description: Append each value onto an array enum: - array - label: Smallest array description: Keep the smallest array within each reduce time window enum: - shortest_array - label: Largest array description: Keep the largest array within each reduce time window enum: - longest_array - label: Unique Array description: All unique values go into a flattened array enum: - flat_unique - label: Join with space description: Concatenate into a single space-delimited string enum: - concat - label: Join with newline description: Concatenate into a single newline-delimited string enum: - concat_newline - label: Join description: Concatenate into a single string without using a delimiter enum: - concat_raw - label: Keep first description: Keep only the first value and discard the rest enum: - discard - label: Keep last description: Keep only the last non-null value enum: - retain - label: Min value description: Keep the minimum numeric value within each reduce time window enum: - min - label: Max value description: Keep the maximum numeric value within each reduce time window enum: - max - label: Sum description: Add numeric values together enum: - sum ui: display_type: list errorMessage: must be one of the allowed merge strategies title: Merge Strategy description: The merge strategy to be used for the specified property required: - field - strategy flush_condition: title: Event Flush Condition description: Force accumulated event reduction to flush the result when a conditional expression evaluates to true on an inbound event. allOf: - type: object properties: when: type: string enum: - none - starts_when - ends_when default: none ui: display_type: list errorMessage: "" title: Flush When required: - when - if: properties: when: not: enum: - none then: properties: conditional: allOf: - $ref: "#condition-group" ui: display_type: rules-engine required: - conditional - if: properties: type: enum: - map-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Map Fields description: Maps data from one field to another, either by moving or copying. additionalProperties: false properties: mappings: type: array title: Mappings description: A list of field mappings. Mappings are applied in the order they are defined. minItems: 1 ui: display_type: sortable items: type: object properties: source_field: type: string title: Source field description: The field to copy data from pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: type: string title: Target field description: The field to copy data into pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference drop_source: type: boolean title: Drop source? description: When enabled, the source field is dropped after the data is copied to the target field. Otherwise, it is preserved. default: false overwrite_target: type: boolean title: Overwrite target? description: When enabled, any existing data in the target field is overwritten. Otherwise, the target field will be preserved and this mapping will have no effect. default: false required: - source_field - target_field - drop_source - overwrite_target if: properties: drop_source: enum: - true required: - drop_source then: properties: source_field: type: string title: Source field description: The field to copy data from pattern: ^(?:metadata|\.|(?:metadata|message)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields). Note that dropping an event's message or timestamp is not allowed required: - source_field required: - mappings - if: properties: type: enum: - metrics-tag-cardinality-limit deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Tag Cardinality Limit (Metrics) description: Limits the cardinality of metric events by either dropping events or tags that exceed a specified value limit additionalProperties: false allOf: - if: title: Mode properties: mode: type: string enum: - exact then: title: Value Limit properties: value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags minimum: 1 maximum: 500 required: - value_limit - if: title: Mode properties: mode: type: string enum: - probabilistic then: title: Value Limit properties: value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags minimum: 1 maximum: 5000 required: - value_limit properties: tags: type: array title: Tags description: A list of tags to apply cardinality limits. If none are provided, all tags will be considered. items: type: string title: Tag minLength: 1 maxLength: 100 minItems: 0 maxItems: 10 exclude_tags: type: array title: Exclude Tags description: A list of tags to explicitly exclude from cardinality limits. items: type: string title: Tag minLength: 1 maxLength: 100 minItems: 0 maxItems: 10 action: type: string title: Action description: The action to take when a tag's cardinality exceeds the value limit enum: - drop_tag - drop_event default: drop_event value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags mode: type: string title: Mode description: The method to used to reduce tag value cardinality enum: - exact - probabilistic default: exact - if: properties: type: enum: - aggregate deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Aggregate description: Aggregates multiple events into a single event using either a tumbling interval window or a sliding interval window. additionalProperties: false properties: group_by: type: array minItems: 1 maxItems: 100 title: Group By Field Path description: When aggregating, group events based on matching values from each of these field paths. Supports nesting via dot-notation. items: type: string title: Field description: The dot-notated property path errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ ui: default_hint: - .name - .namespace - .tags evaluate: additionalProperties: type: string title: Evaluate description: How events should be aggregated within an interval. allOf: - if: properties: operation: enum: - CUSTOM then: properties: script: type: string title: Script ui: display_type: code-editor options: language: js description: The JavaScript representation of an event aggregation. minLength: 1 maxLength: 2048 default: > // Perform custom aggregation of the current event into the accumulated event. // The accumulated event is persisted across the entire aggregation window. // The function must return the accumulated event, which will be emitted when // the aggregation window elapses. function aggregateEvent(accum, event, metadata, timestamp, annotations) { return accum } required: - script properties: operation: title: Operation oneOf: - label: add description: Adds new values together if they are of the same type. For counters and gauges the values are added together. For sets, this is a union of the elements. For distribution, the samples are concatenated. For histograms, values in the same bucket will be added together. enum: - DEFAULT - label: sum description: Values are added together. Works for the GAUGE and COUNT metric types. enum: - SUM - label: minimum description: Get the minimum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MIN - label: maximum description: Get the maximum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MAX - label: average description: Values are averaged together. Works for the GAUGE and COUNT metric types. enum: - AVG - label: set intersection description: The aggregate produces a set of unique values. enum: - SET_INTERSECTION - label: distribution concatenation description: The aggregate produces a list of values combined together. enum: - DIST_CONCAT - label: custom description: Use a custom script to perform aggregation. enum: - CUSTOM ui: display_type: list required: - operation window: default: {} additionalProperties: type: integer title: Window Type description: Sliding windows can overlap, whereas tumbling windows are disjoint. For example, a tumbling window has a fixed time span and any events that fall within the "interval" will be used in the aggregate. Whereas in a sliding window, the aggregation occurs every "window duration" seconds after an event is encountered. allOf: - if: properties: type: enum: - sliding required: - type then: properties: window_min: type: integer title: Slide Duration (seconds) description: The amount of time to wait before creating a new sliding window for future events. minimum: 1 maximum: 200 default: 5 required: - window_min properties: type: type: string title: Type enum: - tumbling - sliding default: tumbling interval: type: integer title: Interval (seconds) description: The interval over which events are aggregated in seconds. default: 5 minimum: 1 maximum: 90000 required: - type - interval event_timestamp: type: string title: Event Timestamp description: The path to a field on the event that contains a epoch timestamp value. If an event does not have a timestamp field, events will be associated to the wall clock value when the event is processed. default: timestamp maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ conditional: ui: display_type: rules-engine title: Event Flush Condition description: Force accumulated event reduction to flush the result when a conditional expression evaluates to true on an inbound event. allOf: - $ref: "#condition-group" required: - evaluate - window - if: properties: type: enum: - event-to-metric deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Event to Metric description: Allows conversion between arbitrary events and a Metric. additionalProperties: false properties: metric_name: type: string title: Metric Name description: The machine name of the metric to emit. minLength: 1 maxLength: 128 pattern: ^[a-zA-Z][a-zA-Z0-9_:]*$ errorMessage: must be a valid metric name metric_kind: title: Kind description: The kind of metric to emit, Absolute or Incremental. Absolute metrics represent a complete value, and will generally replace an existing value for the metric in the target destination. Incremental metrics represent an additive value which is aggregated in the target destination to produce a new value. oneOf: - label: Incremental enum: - incremental - label: Absolute enum: - absolute default: incremental ui: display_type: radio-group metric_type: title: Type oneOf: - label: Counter enum: - counter - label: Sum enum: - sum - label: Gauge enum: - gauge default: counter ui: display_type: list value: title: Value description: The value of this metric. additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value then: title: Value properties: value: type: number title: Value description: The value. required: - value properties: value_type: title: Value type description: Either a value from an event field, or a new value input. oneOf: - label: Value from event field enum: - field - label: New value enum: - value default: field ui: display_type: radio-group value: type: - number - string required: - value_type namespace: title: Namespace description: The namespace for this metric. additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 128 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit the namespace. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: type: - string - "null" required: - value_type tags: type: array maxItems: 10 title: Tags description: A set of tags (also called labels) to apply to the metric event. items: type: object title: Tag description: A tag to apply to the metric event. allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value of this tag. Note that fields with highly-variable values will result in high-cardinality metrics, which may impact storage or cost in downstream destinations. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference fallback: type: string title: Fallback value description: The value to use for this tag when the field value is missing. minLength: 1 maxLength: 512 required: - value - if: properties: value_type: enum: - value then: title: Value properties: value: type: string title: Value description: The value for this tag. minLength: 1 maxLength: 128 required: - value properties: name: type: string title: Name description: The tag name pattern: ^[a-zA-Z][a-zA-Z0-9_]*$ minLength: 1 maxLength: 128 errorMessage: must be a valid metric tag name value_type: title: Value type description: Either a value from an event field, or a new value input. oneOf: - label: Value from event field enum: - field - label: New value enum: - value default: field ui: display_type: radio-group value: type: string required: - name - value_type - value additionalProperties: type: string required: - metric_name - metric_kind - metric_type - value - namespace - if: properties: type: enum: - clustering deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Clustering description: Finds clusters of similar values additionalProperties: false properties: cluster_field: type: string title: Field description: Field to cluster values on pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference default: .message similarity_threshold: type: number title: Similarity threshold description: A threshold from 0.0 (not similar) to 1.0 (exactly similar) that determines how close two values need to be to be consider as being part of the same cluster default: 0.4 minimum: 0 maximum: 1 max_clusters: type: integer title: Maximum cluster description: Maximum number of clusters kept cached by the transform default: 1000 minimum: 1 maximum: 2000 max_node_depth: type: integer title: Maximum tree depth description: Maximum depth of the prefix tree default: 4 minimum: 2 maximum: 8 max_children: type: integer title: Maximum children default: 100 minimum: 2 maximum: 200 description: Maximum number of children per node of the prefix tree required: - cluster_field - if: properties: type: enum: - redact deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Redact description: Detect and anonymize sensitive data in logs and metrics additionalProperties: false properties: field: type: string title: Field description: Detect patterns in the specified field. If the field is empty, the processor will look for patterns across all the fields. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 256 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference mask_patterns: type: array title: Redact Patterns description: List of PII patterns to detect and mask minItems: 1 maxItems: 20 items: type: object additionalProperties: type: string title: Redact Pattern description: A PII pattern to detect and/or anonymize allOf: - if: properties: action: enum: - Replace required: - action then: properties: replacement: type: string title: Replacement description: "The text to replace the detected pattern. Example: 000-00-0000" minLength: 1 maxLength: 512 required: - replacement - if: properties: action: enum: - Hash required: - action then: properties: algorithm: title: Algorithm description: Hash and encode (base16 or hex) matched PII data using the specified algorithm oneOf: - label: sha2 (sha-512_256) enum: - sha2 - label: sha3 (sha3-512) enum: - sha3 ui: display_type: list required: - algorithm - if: properties: name: enum: - custom required: - name then: properties: custom_options: allOf: - type: object additionalProperties: false properties: pattern: type: string title: Pattern description: The custom pattern minLength: 1 maxLength: 4096 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching default: true multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively default: false match_newline: type: boolean title: Dot all mode description: Dot matches newline character default: false crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" default: false ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines default: false required: - pattern title: Replace Expression description: The regular expression used to find replacements. Named capture groups are supported, e.g. `(?P<key>\w+):(?P<value>.+)`. ui: display_type: regex-pattern required: - custom_options properties: name: title: Pattern description: The PII pattern to match oneOf: - label: Credit card number enum: - credit_card - label: Email address enum: - email_address - label: IPv4 address enum: - ipv4_address - label: US or Canada phone number enum: - us_canada_phone_number - label: US social security number enum: - us_social_security_number - label: Custom pattern ui: feature_flag: pipeline-parser-regex enum: - custom ui: display_type: list action: type: string title: Action description: Action to take on the detected PII data enum: - Hash - Replace - Detect Only default: Replace required: - name - action required: - mask_patterns - if: properties: type: enum: - throttle deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Throttle description: Throttle (rate-limit) events passing through this component additionalProperties: false properties: threshold: type: number minimum: 1 maximum: 10000000 title: Limit description: The number of events to allow over the time window. If a Key Field is specified, the limit will be applied to events for each unique value of that field. default: 1 window_ms: type: number minimum: 1000 maximum: 2592000000 title: Window description: The time window over which the configured limit is applied ui: display_type: milliseconds-to-human-readable default: 5000 key_field: type: string title: Key Field description: When configured, the value from this field will be used to group events into separate buckets for throttling. If left blank, or if the event does not contain a value in this field, the event will be throttled as part of the default bucket. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference exclude: allOf: - $ref: "#condition-group" title: Exclude description: When configured, events matching this condition will not be throttled. ui: display_type: rules-engine required: - threshold - window_ms - if: properties: type: enum: - set-timestamp deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Set Timestamp description: Parse a list of fields using a chosen time format. First match will override the default timestamp of the event. additionalProperties: false properties: parsers: type: array title: List of Fields and Parsers description: The list of fields and parsers to use in order of priority, short-circuiting on the first successful match. ui: display_type: multi-output-map options: sortable: false items: type: object properties: field: type: string title: Field description: The field whose value will be parsed by the chosen date pattern pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference options: $ref: "#timestamp-parser-options" required: - field - options minItems: 1 required: - parsers - if: properties: type: enum: - data-profiler deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Data Profiler description: Profile the data sent through your pipeline. The system will generate annotations such as event type and total bytes to be used downstream. It will also normalize the app/host/level/line fields before sending the event downstream. A profile of the data including total, events, event size, log composition, and message templates will be generated as well. additionalProperties: false properties: app_fields: type: array title: App Path(s) description: A list of paths to look for the app value minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: App description: The path to the app maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .app - .application - .container host_fields: type: array title: Host Path(s) description: A list of paths to look for the host value minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Host description: The path to the host maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .host - .hostname level_fields: type: array title: Level Path(s) description: A list of paths to look for the log level minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Level description: The path to the level maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .level - .log_level line_fields: type: array title: Line Path(s) description: A list of paths to look for the event line minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Line description: The path to the line maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .line - .message label_fields: type: array title: Label Path(s) description: A list of paths to look for the event labels maxItems: 10 uniqueItems: true items: type: string title: Label description: The path to the labels maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .labels field_summary: type: boolean default: false title: Field Summary description: If enabled the profiler will attempt to summarize the values in fields of events passing through the node. ui: display_type: hidden ff_default_value: pipeline-node-transform-field-summary required: - app_fields - host_fields - level_fields - line_fields - if: properties: type: enum: - replace deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Replace description: Replace parts of a string using a regular expression or static string additionalProperties: type: object allOf: - if: properties: replacement_type: enum: - regex required: - replacement_type then: properties: options: allOf: - type: object additionalProperties: false properties: pattern: type: string minLength: 1 maxLength: 4096 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching default: true multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively default: false match_newline: type: boolean title: Dot all mode description: Dot matches newline character default: false crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" default: false ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines default: false required: - pattern title: Replace Expression description: The regular expression used to find replacements. Named capture groups are supported, e.g. `(?P<key>\w+):(?P<value>.+)`. ui: display_type: regex-pattern replace_with: type: string minLength: 1 maxLength: 4096 title: Replacement String description: The replacement string can be static or optionally contain capture group references. Capture groups can be positional or named. For example, to switch the position of two strings in the source string `onetwo`, the pattern could be `(?Pone)(?P.+)`, and the replacement would be `$two$one`. To do the same thing with positional capture groups, a pattern of `(\w{3})(\w{3})`, and a replacement of `$2$1` would have the same result.

To render a literal dollar sign `$` in the replacement, escape it with `$$`. So, if you had a capture group named `price`, to prepend a dollar sign to the replacement, you would use `$$$price`. required: - options - replace_with - if: properties: replacement_type: enum: - string required: - replacement_type then: properties: string_pattern: type: string title: String Pattern minLength: 1 maxLength: 4096 description: A static, case-sensitive string, which will be searched and replaced. replace_with: type: string minLength: 1 maxLength: 4096 title: Replacement String description: The static string used in each replacement. required: - string_pattern - replace_with properties: source_field: type: string title: Source Field description: The field that contains a string for replacement to operate on. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference max_times: type: number title: Maximum Replacements description: The maximum number of replacements to be made. If not specified, all occurrences will be replaced. minimum: 1 replacement_type: title: Replacement Type description: Specifies how the source string will be searched for replacement. ui: display_type: list errorMessage: must be one of the available pattern types oneOf: - label: String description: Replace a specified string one or more times. enum: - string - label: Regular Expression description: Replace using a regular expression. enum: - regex required: - source_field - replacement_type - if: properties: type: enum: - trace-sampling deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Trace Sampling description: Performs sampling of traces per a given rate using the Trace ID as the unique identifier. If a trace is rate selected, all spans of the trace are forwarded. If the Trace ID cannot be found, the event is dropped. allOf: - if: properties: sample_type: enum: - head then: properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 unique traces will be forwarded and all others will be dropped default: 10 minimum: 2 maximum: 100000 parent_span_id_field: type: "null" conditionals: type: "null" required: - rate - if: properties: sample_type: enum: - tail then: properties: parent_span_id_field: type: string default: .parent_span_id title: Parent Span ID description: The field to use as the parent span identifier pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference conditionals: type: array minItems: 1 title: Conditionals description: A list of complete conditionals, including expressions and a sample rate to maintain items: type: object additionalProperties: false title: Conditional description: A list of conditions and a sample rate to maintain properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 unique traces will be forwarded and all others will be dropped. A `rate = 1` means keep all traces that match the condition. minimum: 1 maximum: 100000 default: 10 conditional: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. required: - rate - conditional required: - parent_span_id_field - conditionals properties: sample_type: type: string description: The type of trace sampling to apply. enum: - head title: Sample Type default: head trace_id_field: type: string default: .trace_id title: Trace ID description: The field to use as the identifier of a trace. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - sample_type - trace_id_field properties: type: type: string enum: - filter - sample - drop-fields - parse - parse-sequentially - route - stringify - compact-fields - flatten-fields - dedupe - encrypt-fields - decrypt-fields - js-script - unroll - vrl - reduce - map-fields - metrics-tag-cardinality-limit - aggregate - event-to-metric - clustering - redact - throttle - set-timestamp - data-profiler - replace - trace-sampling deploy_type: type: string enum: - saas - local default: saas title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description user_config: type: object inputs: type: array items: type: string description: A list of inputs for the given transform. Typically this is empty when creating a new transform via the UI, but is leveraged by the terraform provider when managing pipeline state. required: - type - user_config required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/transform-group/{pipeline_transform_group_id}: delete: tags: - Pipeline Nodes parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_transform_group_id required: true responses: "200": description: Default Response get: tags: - Pipeline Nodes parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_transform_group_id required: true responses: "200": description: Default Response put: tags: - Pipeline Nodes requestBody: content: application/json: schema: type: object properties: transform_group_revision_id: oneOf: - type: string format: uuid - enum: - latest inputs: type: array items: type: string description: A list of inputs for the given group. Typically this is empty when creating a new transform via the UI, but is leveraged by the terraform provider when managing pipeline state. required: - transform_group_revision_id required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_transform_group_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/transform-group: post: tags: - Pipeline Nodes requestBody: content: application/json: schema: type: object properties: transform_group_id: type: string format: uuid deploy_type: type: string enum: - saas - local default: saas inputs: type: array items: type: string description: A list of inputs for the given group. Typically this is empty when creating a new transform via the UI, but is leveraged by the terraform provider when managing pipeline state. required: - transform_group_id required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/with-components: post: summary: Initialize a pipeline with provided components. All sources, transforms, and sinks are linked. tags: - Pipeline Management description: Populates and publishes a new pipeline as specified. requestBody: content: application/json: schema: type: object additionalProperties: false properties: origin: type: string enum: - ui - system default: system pipeline: oneOf: - type: "null" - type: object properties: id: type: string format: uuid title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name sources: type: array minItems: 1 items: type: object allOf: - if: properties: type: enum: - http deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: HTTP description: Receive data from incoming HTTP requests additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 decoding: oneOf: - enum: - bytes - enum: - json - enum: - ndjson - enum: - auto ui: display_type: list errorMessage: must be equal to one of the allowed values description: The decoding method for converting frames into data events. title: Decoding Method default: auto required: - decoding - if: properties: type: enum: - splunk-hec deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Splunk HEC description: Receive Splunk logs additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - s3 deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS S3 description: Collect data from object contents in AWS S3 additionalProperties: false properties: sqs_queue_url: type: string title: SQS Queue URL description: The URL of a AWS SQS queue configured to receive S3 bucket notifications for the desired S3 buckets format: uri errorMessage: format: Must be a valid URI auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key compression: type: string title: Compression description: The compression format of the S3 objects enum: - auto - gzip - none - zstd default: auto region: type: string title: Region minLength: 1 description: The name of the source AWS region. required: - sqs_queue_url - auth - compression - region - if: properties: type: enum: - sqs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS SQS description: Collect messages from AWS SQS additionalProperties: false properties: queue_url: type: string title: Queue URL description: The URL of an AWS SQS queue format: uri errorMessage: format: Must be a valid URI maxLength: 128 auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id maxLength: 128 secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret maxLength: 128 required: - access_key_id - secret_access_key region: type: string title: Region minLength: 1 description: The name of the source AWS region. maxLength: 128 required: - queue_url - auth - region - if: properties: type: enum: - demo-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Demo Logs description: Generate log messages for testing and demonstration. additionalProperties: type: array title: Log Lines description: A list of user-provided log lines that will be sent through the pipeline infinitely. A maximum of 25 lines is allowed. The lines can be either plain text or ndjson. If ndjson is detected, the encoding for the source will be set to JSON, and lines that are not valid will show up as a visible component error. Otherwise, text encoding will be used. minItems: 1 maxItems: 25 items: type: string minLength: 1 maxLength: 4096 title: Line description: A single line of log data, either stringified JSON or plain text. ui: display_type: user-provided-lines allOf: - if: properties: format: enum: - user_provided required: - format then: properties: lines: type: array title: Log Lines description: A list of user-provided log lines that will be sent through the pipeline infinitely. A maximum of 25 lines is allowed. The lines can be either plain text or ndjson. If ndjson is detected, the encoding for the source will be set to JSON, and lines that are not valid will show up as a visible component error. Otherwise, text encoding will be used. minItems: 1 maxItems: 25 items: type: string minLength: 1 maxLength: 4096 title: Line description: A single line of log data, either stringified JSON or plain text. ui: display_type: user-provided-lines required: - lines - allOf: - errorMessage: "`lines` is only valid when `format` is `user_provided`" if: not: type: object properties: format: enum: - user_provided required: - format then: not: type: object properties: lines: type: array title: Log Lines description: A list of user-provided log lines that will be sent through the pipeline infinitely. A maximum of 25 lines is allowed. The lines can be either plain text or ndjson. If ndjson is detected, the encoding for the source will be set to JSON, and lines that are not valid will show up as a visible component error. Otherwise, text encoding will be used. minItems: 1 maxItems: 25 items: type: string minLength: 1 maxLength: 4096 title: Line description: A single line of log data, either stringified JSON or plain text. ui: display_type: user-provided-lines required: - lines properties: interval_ms: type: number title: Interval description: The amount of time to pause between each demo log. Fractions of a second are accepted to increase throughput. e.g. 500 ms == 120 lines per minute default: 1000 minimum: 500 maximum: 3600000 ui: display_type: milliseconds-to-human-readable format: title: Format description: The format of the log messages. All of the data is generated. ui: display_type: list errorMessage: must be one of the available types of demo data oneOf: - label: HTTP JSON description: HTTP request data in JSON format. enum: - json - label: Sensor JSON description: Data from environment sensors in JSON format. enum: - env_sensor - label: Financial JSON description: Financial data in JSON format. enum: - financial - label: Nginx Requests description: Nginx logs in text format. enum: - nginx - label: Apache Requests description: Apache request logs in text format. enum: - apache_common - label: Apache Errors description: Apache error logs in text format. enum: - apache_error - label: BSD Syslog description: Mock BSD syslog entries in rfc3164 text format. enum: - bsd_syslog - label: Syslog description: Mock syslog entries in rfc5424 text format. enum: - syslog - label: HTTP Metrics description: HTTP-based metrics in the Mezmo metrics JSON format. enum: - http_metrics - label: Generic Metrics description: Generic metrics in the Mezmo metrics JSON format enum: - generic_metrics - label: User-Provided Logs description: Custom logs provided by the user. These can either be plain text or serialized JSON. enum: - user_provided - label: Infrastructure Logs description: Infrastructure logs consisting of sysdig, nginx, HDFS and spark logs enum: - infrastructure - label: Kubernetes Logs description: Kubernetes logs in JSON format enum: - kubernetes default: json required: - format - if: properties: type: enum: - kinesis-firehose deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Kinesis Firehose description: Receive Kinesis Firehose data additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 format: type: string enum: - json - text default: json title: Data Format description: This specifies what the data format will be after it is base64 decoded. If it is JSON, it will be automatically parsed. required: - format - if: properties: type: enum: - fluent deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Fluent (Fluentd, Fluent Bit) description: Receive data from Fluentd or Fluent Bit additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 decoding: type: string description: The decoding method for converting frames into data events. enum: - bytes - json - ndjson title: Decoding Method default: json required: - decoding - if: properties: type: enum: - log-analysis deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Log Analysis description: Receive data directly from your Mezmo Log Analysis account additionalProperties: false - if: properties: type: enum: - log-analysis-ingestion deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Log Analysis Ingestion description: Receive unparsed source data directly from your Mezmo Log Analysis account. additionalProperties: type: string - if: properties: type: enum: - logstash deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Logstash description: Receive Logstash data additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 format: type: string enum: - json - text default: json title: Data Format description: The format of the logstash data required: - format - if: properties: type: enum: - mezmo-agent deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Agent description: Receive logs from the Mezmo Agent additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - mezmo-datadog-source deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Datadog Agent description: Send logs and metrics data directly from an installed datadog agent additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - pipeline-data-restoration deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Pipeline Data Restoration description: Receive data directly from your Mezmo Pipline Data Restoration additionalProperties: false - if: properties: type: enum: - webhook deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Webhook (WebSub) description: Receive data from incoming webhooks using the WebSub protocol. additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - kafka deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Kafka Source description: Collect events from Kafka compatible brokers additionalProperties: type: string properties: brokers: type: array title: Brokers uniqueItems: true minItems: 1 description: An array of key/val objects for each host/port of a Kafka broker. items: type: object title: Host/Port description: The host and port for the Kafka broker. additionalProperties: false properties: host: title: Host description: The hostname or IP address of the broker. anyOf: - type: string format: ipv4 errorMessage: format: Must be a valid IPv4 address - type: string format: hostname maxLength: 512 errorMessage: format: Must be a valid hostname port: type: number minimum: 1 maximum: 65535 title: Port description: The port that the broker listens on. required: - host - port topics: type: array title: Topics items: type: string title: Topic minLength: 1 maxLength: 256 description: The Kafka topic name. minItems: 1 description: List of Kafka topics names to read events from. group_id: type: string title: Consumer Group minLength: 1 description: The consumer group name to be used to consume events from Kafka. tls_enabled: type: boolean title: TLS enabled description: Whether or not to require TLS for outgoing connections. default: true sasl_enabled: type: boolean title: SASL/SCRAM Enabled description: Enable SASL/SCRAM authentication with Kafka default: true decoding_codec: type: string enum: - bytes - json title: Decoding description: Configures how events are decoded from raw bytes default: bytes if: properties: sasl_enabled: enum: - true then: properties: sasl_mechanism: type: string enum: - SCRAM-SHA-512 - SCRAM-SHA-256 - PLAIN title: SASL description: The Kafka SASL/SCRAM mechanism default: PLAIN sasl_username: type: string title: Username description: The SASL/SCRAM username minLength: 1 sasl_password: type: string title: Password description: The SASL/SCRAM username ui: display_type: secret minLength: 1 required: - sasl_mechanism - sasl_username - sasl_password required: - brokers - topics - group_id - decoding_codec - if: properties: type: enum: - azure-event-hub deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Azure Event Hub Source description: Collect events from Azure Event Hub using Azure Kafka Endpoint additionalProperties: false properties: connection_string: type: string title: Connection String minLength: 1 maxLength: 512 description: The Connection String as it appears in hub consumer SAS Policy ui: display_type: secret namespace: type: string title: Namespace minLength: 1 maxLength: 256 description: The Event Hub Namespace topics: type: array title: Event Hub Names items: type: string title: Name minLength: 1 maxLength: 256 description: Azure Event Hub instance name. minItems: 1 description: The list of Azure Event Hub name(s) to read events from. group_id: type: string title: Consumer Group minLength: 1 maxLength: 256 description: The consumer group name that this consumer belongs to. decoding_codec: type: string enum: - bytes - json title: Decoding description: Configures how events are decoded from raw bytes default: bytes required: - connection_string - namespace - topics - group_id - decoding_codec - if: properties: type: enum: - prometheus-remote-write deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Prometheus Remote Write description: Receive prometheus remote write protobuf data from HTTP requests additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - open-telemetry-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry Metrics description: Receive metrics from OpenTelemetry over HTTP additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - open-telemetry-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry Logs description: Receive logs from OpenTelemetry over HTTP additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - open-telemetry-traces deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry Traces description: Receive traces from OpenTelemetry over HTTP additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 properties: title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description type: type: string enum: - http - splunk-hec - s3 - sqs - demo-logs - kinesis-firehose - fluent - log-analysis - log-analysis-ingestion - logstash - mezmo-agent - mezmo-datadog-source - pipeline-data-restoration - webhook - kafka - azure-event-hub - prometheus-remote-write - open-telemetry-metrics - open-telemetry-logs - open-telemetry-traces user_config: type: object required: - user_config transforms: type: array minItems: 1 items: type: object allOf: - if: properties: type: enum: - filter deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Filter description: Define condition(s) to include or exclude events from the pipeline additionalProperties: false properties: action: title: Action description: How to handle events matching this criteria oneOf: - label: Allow events matching this criteria enum: - allow - label: Drop events matching this criteria enum: - drop default: allow ui: display_type: radio-group conditional: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine required: - action - conditional - if: properties: type: enum: - sample deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sample description: Sample data at a given rate, retaining only a subset of data events for further processing. additionalProperties: false properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 events will be forwarded and the rest will be dropped minimum: 2 maximum: 10000 default: 10 always_include: title: Always Include description: Certain events may always want to be included (not sampled), regardless of the sample rate. Events matching this criteria will always show up in the results. additionalProperties: type: - string - number dependencies: field: - str_operator str_operator: - field allOf: - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: errorMessage: pattern: Value must be a number type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 pattern: ^\s*-?(\d*\.)?\d+\s*$ required: - value - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix - percent_change_greater - percent_change_greater_or_equal - percent_change_less - percent_change_less_or_equal - value_change_greater - value_change_greater_or_equal - value_change_less - value_change_less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 required: - value - if: title: Operator properties: str_operator: type: string enum: - contains - equal - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix ui: display_type: list-operators title: Operator required: - str_operator then: title: Case sensitive properties: case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive comparison? default: true properties: field: type: string title: Field description: The JSON field to use in a condition to always include in sampling pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - is_metric - is_array - is_boolean - is_empty - is_null - is_number - is_object - is_string - exists - ends_with - starts_with - contains_token_term - contains_token_prefix - regex_match ui: display_type: list-operators title: Operator description: The comparison operator negate: type: boolean default: false title: Negate description: Negate the comparison expression required: - rate - if: properties: type: enum: - drop-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Remove Fields description: Remove fields additionalProperties: false properties: fields: type: array title: Fields description: A list of fields to be removed items: type: string title: Field pattern: ^(?:metadata|\.|(?:metadata|message)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields). Note that dropping an event's message or timestamp is not allowed minItems: 1 required: - fields - if: properties: type: enum: - parse deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Parse description: Parse a specified field using the chosen parser additionalProperties: type: object errorMessage: properties: target_field: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) allOf: - if: properties: parser: enum: - parse_key_value then: properties: options: $ref: "#key-value-parser-options" - if: properties: parser: enum: - parse_timestamp then: properties: options: $ref: "#timestamp-parser-options" required: - options - if: properties: parser: enum: - parse_grok then: properties: options: $ref: "#grok-pattern-parser-options" required: - options - if: properties: parser: enum: - parse_regex then: properties: options: allOf: - $ref: "#regex-parser-options" ui: display_type: regex-pattern required: - options - if: properties: parser: enum: - parse_apache_log then: properties: options: $ref: "#apache-parser-options" required: - options - if: properties: parser: enum: - parse_nginx_log then: properties: options: $ref: "#nginx-parser-options" required: - options - if: properties: parser: enum: - parse_cef then: properties: options: $ref: "#cef-parser-options" - if: properties: parser: enum: - parse_csv then: properties: options: $ref: "#csv-parser-options" properties: field: type: string title: Field description: The JSON field whose value should be parsed. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: oneOf: - type: "null" - type: string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference title: Target field description: The field into which the parsed value should be inserted. Leave blank to insert the parsed data into the original field. default: null parser: oneOf: - label: AWS Cloudfront Access Log group: Line Parsers ui: {} enum: - aws_cloudfront_access_log - label: AWS Cloudwatch Log group: Line Parsers ui: {} enum: - parse_aws_cloudwatch_log_subscription_message - label: AWS Load Balancer Log group: Line Parsers ui: {} enum: - parse_aws_alb_log - label: AWS S3 Access Log group: Line Parsers ui: {} enum: - aws_s3_access_log - label: Apache Log group: Line Parsers ui: {} enum: - parse_apache_log - label: CSV group: Line Parsers ui: {} enum: - parse_csv - label: Common Event Format (CEF) Log group: Line Parsers ui: {} enum: - parse_cef - label: Common Log group: Line Parsers ui: {} enum: - parse_common_log - label: Elasticsearch Log group: Line Parsers ui: {} enum: - elasticsearch_log - label: Golang Log group: Line Parsers ui: {} enum: - golang_log - label: Google Log group: Line Parsers ui: {} enum: - parse_glog - label: Grok Pattern group: Line Parsers ui: {} enum: - parse_grok - label: Integer group: Field Parsers ui: {} enum: - parse_int - label: JSON group: Line Parsers ui: {} enum: - parse_json - label: Key/Value group: Line Parsers ui: {} enum: - parse_key_value - label: Kubernetes Log group: Line Parsers ui: {} enum: - parse_klog - label: Linux Authorization group: Line Parsers ui: {} enum: - parse_linux_authorization - label: Mongo Log group: Line Parsers ui: {} enum: - mongo_log - label: Mongo3 Log group: Line Parsers ui: {} enum: - mongo3_log - label: Nginx Log group: Line Parsers ui: {} enum: - parse_nginx_log - label: PSQL Log group: Line Parsers ui: {} enum: - psql_log - label: Query String group: Field Parsers ui: {} enum: - parse_query_string - label: Regular Expression group: Line Parsers ui: feature_flag: pipeline-parser-regex enum: - parse_regex - label: Syslog group: Line Parsers ui: {} enum: - parse_syslog - label: Timestamp group: Field Parsers ui: {} enum: - parse_timestamp - label: Tokens group: Field Parsers ui: {} enum: - parse_tokens - label: URL group: Field Parsers ui: {} enum: - parse_url - label: User Agent group: Field Parsers ui: {} enum: - parse_user_agent - label: VPC Flow Log group: Line Parsers ui: {} description: Parse VPC logs using default format for fields from v2 to v4 enum: - parse_aws_vpc_flow_log ui: display_type: list errorMessage: must be one of allowed parsers title: Parser description: The kind of parser to use against the input value from "field". required: - field - parser - if: properties: type: enum: - parse-sequentially deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Parse Sequentially description: Parse fields using one or more parsers in order, sending out the event on the first successful match. additionalProperties: false errorMessage: properties: target_field: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) properties: field: type: string title: Field description: The field whose value should be parsed. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: oneOf: - type: "null" - type: string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference title: Target field description: The field into which the parsed value should be inserted. Leave blank to insert the parsed data into the original field. default: null parsers: type: array title: List of parsers description: The list of parsers to use in order against the input valuefrom "field", short-circuiting on the first successful match. items: type: object allOf: - if: properties: parser: enum: - parse_key_value then: properties: options: $ref: "#key-value-parser-options" - if: properties: parser: enum: - parse_timestamp then: properties: options: $ref: "#timestamp-parser-options" required: - options - if: properties: parser: enum: - parse_grok then: properties: options: $ref: "#grok-pattern-parser-options" required: - options - if: properties: parser: enum: - parse_regex then: properties: options: allOf: - $ref: "#regex-parser-options" ui: display_type: regex-pattern required: - options - if: properties: parser: enum: - parse_apache_log then: properties: options: $ref: "#apache-parser-options" required: - options - if: properties: parser: enum: - parse_nginx_log then: properties: options: $ref: "#nginx-parser-options" required: - options - if: properties: parser: enum: - parse_cef then: properties: options: $ref: "#cef-parser-options" - if: properties: parser: enum: - parse_csv then: properties: options: $ref: "#csv-parser-options" properties: label: oneOf: - type: "null" - type: string maxLength: 20 title: Title description: An arbitrary name you choose to identify the results of this processor for use when connecting them to other components. parser: oneOf: - label: AWS Cloudfront Access Log group: Line Parsers ui: {} enum: - aws_cloudfront_access_log - label: AWS Cloudwatch Log group: Line Parsers ui: {} enum: - parse_aws_cloudwatch_log_subscription_message - label: AWS Load Balancer Log group: Line Parsers ui: {} enum: - parse_aws_alb_log - label: AWS S3 Access Log group: Line Parsers ui: {} enum: - aws_s3_access_log - label: Apache Log group: Line Parsers ui: {} enum: - parse_apache_log - label: CSV group: Line Parsers ui: {} enum: - parse_csv - label: Common Event Format (CEF) Log group: Line Parsers ui: {} enum: - parse_cef - label: Common Log group: Line Parsers ui: {} enum: - parse_common_log - label: Elasticsearch Log group: Line Parsers ui: {} enum: - elasticsearch_log - label: Golang Log group: Line Parsers ui: {} enum: - golang_log - label: Google Log group: Line Parsers ui: {} enum: - parse_glog - label: Grok Pattern group: Line Parsers ui: {} enum: - parse_grok - label: Integer group: Field Parsers ui: {} enum: - parse_int - label: JSON group: Line Parsers ui: {} enum: - parse_json - label: Key/Value group: Line Parsers ui: {} enum: - parse_key_value - label: Kubernetes Log group: Line Parsers ui: {} enum: - parse_klog - label: Linux Authorization group: Line Parsers ui: {} enum: - parse_linux_authorization - label: Mongo Log group: Line Parsers ui: {} enum: - mongo_log - label: Mongo3 Log group: Line Parsers ui: {} enum: - mongo3_log - label: Nginx Log group: Line Parsers ui: {} enum: - parse_nginx_log - label: PSQL Log group: Line Parsers ui: {} enum: - psql_log - label: Query String group: Field Parsers ui: {} enum: - parse_query_string - label: Regular Expression group: Line Parsers ui: feature_flag: pipeline-parser-regex enum: - parse_regex - label: Syslog group: Line Parsers ui: {} enum: - parse_syslog - label: Timestamp group: Field Parsers ui: {} enum: - parse_timestamp - label: Tokens group: Field Parsers ui: {} enum: - parse_tokens - label: URL group: Field Parsers ui: {} enum: - parse_url - label: User Agent group: Field Parsers ui: {} enum: - parse_user_agent - label: VPC Flow Log group: Line Parsers ui: {} description: Parse VPC logs using default format for fields from v2 to v4 enum: - parse_aws_vpc_flow_log ui: display_type: list errorMessage: must be one of allowed parsers title: Parser description: The kind of parser to use against the input value from "field". _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. default: "" required: - parser - _output_name additionalProperties: type: object minItems: 1 ui: display_type: multi-output-map unique_parser_and_options: true errorMessage: uniqueItemProperties: parser can not be defined more than once required: - field - parsers - if: properties: type: enum: - route deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Route description: Route data based on whether it matches (or doesn't match) logical comparisons additionalProperties: false properties: conditionals: type: array minItems: 1 title: Conditionals description: A list of complete conditionals, including expressions and label items: type: object additionalProperties: false title: Conditional description: A list of conditions and a label properties: conditional: $ref: "#condition-group" label: type: string minLength: 1 maxLength: 255 title: Label description: An arbitrary name you choose to identify the results of this processor for use when connecting them to other components. _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. default: "" required: - conditional - label - _output_name required: - conditionals - if: properties: type: enum: - stringify deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Stringify description: Stringify messages as JSON additionalProperties: false - if: properties: type: enum: - compact-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Compact Fields description: Remove empty values from a list of fields additionalProperties: false properties: fields: type: array title: Fields description: A list of fields to remove empty values from items: type: string title: Field description: Field to remove empty value from pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference minItems: 1 options: type: object additionalProperties: false title: Compact Options description: Additional options for removing empty values from fields properties: compact_array: type: boolean title: Compact Array description: Remove empty arrays from a field default: true compact_object: type: boolean title: Compact Object description: Remove empty objects from a field default: true required: - fields - if: properties: type: enum: - flatten-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Flatten Fields description: Flattens the object or array value of a field into a single level representation additionalProperties: false properties: fields: type: array title: Fields description: Define specific nested objects you wish to flatten. This will not flatten the entire object, just the specified fields. Leave blank to flatten the entire object items: type: string title: Field description: Field to flatten pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference options: type: object additionalProperties: false title: Flatten Options description: Additional options for flattening fields properties: delimiter: type: string default: _ title: Delimiter description: When flattening, the nested field labels will be appended to the higher level field names along with a delimiter to allow understanding of the original structure. The default is an underscore (_) to ensure no collisions with the . notation used for field identifiers minLength: 1 maxLength: 1 - if: properties: type: enum: - dedupe deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Dedupe description: Dedupe (remove duplicates) from the data stream additionalProperties: false properties: number_of_events: type: number description: Number of events to compare across title: Number of Events minimum: 2 maximum: 5000 default: 5000 comparison_type: type: string enum: - Ignore - Match description: "Match: Only compares across the fields which are specified by the user.Ignore: Compares everything but the fields specified by the user." title: Comparison Type default: Match fields: type: array title: Fields description: A list of fields on which to base deduping items: type: string title: Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference minItems: 1 required: - number_of_events - comparison_type - fields - if: properties: type: enum: - encrypt-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Encrypt Field description: Encrypts the value of the provided field additionalProperties: false allOf: - if: properties: algorithm: enum: - AES-256-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key properties: field: type: string title: Field description: Field to encrypt. The value of the field must be a primitive (string, number, boolean) pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference algorithm: oneOf: - label: AES-256-CFB (key = 32 characters, iv = 16 characters) description: AES-256-CFB (key = 32 characters, iv = 16 characters) enum: - AES-256-CFB - label: AES-192-CFB (key = 24 characters, iv = 16 characters) description: AES-192-CFB (key = 24 characters, iv = 16 characters) enum: - AES-192-CFB - label: AES-128-CFB (key = 16 characters, iv = 16 characters) description: AES-128-CFB (key = 16 characters, iv = 16 characters) enum: - AES-128-CFB - label: AES-256-OFB (key = 32 characters, iv = 16 characters) description: AES-256-OFB (key = 32 characters, iv = 16 characters) enum: - AES-256-OFB - label: AES-192-OFB (key = 24 characters, iv = 16 characters) description: AES-192-OFB (key = 24 characters, iv = 16 characters) enum: - AES-192-OFB - label: AES-128-OFB (key = 16 characters, iv = 16 characters) description: AES-128-OFB (key = 16 characters, iv = 16 characters) enum: - AES-128-OFB - label: AES-256-CTR (key = 32 characters, iv = 16 characters) description: AES-256-CTR (key = 32 characters, iv = 16 characters) enum: - AES-256-CTR - label: AES-192-CTR (key = 24 characters, iv = 16 characters) description: AES-192-CTR (key = 24 characters, iv = 16 characters) enum: - AES-192-CTR - label: AES-128-CTR (key = 16 characters, iv = 16 characters) description: AES-128-CTR (key = 16 characters, iv = 16 characters) enum: - AES-128-CTR - label: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-PKCS7 - label: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-PKCS7 - label: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-PKCS7 - label: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ANSIX923 - label: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ANSIX923 - label: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ANSIX923 - label: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO7816 - label: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO7816 - label: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO7816 - label: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO10126 - label: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO10126 - label: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO10126 ui: display_type: list title: Encryption algorithm description: The encryption algorithm to use key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 32 iv_field: type: string title: Initialization vector (IV) field description: The field in which to store the generated initialization vector, IV. Each encrypted value will have a unique IV pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference encode_raw_bytes: type: boolean title: Encode encrypted field and IV as Base 64 text description: Encode the encrypted value and generated initialization vector as Base64 text default: true required: - field - algorithm - key - iv_field - if: properties: type: enum: - decrypt-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Decrypt Field additionalProperties: false description: Decrypts the value of the provided field allOf: - if: properties: algorithm: enum: - AES-256-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key properties: field: type: string title: Field description: Field to decrypt. The value of the field must be a string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference algorithm: oneOf: - label: AES-256-CFB (key = 32 characters, iv = 16 characters) description: AES-256-CFB (key = 32 characters, iv = 16 characters) enum: - AES-256-CFB - label: AES-192-CFB (key = 24 characters, iv = 16 characters) description: AES-192-CFB (key = 24 characters, iv = 16 characters) enum: - AES-192-CFB - label: AES-128-CFB (key = 16 characters, iv = 16 characters) description: AES-128-CFB (key = 16 characters, iv = 16 characters) enum: - AES-128-CFB - label: AES-256-OFB (key = 32 characters, iv = 16 characters) description: AES-256-OFB (key = 32 characters, iv = 16 characters) enum: - AES-256-OFB - label: AES-192-OFB (key = 24 characters, iv = 16 characters) description: AES-192-OFB (key = 24 characters, iv = 16 characters) enum: - AES-192-OFB - label: AES-128-OFB (key = 16 characters, iv = 16 characters) description: AES-128-OFB (key = 16 characters, iv = 16 characters) enum: - AES-128-OFB - label: AES-256-CTR (key = 32 characters, iv = 16 characters) description: AES-256-CTR (key = 32 characters, iv = 16 characters) enum: - AES-256-CTR - label: AES-192-CTR (key = 24 characters, iv = 16 characters) description: AES-192-CTR (key = 24 characters, iv = 16 characters) enum: - AES-192-CTR - label: AES-128-CTR (key = 16 characters, iv = 16 characters) description: AES-128-CTR (key = 16 characters, iv = 16 characters) enum: - AES-128-CTR - label: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-PKCS7 - label: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-PKCS7 - label: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-PKCS7 - label: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ANSIX923 - label: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ANSIX923 - label: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ANSIX923 - label: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO7816 - label: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO7816 - label: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO7816 - label: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO10126 - label: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO10126 - label: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO10126 ui: display_type: list title: Decryption algorithm description: The algorithm with which the data was encrypted key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 32 iv_field: type: string title: Initialization vector (IV) field description: The field from which to read the initialization vector, IV pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference decode_raw_bytes: type: boolean title: Decode the encrypted value and IV as Base 64 text description: Decode the encrypted value and initialization vector as Base64 text default: true required: - field - algorithm - key - iv_field - if: properties: type: enum: - js-script deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Script Execution description: Use JavaScript to reshape and transform your data. You can combine multiple actions like filtering, dropping, mapping, and casting inside of a single js script. additionalProperties: true properties: script: type: string title: Script ui: display_type: code-editor options: language: js description: The JavaScript representation of the event transformation. minLength: 1 maxLength: 10000 default: > // Modify the event using a subset of the JavaScript language. // The function must return the modified event function processEvent(message, metadata, timestamp, annotations) { return message } - if: properties: type: enum: - unroll deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Unroll description: Takes an array of events and emits them all as individual events. additionalProperties: false properties: field: type: string title: Field description: The JSON field name that contains an array of events pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference values_only: type: boolean title: Values only? description: When enabled, the values from the specified array field will be emitted as new events. Otherwise, the original event will be duplicated for each value in the array field, with the unrolled value present in the field specified. default: true required: - field - if: properties: type: enum: - vrl deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: VRL description: Modify your observability data using Vector Remap Language (VRL) additionalProperties: false properties: source: type: string title: VRL Source description: VRL code to execute for each event ui: display_type: code-editor options: language: vrl minLength: 1 default: .message = .message required: - source - if: properties: type: enum: - reduce deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Reduce description: Combine multiple events over time into one based on a set of criteria additionalProperties: false properties: duration_ms: type: number maximum: 7200000 title: Duration description: The amount of time (in milliseconds) to allow streaming events to accumulate into a single "reduced" event. The process repeats indefinitely, or until an "ends when" condition is satisfied. ui: display_type: milliseconds-to-human-readable default: 30000 max_events: type: number minimum: 1 maximum: 100000 title: Max Events description: The maximum number of events that can be included in a time window (specified by Duration above). The reduce operation will stop once it has reached this number of events, regardless of whether the Duration seconds have elapsed. group_by: type: array minItems: 1 title: Group By Field Path items: type: string title: Field description: The dot-notated property path errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ description: Before reducing, group events based on matching data from each of these field paths. Supports nesting via dot-notation. date_formats: type: array title: Date Formats description: Describes which root-level properties are dates, and their expected format. Dot-notation is supported, but nested field lookup paths will be an error. minItems: 1 items: type: object additionalProperties: false properties: field: type: string title: Date Field Path description: Specifies a root-level path property that contains a date value. maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ format: type: string title: Date Format description: The template describing the date format format: strftime_datetime errorMessage: must be a valid strftime datetime format maxLength: 200 required: - field - format merge_strategies: type: array title: Merge Strategy per Field minItems: 1 description: Specify merge strategies for individual root-level properties. Dot-notation is supported, but nested field lookup paths will be an error. items: type: object additionalProperties: false properties: field: type: string title: Field Path description: This is a root-level path property to apply a merge strategy to its value maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ strategy: oneOf: - label: Array description: Append each value onto an array enum: - array - label: Smallest array description: Keep the smallest array within each reduce time window enum: - shortest_array - label: Largest array description: Keep the largest array within each reduce time window enum: - longest_array - label: Unique Array description: All unique values go into a flattened array enum: - flat_unique - label: Join with space description: Concatenate into a single space-delimited string enum: - concat - label: Join with newline description: Concatenate into a single newline-delimited string enum: - concat_newline - label: Join description: Concatenate into a single string without using a delimiter enum: - concat_raw - label: Keep first description: Keep only the first value and discard the rest enum: - discard - label: Keep last description: Keep only the last non-null value enum: - retain - label: Min value description: Keep the minimum numeric value within each reduce time window enum: - min - label: Max value description: Keep the maximum numeric value within each reduce time window enum: - max - label: Sum description: Add numeric values together enum: - sum ui: display_type: list errorMessage: must be one of the allowed merge strategies title: Merge Strategy description: The merge strategy to be used for the specified property required: - field - strategy flush_condition: title: Event Flush Condition description: Force accumulated event reduction to flush the result when a conditional expression evaluates to true on an inbound event. allOf: - type: object properties: when: type: string enum: - none - starts_when - ends_when default: none ui: display_type: list errorMessage: "" title: Flush When required: - when - if: properties: when: not: enum: - none then: properties: conditional: allOf: - $ref: "#condition-group" ui: display_type: rules-engine required: - conditional - if: properties: type: enum: - map-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Map Fields description: Maps data from one field to another, either by moving or copying. additionalProperties: false properties: mappings: type: array title: Mappings description: A list of field mappings. Mappings are applied in the order they are defined. minItems: 1 ui: display_type: sortable items: type: object properties: source_field: type: string title: Source field description: The field to copy data from pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: type: string title: Target field description: The field to copy data into pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference drop_source: type: boolean title: Drop source? description: When enabled, the source field is dropped after the data is copied to the target field. Otherwise, it is preserved. default: false overwrite_target: type: boolean title: Overwrite target? description: When enabled, any existing data in the target field is overwritten. Otherwise, the target field will be preserved and this mapping will have no effect. default: false required: - source_field - target_field - drop_source - overwrite_target if: properties: drop_source: enum: - true required: - drop_source then: properties: source_field: type: string title: Source field description: The field to copy data from pattern: ^(?:metadata|\.|(?:metadata|message)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields). Note that dropping an event's message or timestamp is not allowed required: - source_field required: - mappings - if: properties: type: enum: - metrics-tag-cardinality-limit deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Tag Cardinality Limit (Metrics) description: Limits the cardinality of metric events by either dropping events or tags that exceed a specified value limit additionalProperties: false allOf: - if: title: Mode properties: mode: type: string enum: - exact then: title: Value Limit properties: value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags minimum: 1 maximum: 500 required: - value_limit - if: title: Mode properties: mode: type: string enum: - probabilistic then: title: Value Limit properties: value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags minimum: 1 maximum: 5000 required: - value_limit properties: tags: type: array title: Tags description: A list of tags to apply cardinality limits. If none are provided, all tags will be considered. items: type: string title: Tag minLength: 1 maxLength: 100 minItems: 0 maxItems: 10 exclude_tags: type: array title: Exclude Tags description: A list of tags to explicitly exclude from cardinality limits. items: type: string title: Tag minLength: 1 maxLength: 100 minItems: 0 maxItems: 10 action: type: string title: Action description: The action to take when a tag's cardinality exceeds the value limit enum: - drop_tag - drop_event default: drop_event value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags mode: type: string title: Mode description: The method to used to reduce tag value cardinality enum: - exact - probabilistic default: exact - if: properties: type: enum: - aggregate deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Aggregate description: Aggregates multiple events into a single event using either a tumbling interval window or a sliding interval window. additionalProperties: false properties: group_by: type: array minItems: 1 maxItems: 100 title: Group By Field Path description: When aggregating, group events based on matching values from each of these field paths. Supports nesting via dot-notation. items: type: string title: Field description: The dot-notated property path errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ ui: default_hint: - .name - .namespace - .tags evaluate: additionalProperties: type: string title: Evaluate description: How events should be aggregated within an interval. allOf: - if: properties: operation: enum: - CUSTOM then: properties: script: type: string title: Script ui: display_type: code-editor options: language: js description: The JavaScript representation of an event aggregation. minLength: 1 maxLength: 2048 default: > // Perform custom aggregation of the current event into the accumulated event. // The accumulated event is persisted across the entire aggregation window. // The function must return the accumulated event, which will be emitted when // the aggregation window elapses. function aggregateEvent(accum, event, metadata, timestamp, annotations) { return accum } required: - script properties: operation: title: Operation oneOf: - label: add description: Adds new values together if they are of the same type. For counters and gauges the values are added together. For sets, this is a union of the elements. For distribution, the samples are concatenated. For histograms, values in the same bucket will be added together. enum: - DEFAULT - label: sum description: Values are added together. Works for the GAUGE and COUNT metric types. enum: - SUM - label: minimum description: Get the minimum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MIN - label: maximum description: Get the maximum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MAX - label: average description: Values are averaged together. Works for the GAUGE and COUNT metric types. enum: - AVG - label: set intersection description: The aggregate produces a set of unique values. enum: - SET_INTERSECTION - label: distribution concatenation description: The aggregate produces a list of values combined together. enum: - DIST_CONCAT - label: custom description: Use a custom script to perform aggregation. enum: - CUSTOM ui: display_type: list required: - operation window: default: {} additionalProperties: type: integer title: Window Type description: Sliding windows can overlap, whereas tumbling windows are disjoint. For example, a tumbling window has a fixed time span and any events that fall within the "interval" will be used in the aggregate. Whereas in a sliding window, the aggregation occurs every "window duration" seconds after an event is encountered. allOf: - if: properties: type: enum: - sliding required: - type then: properties: window_min: type: integer title: Slide Duration (seconds) description: The amount of time to wait before creating a new sliding window for future events. minimum: 1 maximum: 200 default: 5 required: - window_min properties: type: type: string title: Type enum: - tumbling - sliding default: tumbling interval: type: integer title: Interval (seconds) description: The interval over which events are aggregated in seconds. default: 5 minimum: 1 maximum: 90000 required: - type - interval event_timestamp: type: string title: Event Timestamp description: The path to a field on the event that contains a epoch timestamp value. If an event does not have a timestamp field, events will be associated to the wall clock value when the event is processed. default: timestamp maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ conditional: ui: display_type: rules-engine title: Event Flush Condition description: Force accumulated event reduction to flush the result when a conditional expression evaluates to true on an inbound event. allOf: - $ref: "#condition-group" required: - evaluate - window - if: properties: type: enum: - event-to-metric deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Event to Metric description: Allows conversion between arbitrary events and a Metric. additionalProperties: false properties: metric_name: type: string title: Metric Name description: The machine name of the metric to emit. minLength: 1 maxLength: 128 pattern: ^[a-zA-Z][a-zA-Z0-9_:]*$ errorMessage: must be a valid metric name metric_kind: title: Kind description: The kind of metric to emit, Absolute or Incremental. Absolute metrics represent a complete value, and will generally replace an existing value for the metric in the target destination. Incremental metrics represent an additive value which is aggregated in the target destination to produce a new value. oneOf: - label: Incremental enum: - incremental - label: Absolute enum: - absolute default: incremental ui: display_type: radio-group metric_type: title: Type oneOf: - label: Counter enum: - counter - label: Sum enum: - sum - label: Gauge enum: - gauge default: counter ui: display_type: list value: title: Value description: The value of this metric. additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value then: title: Value properties: value: type: number title: Value description: The value. required: - value properties: value_type: title: Value type description: Either a value from an event field, or a new value input. oneOf: - label: Value from event field enum: - field - label: New value enum: - value default: field ui: display_type: radio-group value: type: - number - string required: - value_type namespace: title: Namespace description: The namespace for this metric. additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 128 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit the namespace. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: type: - string - "null" required: - value_type tags: type: array maxItems: 10 title: Tags description: A set of tags (also called labels) to apply to the metric event. items: type: object title: Tag description: A tag to apply to the metric event. allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value of this tag. Note that fields with highly-variable values will result in high-cardinality metrics, which may impact storage or cost in downstream destinations. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference fallback: type: string title: Fallback value description: The value to use for this tag when the field value is missing. minLength: 1 maxLength: 512 required: - value - if: properties: value_type: enum: - value then: title: Value properties: value: type: string title: Value description: The value for this tag. minLength: 1 maxLength: 128 required: - value properties: name: type: string title: Name description: The tag name pattern: ^[a-zA-Z][a-zA-Z0-9_]*$ minLength: 1 maxLength: 128 errorMessage: must be a valid metric tag name value_type: title: Value type description: Either a value from an event field, or a new value input. oneOf: - label: Value from event field enum: - field - label: New value enum: - value default: field ui: display_type: radio-group value: type: string required: - name - value_type - value additionalProperties: type: string required: - metric_name - metric_kind - metric_type - value - namespace - if: properties: type: enum: - clustering deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Clustering description: Finds clusters of similar values additionalProperties: false properties: cluster_field: type: string title: Field description: Field to cluster values on pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference default: .message similarity_threshold: type: number title: Similarity threshold description: A threshold from 0.0 (not similar) to 1.0 (exactly similar) that determines how close two values need to be to be consider as being part of the same cluster default: 0.4 minimum: 0 maximum: 1 max_clusters: type: integer title: Maximum cluster description: Maximum number of clusters kept cached by the transform default: 1000 minimum: 1 maximum: 2000 max_node_depth: type: integer title: Maximum tree depth description: Maximum depth of the prefix tree default: 4 minimum: 2 maximum: 8 max_children: type: integer title: Maximum children default: 100 minimum: 2 maximum: 200 description: Maximum number of children per node of the prefix tree required: - cluster_field - if: properties: type: enum: - redact deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Redact description: Detect and anonymize sensitive data in logs and metrics additionalProperties: false properties: field: type: string title: Field description: Detect patterns in the specified field. If the field is empty, the processor will look for patterns across all the fields. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 256 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference mask_patterns: type: array title: Redact Patterns description: List of PII patterns to detect and mask minItems: 1 maxItems: 20 items: type: object additionalProperties: type: string title: Redact Pattern description: A PII pattern to detect and/or anonymize allOf: - if: properties: action: enum: - Replace required: - action then: properties: replacement: type: string title: Replacement description: "The text to replace the detected pattern. Example: 000-00-0000" minLength: 1 maxLength: 512 required: - replacement - if: properties: action: enum: - Hash required: - action then: properties: algorithm: title: Algorithm description: Hash and encode (base16 or hex) matched PII data using the specified algorithm oneOf: - label: sha2 (sha-512_256) enum: - sha2 - label: sha3 (sha3-512) enum: - sha3 ui: display_type: list required: - algorithm - if: properties: name: enum: - custom required: - name then: properties: custom_options: allOf: - type: object additionalProperties: false properties: pattern: type: string title: Pattern description: The custom pattern minLength: 1 maxLength: 4096 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching default: true multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively default: false match_newline: type: boolean title: Dot all mode description: Dot matches newline character default: false crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" default: false ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines default: false required: - pattern title: Replace Expression description: The regular expression used to find replacements. Named capture groups are supported, e.g. `(?P<key>\w+):(?P<value>.+)`. ui: display_type: regex-pattern required: - custom_options properties: name: title: Pattern description: The PII pattern to match oneOf: - label: Credit card number enum: - credit_card - label: Email address enum: - email_address - label: IPv4 address enum: - ipv4_address - label: US or Canada phone number enum: - us_canada_phone_number - label: US social security number enum: - us_social_security_number - label: Custom pattern ui: feature_flag: pipeline-parser-regex enum: - custom ui: display_type: list action: type: string title: Action description: Action to take on the detected PII data enum: - Hash - Replace - Detect Only default: Replace required: - name - action required: - mask_patterns - if: properties: type: enum: - throttle deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Throttle description: Throttle (rate-limit) events passing through this component additionalProperties: false properties: threshold: type: number minimum: 1 maximum: 10000000 title: Limit description: The number of events to allow over the time window. If a Key Field is specified, the limit will be applied to events for each unique value of that field. default: 1 window_ms: type: number minimum: 1000 maximum: 2592000000 title: Window description: The time window over which the configured limit is applied ui: display_type: milliseconds-to-human-readable default: 5000 key_field: type: string title: Key Field description: When configured, the value from this field will be used to group events into separate buckets for throttling. If left blank, or if the event does not contain a value in this field, the event will be throttled as part of the default bucket. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference exclude: allOf: - $ref: "#condition-group" title: Exclude description: When configured, events matching this condition will not be throttled. ui: display_type: rules-engine required: - threshold - window_ms - if: properties: type: enum: - set-timestamp deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Set Timestamp description: Parse a list of fields using a chosen time format. First match will override the default timestamp of the event. additionalProperties: false properties: parsers: type: array title: List of Fields and Parsers description: The list of fields and parsers to use in order of priority, short-circuiting on the first successful match. ui: display_type: multi-output-map options: sortable: false items: type: object properties: field: type: string title: Field description: The field whose value will be parsed by the chosen date pattern pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference options: $ref: "#timestamp-parser-options" required: - field - options minItems: 1 required: - parsers - if: properties: type: enum: - data-profiler deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Data Profiler description: Profile the data sent through your pipeline. The system will generate annotations such as event type and total bytes to be used downstream. It will also normalize the app/host/level/line fields before sending the event downstream. A profile of the data including total, events, event size, log composition, and message templates will be generated as well. additionalProperties: false properties: app_fields: type: array title: App Path(s) description: A list of paths to look for the app value minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: App description: The path to the app maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .app - .application - .container host_fields: type: array title: Host Path(s) description: A list of paths to look for the host value minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Host description: The path to the host maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .host - .hostname level_fields: type: array title: Level Path(s) description: A list of paths to look for the log level minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Level description: The path to the level maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .level - .log_level line_fields: type: array title: Line Path(s) description: A list of paths to look for the event line minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Line description: The path to the line maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .line - .message label_fields: type: array title: Label Path(s) description: A list of paths to look for the event labels maxItems: 10 uniqueItems: true items: type: string title: Label description: The path to the labels maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .labels field_summary: type: boolean default: false title: Field Summary description: If enabled the profiler will attempt to summarize the values in fields of events passing through the node. ui: display_type: hidden ff_default_value: pipeline-node-transform-field-summary required: - app_fields - host_fields - level_fields - line_fields - if: properties: type: enum: - replace deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Replace description: Replace parts of a string using a regular expression or static string additionalProperties: type: object allOf: - if: properties: replacement_type: enum: - regex required: - replacement_type then: properties: options: allOf: - type: object additionalProperties: false properties: pattern: type: string minLength: 1 maxLength: 4096 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching default: true multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively default: false match_newline: type: boolean title: Dot all mode description: Dot matches newline character default: false crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" default: false ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines default: false required: - pattern title: Replace Expression description: The regular expression used to find replacements. Named capture groups are supported, e.g. `(?P<key>\w+):(?P<value>.+)`. ui: display_type: regex-pattern replace_with: type: string minLength: 1 maxLength: 4096 title: Replacement String description: The replacement string can be static or optionally contain capture group references. Capture groups can be positional or named. For example, to switch the position of two strings in the source string `onetwo`, the pattern could be `(?Pone)(?P.+)`, and the replacement would be `$two$one`. To do the same thing with positional capture groups, a pattern of `(\w{3})(\w{3})`, and a replacement of `$2$1` would have the same result.

To render a literal dollar sign `$` in the replacement, escape it with `$$`. So, if you had a capture group named `price`, to prepend a dollar sign to the replacement, you would use `$$$price`. required: - options - replace_with - if: properties: replacement_type: enum: - string required: - replacement_type then: properties: string_pattern: type: string title: String Pattern minLength: 1 maxLength: 4096 description: A static, case-sensitive string, which will be searched and replaced. replace_with: type: string minLength: 1 maxLength: 4096 title: Replacement String description: The static string used in each replacement. required: - string_pattern - replace_with properties: source_field: type: string title: Source Field description: The field that contains a string for replacement to operate on. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference max_times: type: number title: Maximum Replacements description: The maximum number of replacements to be made. If not specified, all occurrences will be replaced. minimum: 1 replacement_type: title: Replacement Type description: Specifies how the source string will be searched for replacement. ui: display_type: list errorMessage: must be one of the available pattern types oneOf: - label: String description: Replace a specified string one or more times. enum: - string - label: Regular Expression description: Replace using a regular expression. enum: - regex required: - source_field - replacement_type - if: properties: type: enum: - trace-sampling deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Trace Sampling description: Performs sampling of traces per a given rate using the Trace ID as the unique identifier. If a trace is rate selected, all spans of the trace are forwarded. If the Trace ID cannot be found, the event is dropped. allOf: - if: properties: sample_type: enum: - head then: properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 unique traces will be forwarded and all others will be dropped default: 10 minimum: 2 maximum: 100000 parent_span_id_field: type: "null" conditionals: type: "null" required: - rate - if: properties: sample_type: enum: - tail then: properties: parent_span_id_field: type: string default: .parent_span_id title: Parent Span ID description: The field to use as the parent span identifier pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference conditionals: type: array minItems: 1 title: Conditionals description: A list of complete conditionals, including expressions and a sample rate to maintain items: type: object additionalProperties: false title: Conditional description: A list of conditions and a sample rate to maintain properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 unique traces will be forwarded and all others will be dropped. A `rate = 1` means keep all traces that match the condition. minimum: 1 maximum: 100000 default: 10 conditional: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. required: - rate - conditional required: - parent_span_id_field - conditionals properties: sample_type: type: string description: The type of trace sampling to apply. enum: - head title: Sample Type default: head trace_id_field: type: string default: .trace_id title: Trace ID description: The field to use as the identifier of a trace. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - sample_type - trace_id_field properties: title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description type: type: string enum: - filter - sample - drop-fields - parse - parse-sequentially - route - stringify - compact-fields - flatten-fields - dedupe - encrypt-fields - decrypt-fields - js-script - unroll - vrl - reduce - map-fields - metrics-tag-cardinality-limit - aggregate - event-to-metric - clustering - redact - throttle - set-timestamp - data-profiler - replace - trace-sampling user_config: type: object required: - user_config sinks: type: array minItems: 1 items: type: object additionalProperties: false allOf: - if: properties: type: enum: - mezmo deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Log Analysis description: Send log data to Mezmo Log Analysis additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true mezmo_host: type: string title: Mezmo Host description: The host for your Log Analysis environment default: null minLength: 1 ingestion_key: type: string title: Ingestion Key description: Ingestion key ui: display_type: mezmo-ingestion-key minLength: 1 query: type: object title: Query Parameters default: hostname: mezmo properties: hostname: type: string title: Hostname description: Hostname string or template to extract event values to form a hostname. Depending on the type of ingestion, this value is usually set from other places so the default value is often sufficient. If the value in the event is null, empty, or doesn't exist, when using a template then it will be set to "mezmo" default: mezmo minLength: 1 ui: display_type: data-template tags: type: array title: Tags description: List of tag strings or templates to attach to logs. Template must resolve to either an array of strings, or a string. Commas are treated as tag delimiters. items: type: string title: Tag description: Tag string or template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template ip: type: string title: IP description: IP address template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template mac: type: string title: MAC description: MAC address template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template message: title: Log Construction Scheme default: scheme: Explicit field selection allOf: - if: properties: scheme: enum: - Explicit field selection then: properties: line: type: string title: Line description: Template or field reference to use as the log line. Field reference can point to an object. minLength: 1 maxLength: 1024 meta_field: type: string title: Meta Field description: Field containing the meta object for the log pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference maxLength: 512 app: type: string title: App description: App name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template file: type: string title: File description: File name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template timestamp_field: type: string title: Timestamp description: Field containing the timestamp for the log pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference maxLength: 512 env: type: string title: Env description: Environment name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template properties: scheme: type: string title: Scheme description: How to construct the log message. enum: - Explicit field selection - Message pass-through default: Explicit field selection required: - scheme required: - mezmo_host - ingestion_key - if: properties: type: enum: - mezmo-archive deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Archive Destination description: Store logs for future imports into mezmo pipeline or log analysis. additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true destination: title: Destination description: "" allOf: - if: properties: archive_provider: enum: - Azure then: properties: container_name: type: string title: Container Name minLength: 1 description: The name of the container for blob storage. connection_string: type: string title: Connection String minLength: 1 description: A connection string for the account that contains an access key required: - container_name - connection_string - if: properties: archive_provider: enum: - S3 then: properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret bucket: type: string title: Bucket pattern: (?!(^xn--|.+-s3alias$))^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ description: The S3 bucket name. Do not include a leading s3:// or a trailing / region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - access_key_id - secret_access_key - bucket - region properties: archive_provider: type: string title: Archive Provider description: The storage destination for your data. enum: - Azure - S3 default: Azure required: - archive_provider required: - destination - if: properties: type: enum: - http deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: HTTP description: Send data to a HTTP endpoint additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI to make HTTP requests to. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none auth: title: Authentication description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 description: The basic authentication user password: type: string title: Password minLength: 1 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none headers: type: array title: HTTP Headers description: An array of key/value objects for the headers to be sent with the request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 description: The value of the header. required: - header_name - header_value advanced_options: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-http-advanced-options title: Advanced Options description: Settings available for fine-tuning the destination behavior. additionalProperties: false dependencies: payload_prefix: - payload_suffix payload_suffix: - payload_prefix properties: max_bytes: type: integer title: Payload Size description: The maximum number of uncompressed bytes when batching data to the destination. minimum: 1 maximum: 2097152 timeout_secs: type: integer title: Timeout Seconds description: The number of seconds before a destination write timeout. minimum: 5 method: type: string enum: - post - put - patch - delete - get - head - options - trace title: HTTP Method description: The HTTP method to use for the destination. payload_prefix: type: string title: Payload Prefix description: Add a prefix to the payload. Only used for serialized JSON chunks. This option also requires "Payload Suffix" to form a valid JSON string. examples: - '{"event":"' - '{"extra_prop": true, "our_thing":"' minLength: 1 payload_suffix: type: string title: Payload Suffix description: Used in combination with "Payload Prefix" to form valid JSON from the payload. examples: - "}" - '"extra_prop": true }' minLength: 1 proxy: type: object additionalProperties: false title: Proxy Settings description: Configure proxy settings for the destination. properties: enabled: type: boolean title: Enabled description: Turns proxying on/off default: false endpoint: type: string minLength: 1 title: HTTP(S) Endpoint description: HTTP or HTTPS Endpoint to use for traffic. format: uri errorMessage: format: Must be a valid URI hosts_bypass_proxy: type: array minItems: 1 title: Hosts to Bypass description: A list of hosts to bypass proxying. Can be specified as a domain name, IP address, or CIDR block. Wildcards are supported as a dot (.) in domain names, or as a star (*) to match all hosts. examples: - 93.184.216.34 - ignorehost.example.com - .ignoreall.example.com - 93.184.216.0/24 items: type: string title: Host description: Domain name, IP address, or CIDR block. minLength: 1 tls_protocols: type: array title: TLS Protocols description: A list of ALPN protocols to use during TLS negotiation. They are attempted in the order they appear. items: type: string title: Protocol description: An ALPN protocol name minLength: 1 minItems: 1 rate_limiting: type: object additionalProperties: false title: Rate Limiting description: Settings for controlling rate limiting to the destination. properties: request_limit: type: integer minimum: 1 title: Request Limit description: The max number of requests allowed within the specified "Duration Seconds" setting. duration_secs: type: integer default: 1 title: Duration Seconds description: The window of time used to apply "Request Limit." minimum: 1 required: - uri - encoding - compression - if: properties: type: enum: - cloudwatch-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Cloudwatch Logs description: Publish log events to AWS Cloudwatch Logs additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none group_name: type: string title: Group Name minLength: 1 description: The name of the log group for the targeted log stream. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the targeted log stream. required: - auth - encoding - compression - group_name - region - stream_name - if: properties: type: enum: - cloudwatch-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Cloudwatch Metrics description: Send metric events to AWS Cloudwatch Metrics additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none namespace: type: string title: Namespace minLength: 1 description: Name for the container that will isolate metrics from one another. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - auth - compression - namespace - region - if: properties: type: enum: - sqs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Simple Queue Service (SQS) description: Publish log events to AWS Simple Queue Service additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text sqs_queue_url: type: string title: SQS Queue URL format: url errorMessage: format: Must be a valid URL description: The URL of the SQS queue that the data is publishing to. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - auth - encoding - sqs_queue_url - region - if: properties: type: enum: - notification-channel deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Notification Channel description: Send a Message to a 3rd Party Service. additionalProperties: false properties: service: title: Service description: Configure a service to receive the payload. Currently supports Slack, PagerDuty, Webhook or Mezmo (Log Analysis). Where allowed, messagetemplating can be used. To use a template, surround field names with double curly braces, e.g., {{.my_field}} or {{."my-object".thing}}. See [documentation](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) for more examples. additionalProperties: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 allOf: - if: properties: name: enum: - slack required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The text to be sent as a Slack message. required: - uri - message_text - if: properties: name: enum: - pager_duty required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 summary: type: string title: Summary description: Summarize the alert details for PagerDuty. minLength: 1 ui: display_type: data-template severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. source: type: string title: Source description: The source of the alert. minLength: 1 ui: display_type: data-template routing_key: type: string title: Routing Key description: The service integration routing key. minLength: 1 ui: display_type: secret event_action: type: string enum: - trigger - acknowledge - resolve title: Event Action description: The event action associated with the alert. required: - uri - summary - severity - source - routing_key - event_action - if: properties: name: enum: - webhook required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 method: type: string title: HTTP Method description: The HTTP method to use for the request. enum: - post - put - patch - delete - get - head - options - trace ui: default_hint: post message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The Webhook payload to send. This can be a text string or stringified JSON. If the message can be parsed as JSON, it will be sent as such. headers: type: array title: HTTP Headers description: A list of key/value pairs to be included in the HTTP request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 maxLength: 1024 description: The value of the header. required: - header_name - header_value maxItems: 20 auth: title: Authentication Options description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 description: The basic authentication user password: type: string title: Password minLength: 1 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: Choose basic or token-based authentication. default: none required: - uri - message_text - if: properties: name: enum: - log_analysis required: - name then: properties: severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. default: INFO subject: type: string title: Subject description: A string to be used as the subject of the alert. minLength: 1 maxLength: 200 ui: display_type: data-template body: type: string title: Body description: A string to be used as the alert message body. minLength: 1 maxLength: 1024 ui: display_type: data-template ingestion_key: type: string title: Ingestion Key description: The key required to communicate to Log Analysis for sending the alert. ui: display_type: mezmo-ingestion-key minLength: 1 required: - severity - subject - body - ingestion_key properties: name: title: Name description: The name of the service that will receive the notification. ui: display_type: list errorMessage: must be a valid service name oneOf: - label: Slack enum: - slack - label: PagerDuty enum: - pager_duty - label: Webhook enum: - webhook - label: Mezmo Log Analysis enum: - log_analysis required: - name throttling: type: object title: Throttling description: Only allow a certain number of notifications to be sent within a given time window. properties: window_secs: type: integer title: Window Seconds description: The time frame during which the number of notifications (set by the Threshold) is permitted. default: 60 minimum: 1 threshold: type: integer title: Threshold description: 'The maximum number of notifications allowed over the given time window set by "Window Seconds" (default: 1).' default: 1 minimum: 1 required: - service - throttling - if: properties: type: enum: - azure-blob-storage deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Azure Blob Storage description: Publish log events to Azure Blob Storage additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none container_name: type: string title: Container Name minLength: 1 description: The name of the container for blob storage. connection_string: type: string title: Connection String minLength: 1 description: A connection string for the account that contains an access key prefix: type: string title: Prefix minLength: 1 description: A prefix to be applied to all object keys ui: display_type: data-template file_consolidation: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-azure-blob-file-consolidation title: File Consolidation description: This sink writes many small files out to azure blob storage. Enabling this process will allow the automatic consolidation of these small files into larger files of your choosing. This process will enable upon deployment and run on the chosen interval from `Processing Interval` creating files named `merged_[timestamp].log` where `timestamp` is the time since epoch when the actual file was created. The process will recursively access all files under the `Base Path` to handle merging sub-directory logging structures. additionalProperties: false properties: enabled: type: boolean default: false title: Enabled description: Toggles whether the process is enabled. process_every_seconds: type: integer title: Processing Interval (seconds) description: How often to run the consolidation process. default: 600 minimum: 300 maximum: 3600 requested_size_bytes: type: integer title: Requested File Size (bytes) description: The requested size of the consolidated files. default: 500000000 minimum: 50000000 maximum: 10000000000 base_path: type: string title: Base Path default: "" maxLength: 512 description: The path from the container to begin recursivelylooking for files to merge. A blank value indicates root.Merged files will only contain data from the respectivefolder. required: - encoding - compression - container_name - connection_string - if: properties: type: enum: - kinesis-firehose deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Kinesis Data Firehose description: Publish log events to AWS Kinesis Data Firehose additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the Kinesis Firehose stream required: - auth - encoding - compression - region - stream_name - if: properties: type: enum: - kinesis-streams deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Kinesis Streams description: Publish log events to AWS Kinesis Streams additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none partition_key_field: type: string title: Partition Key Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ description: The field in the log whose value is used as Kinesis' partition key. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the targeted log stream. required: - auth - encoding - compression - region - stream_name - if: properties: type: enum: - elasticsearch deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Elasticsearch description: Publish log events to Elasticsearch additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none auth: title: ElasticSearch Auth Strategy description: ElasticSearch authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - aws then: properties: access_key_id: type: string title: Access Key ID minLength: 1 maxLength: 1024 description: AWS access key ID ui: display_type: secret secret_access_key: type: string title: Secrete Access Key minLength: 1 maxLength: 1024 description: AWS secrete access key ui: display_type: secret region: type: string title: Region minLength: 1 maxLength: 128 description: The name of the AWS region that is targeted. required: - access_key_id - secret_access_key - region properties: strategy: type: string title: Strategy description: The ES authentication strategy to use enum: - basic - aws required: - strategy endpoints: type: array title: Endpoints uniqueItems: true minItems: 1 description: An array of ElasticSearch endpoints items: type: string title: URL format: url errorMessage: format: Must be a valid URL description: The full Elasticsearch URL to send data to. minLength: 1 maxLength: 1024 pipeline: type: string title: Pipeline minLength: 1 description: Name of an ES ingest pipeline to include. index: type: string minLength: 1 maxLength: 512 title: Custom Index description: Index to use when writing ES events (default = mezmo-%Y.%m.%d) default: mezmo-%Y.%m.%d required: - compression - auth - endpoints - if: properties: type: enum: - gcp-cloud-monitoring deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud Monitoring description: Publish metrics events to GCP Cloud Monitoring additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. resource_type: type: string title: Resource Type minLength: 1 description: The `monitored-resource` type as defined in Monitoring. resource_labels: type: array title: Resource Labels description: An array of key/val objects describing labels of the Monitoring resource. items: type: object title: Label/Value description: A key/val object describing a label name and its value. additionalProperties: false properties: label_name: type: string title: Label Name minLength: 1 description: The name of the label. label_value: type: string title: Label Value minLength: 1 description: The value of the label. required: - label_name - label_value required: - credentials_json - project_id - resource_type - if: properties: type: enum: - gcp-cloud-storage deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud Storage description: Publish log events to GCP Cloud Storage additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file encoding: type: string description: Dictates how the data will be serialized before storing. enum: - json - ndjson - text title: Encoding default: text bucket: type: string title: Bucket minLength: 1 description: The name of the bucket in GCP where the data will be stored. compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none bucket_prefix: type: string title: Bucket Prefix minLength: 1 description: The prefix applied to the bucket name, giving the appearance of having directories. required: - credentials_json - encoding - bucket - compression - if: properties: type: enum: - gcp-cloud-operations deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud Operations description: Publish log events to GCP Cloud Operations additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file log_id: type: string title: Log ID minLength: 1 description: Concise reference for the log stream name. project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. resource_type: type: string title: Resource Type minLength: 1 description: The `monitored-resource` type as defined in Monitoring. resource_labels: type: array title: Resource Labels description: An array of key/val objects describing labels of the Monitoring resource. items: type: object title: Label/Value description: A key/val object describing a label name and its value. additionalProperties: false properties: label_name: type: string title: Label Name minLength: 1 description: The name of the label. label_value: type: string title: Label Value minLength: 1 description: The value of the label. required: - label_name - label_value required: - credentials_json - log_id - project_id - resource_type - if: properties: type: enum: - gcp-cloud-pubsub deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud PubSub description: Publish log events to GCP Cloud PubSub additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file encoding: type: string description: Dictates how the data will be serialized before storing. enum: - json - text title: Encoding default: text project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. topic: type: string title: Topic minLength: 1 description: The name of the topic to publish messages to. required: - credentials_json - encoding - project_id - topic - if: properties: type: enum: - honeycomb-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Honeycomb Logs description: Send log data to Honeycomb additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Honeycomb API key description: Your Honeycomb API key minLength: 1 maxLength: 512 ui: display_type: secret dataset: type: string title: Dataset description: The name of the targeted dataset. minLength: 1 maxLength: 512 required: - api_key - dataset - if: properties: type: enum: - indexed-search deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Indexed Search description: Publish log events to Mezmo's data store for Indexed Search capabilities additionalProperties: true properties: search_field: type: string title: Full Text Search Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: .line maxLength: 256 description: If your data reaches this stage as a plain-text string, the entire payload will be indexed for full-text searching. If your data reaches this stage as an object, the field you enter here will be used as the full-text search field. Nested fields are allowed. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference timestamp_field: type: string title: Timestamp Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: timestamp maxLength: 256 description: If your data reaches this stage as an object, this field represents the timestamp value to be cataloged for searching. If the field does not exists the current time will be used. We accept UNIX timestamps (in milliseconds), ISO8601/RFC3389, and RFC2822 formats. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - search_field - timestamp_field - if: properties: type: enum: - kafka deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Kafka description: Publish log events to Kafka additionalProperties: type: string allOf: - if: properties: sasl_enabled: enum: - true then: properties: sasl_mechanism: type: string enum: - SCRAM-SHA-512 - SCRAM-SHA-256 - PLAIN title: SASL description: The Kafka SASL/SCRAM mechanism default: SCRAM-SHA-512 sasl_username: type: string title: Username description: The SASL/SCRAM username minLength: 1 sasl_password: type: string title: Password description: The SASL/SCRAM username ui: display_type: secret minLength: 1 required: - sasl_mechanism - sasl_username - sasl_password - if: properties: tls_enabled: enum: - true then: properties: tls_ca_certificate: type: string title: TLS CA Certificate Chain minLength: 1 description: The CA certificate chain in the PEM format ui: display_type: secret-file format: x509_certificate errorMessage: format: Must be a valid PEM or DER certificate tls_verify_certificate: type: boolean title: TLS Verify Certificate description: Verify the TLS certificates default: true required: - tls_verify_certificate properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - lz4 - snappy - zstd - none title: Compression default: none event_key_field: type: string title: Event Key Field minLength: 1 description: The field in the log whose value is used as Kafka's event key. brokers: type: array title: Brokers uniqueItems: true minItems: 1 description: An array of key/val objects for each host/port of a Kafka broker. items: type: object title: Host/Port description: The host and port for the Kafka broker. additionalProperties: false properties: host: title: Host description: The hostname or IP address of the broker. anyOf: - type: string format: ipv4 errorMessage: format: Must be a valid IPv4 address - type: string format: hostname errorMessage: format: Must be a valid hostname port: type: number title: Port description: The port that the broker listens on. required: - host - port topic: type: string title: Topic minLength: 1 description: The name of the topic to publish to. group_id: type: string title: Group ID minLength: 1 description: The consumer group identifier sasl_enabled: type: boolean title: SASL/SCRAM Enabled description: Enable SASL/SCRAM authentication with Kafka default: false tls_enabled: type: boolean title: Use TLS description: Enable TLS during connections to the remote default: false required: - encoding - compression - brokers - topic - if: properties: type: enum: - loki deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Loki description: Publish log events to Loki additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: title: Loki Auth Strategy description: Loki authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token (passed as-is) ui: display_type: secret required: - token properties: strategy: type: string title: Strategy description: The authentication strategy to use (only basic supported) enum: - basic - bearer required: - strategy encoding: type: object title: Encoding description: Configures how event are encoded properties: codec: type: string title: Codec enum: - json - text required: - codec endpoint: type: string title: Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Loki base URL path: type: string title: Path minLength: 1 maxLength: 512 description: The path appended to the Loki base URL(defaults to /loki/api/v1/push) default: /loki/api/v1/push labels: type: array title: Loki Labels description: Key/Value pair used to describe Loki data items: type: object title: Labels description: Label name and values additionalProperties: false properties: label_name: type: string title: Label Name description: The name of the label minLength: 1 maxLength: 512 label_value: type: string title: Label Value description: The value of the label minLength: 1 maxLength: 512 required: - label_name - label_value required: - auth - encoding - endpoint - labels - if: properties: type: enum: - new-relic deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: New Relic description: Publish to New Relic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true account_id: type: string title: New Relic Account ID description: Account ID for your New Relic account minLength: 1 maxLength: 1024 ui: display_type: secret api: type: string title: API description: New Relic API endpoint (events not supported) enum: - logs - metrics default: logs license_key: type: string title: New Relic License Key description: License key for your New Relic account minLength: 1 maxLength: 1024 ui: display_type: secret required: - account_id - api - license_key - if: properties: type: enum: - prometheus-remote-write deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Prometheus Remote Write description: Publish metrics to Prometheus Remote Write endpoint additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Endpoint format: uri errorMessage: format: Must be a valid URI description: "The full URI to make HTTP requests to. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. Example: http://example.org:8080/api/v1/push" minLength: 1 maxLength: 2000 auth: title: Authentication description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none required: - strategy required: - endpoint - if: properties: type: enum: - pulsar deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Pulsar description: Publish log events to Apache Pulsar additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text url: type: string title: URL format: url errorMessage: format: Must be a valid URL description: The full Pulsar URL to send data to. topic: type: string title: Topic minLength: 1 description: The name of the Pulsar topic to publish to. required: - encoding - url - topic - if: properties: type: enum: - redis deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Redis description: Publish log events to Redis additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text url: type: string title: URL format: uri errorMessage: format: Must be a valid URL description: The full Redis URL to send data to. key: type: string title: Key minLength: 1 description: The name of the Redis key to write data to. required: - encoding - url - key - if: properties: type: enum: - splunk-hec-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Splunk HTTP Event Collector description: Publish log events to Splunk HTTP Event Collector additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none url: type: string title: Endpoint format: url errorMessage: format: Must be a valid URL description: The base URL for the Splunk instance. The collector path, such as `/services/collector/events`, will be automatically inferred from the destination's configuration. token: type: string title: Token minLength: 1 ui: display_type: secret description: The default token to authenticate to Splunk HEC. tls_verify_certificate: type: boolean title: Verify TLS Certificate description: If unchecked, the TLS certificate of the remote host will not be verified. **This is insecure**, but may be useful for testing or when evaluating Splunk in a trial instance. default: true host_field: type: string title: Host Field description: The field that contains the hostname to include in the event. default: metadata.host timestamp_field: type: string title: Timestamp Field description: The field that contains the timestamp to include in the event. default: metadata.time pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: feature_flag: pipeline-splunk-sink-v2 display_type: field-reference source: title: Source description: The source of events sent to this sink. This is typically the filename the logs originated from. Use the field path ".metadata.source" to use the upstream source value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - "null" required: - value_type sourcetype: title: Source Type description: The sourcetype of events sent to this sink. Use the field path ".metadata.sourcetype" to use the upstream sourcetype value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - "null" required: - value_type index: title: Index description: The name of the index to send events to. Use the field path ".metadata.index" to use the upstream index value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - "null" required: - value_type required: - compression - url - token - source - sourcetype - index - if: properties: type: enum: - sumo-logic-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sumo Logic Logs description: Publish log data to Sumo Logic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Sumo Logic Collector URL that receives the data ui: display_type: secret compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none category: type: string minLength: 1 maxLength: 512 title: Category description: Category to use when publishing data to Sumo Logic (default = mezmo_default_category) default: mezmo_default_category required: - endpoint - compression - if: properties: type: enum: - sumo-logic-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sumo Logic Metrics description: Publish metrics data to Sumo Logic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Sumo Logic Collector URL that receives the data ui: display_type: secret compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none category: type: string minLength: 1 maxLength: 512 title: Category description: Category to use when publishing data to Sumo Logic (default = mezmo_default_category) default: mezmo_default_category required: - endpoint - compression - if: properties: type: enum: - vector deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Vector description: Publish log events to Vector additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none vector_ip: type: string title: Vector IP format: ipv4 description: The IP portion of the Vector host vector_port: type: number title: Vector Port description: The port that the Vector host listens on. version: type: string enum: - "1" - "2" title: Version default: "2" description: The version of the Vector sink which must match the source version. required: - compression - vector_ip - vector_port - version - if: properties: type: enum: - s3 deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS S3 description: Publish events as objects in AWS S3 additionalProperties: false errorMessage: properties: bucket: must be a valid bucket name tags: must be a valid tag identifier properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key bucket: type: string title: Bucket pattern: (?!(^xn--|.+-s3alias$))^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ description: The S3 bucket name. Do not include a leading s3:// or a trailing / prefix: type: string title: Prefix description: A prefix to apply to all object key names. default: / ui: display_type: data-template tags: type: array maxItems: 10 title: Tags description: A set of tags (key/value pairs) to apply to the object in S3. items: type: object properties: key: type: string title: Name description: The tag name pattern: ^[A-Za-z0-9_.:=+@/\-\s]*$ minLength: 1 maxLength: 128 value: type: string title: Value description: The tag value default: "" pattern: ^[A-Za-z0-9_.:=+@/\-\s]*$ minLength: 0 maxLength: 256 required: - key - value encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. file_consolidation: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-s3-file-consolidation title: File Consolidation description: "This sink writes many small files out to S3. Enabling this process will allow the automatic consolidation of these small files into larger files of your choosing. This process will enable upon deployment and run on the chosen interval from `Processing Interval` creating files named `merged_[timestamp].log` where `timestamp` is the time since epoch. *NOTE*: only `text` and `ndjson` files written by the sink are consolidated." additionalProperties: false properties: enabled: type: boolean default: false title: Enabled description: Toggles whether the process is enabled. process_every_seconds: type: integer title: Processing Interval (seconds) description: How often to run the consolidation process. default: 600 minimum: 300 maximum: 3600 requested_size_bytes: type: integer title: Requested File Size (bytes) description: The size of the requested files. default: 500000000 minimum: 50000000 maximum: 10000000000 base_path: type: string title: Base Path default: "" maxLength: 512 description: The path from the container to begin recursivelylooking for files to merge. A blank value indicates root.Merged files will only contain data from the respectivefolder. required: - auth - bucket - prefix - encoding - compression - region - if: properties: type: enum: - blackhole deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Blackhole description: Drop data arriving at this destination additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true - if: properties: type: enum: - datadog-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Datadog Logs description: Publish log events to Datadog additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Datadog API Key description: Datadog log application API key minLength: 1 ui: display_type: secret compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: gzip site: type: string description: The Datadog site (region) to send logs to. enum: - us1 - us3 - us5 - eu1 title: Datadog Site default: us1 required: - api_key - compression - site - if: properties: type: enum: - datadog-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Datadog Metrics description: Publish metrics events to Datadog. Sending custom metrics to Datadog may result in extra charges on your Datadog bill. Please go to https://docs.datadoghq.com/metrics/custom_metrics/ for more information on Datadog custom metrics pricing. additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Datadog API Key description: Datadog metrics application API key minLength: 1 maxLength: 512 ui: display_type: secret site: type: string description: The Datadog site (region) to send metrics to. enum: - us1 - us3 - us5 - eu1 title: Datadog Site default: us1 required: - api_key - site - if: properties: type: enum: - clickhouse deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Clickhouse description: Deliver log data to the ClickHouse database additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string minLength: 1 maxLength: 2048 title: Endpoint description: Endpoint of the Clickhouse server format: uri errorMessage: format: Must be a valid URI auth: title: Auth Strategy description: Authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 2048 description: The bearer token to use for authentication ui: display_type: secret required: - token properties: strategy: type: string title: Strategy description: The ES authentication strategy to use enum: - basic - bearer required: - strategy table: type: string title: Table description: The table that data is inserted into pattern: ^[a-zA-Z0-9_]+$ errorMessage: Invalid table name ui: display_type: data-template database: type: string title: Database description: The database that contains the table that data is inserted into pattern: ^[a-zA-Z0-9_]+$ errorMessage: Invalid database name ui: display_type: data-template compression: type: string enum: - gzip - none - snappy - zlib - zstd title: Compression description: The compression algorithm to use default: gzip skip_unknown_fields: type: boolean title: Skip Unknown Fields description: Allows ClickHouse to discard fields not present in the table schema default: false required: - endpoint - auth - table - if: properties: type: enum: - opentelemetry deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry description: Publish data to OpenTelemetry additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The OpenTelemetry Collector URL that receives the data auth: title: Authentication description: Configures HTTP authentication additionalProperties: type: string allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none headers: type: array title: HTTP Headers description: An array of key/value objects for the headers to be sent with the request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 description: The value of the header. required: - header_name - header_value compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none required: - endpoint - compression properties: title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description type: type: string enum: - mezmo - mezmo-archive - http - cloudwatch-logs - cloudwatch-metrics - sqs - notification-channel - azure-blob-storage - kinesis-firehose - kinesis-streams - elasticsearch - gcp-cloud-monitoring - gcp-cloud-storage - gcp-cloud-operations - gcp-cloud-pubsub - honeycomb-logs - indexed-search - kafka - loki - new-relic - prometheus-remote-write - pulsar - redis - splunk-hec-logs - sumo-logic-logs - sumo-logic-metrics - vector - s3 - blackhole - datadog-logs - datadog-metrics - clickhouse - opentelemetry user_config: type: object required: - user_config required: - sources - transforms - sinks required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/usage-threshold-alert/{usage_threshold_alert_id}: delete: summary: Remove usage threshold alert tags: - Alerting description: Removes an existing threshold alert parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: usage_threshold_alert_id required: true responses: "200": description: Default Response get: summary: Retrieves an existing usage threshold alert tags: - Alerting description: Returns usage threshold configuration parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: usage_threshold_alert_id required: true responses: "200": description: Default Response patch: summary: Partially update an existing threshold alert tags: - Alerting description: Update a part of usage threshold alert requestBody: content: application/json: schema: type: object properties: enabled: type: boolean required: - enabled required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: usage_threshold_alert_id required: true responses: "200": description: Default Response put: summary: Update threshold alert tags: - Alerting description: Replace an existing usage alert with this one requestBody: content: application/json: schema: type: object additionalProperties: false allOf: - if: allOf: - type: object properties: no_data: not: enum: - true - type: object properties: percent_decrease: anyOf: - enum: - 0 - type: "null" - type: object properties: percent_increase: anyOf: - enum: - 0 - type: "null" then: properties: threshold_bytes: type: number minimum: 1 errorMessage: minimum: must be greater than or equal to 1 required: - threshold_bytes - if: allOf: - type: object properties: no_data: not: enum: - true - type: object properties: percent_increase: anyOf: - enum: - 0 - type: "null" - type: object properties: threshold_bytes: anyOf: - enum: - 0 - type: "null" then: properties: percent_decrease: type: number minimum: 1 errorMessage: minimum: must be greater than or equal to 1 required: - percent_decrease - if: allOf: - type: object properties: no_data: not: enum: - true - type: object properties: percent_decrease: anyOf: - enum: - 0 - type: "null" - type: object properties: threshold_bytes: anyOf: - enum: - 0 - type: "null" then: properties: percent_increase: type: number minimum: 1 errorMessage: minimum: must be greater than or equal to 1 required: - percent_increase properties: data_window: type: object additionalProperties: false title: Data period options properties: value: $ref: "#alert_period_value" unit: $ref: "#alert_period_unit" if: properties: unit: title: Data period unit enum: - minute required: - unit then: properties: value: type: number title: Data period value minimum: 5 required: - value required: - value - unit max_daily_notifications: $ref: "#alert_max_notifications" no_data: type: boolean percent_decrease: $ref: "#alert_percent_decrease" percent_increase: $ref: "#alert_percent_increase" threshold_bytes: $ref: "#alert_threshold_bytes" title: $ref: "#alert_title" traffic_type: $ref: "#alert_traffic_type" alert_channels: $ref: "#alert_channels" required: - traffic_type - alert_channels required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/{pipeline_id}/usage-threshold-alert: get: summary: List all threshold alerts tags: - Alerting description: Returns configuration of existing threshold alerts parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response post: summary: Add usage alert tags: - Alerting description: Creates a new usage threshold alert configuration requestBody: content: application/json: schema: type: object additionalProperties: false allOf: - if: allOf: - type: object properties: no_data: not: enum: - true - type: object properties: percent_decrease: anyOf: - enum: - 0 - type: "null" - type: object properties: percent_increase: anyOf: - enum: - 0 - type: "null" then: properties: threshold_bytes: type: number minimum: 1 errorMessage: minimum: must be greater than or equal to 1 required: - threshold_bytes - if: allOf: - type: object properties: no_data: not: enum: - true - type: object properties: percent_increase: anyOf: - enum: - 0 - type: "null" - type: object properties: threshold_bytes: anyOf: - enum: - 0 - type: "null" then: properties: percent_decrease: type: number minimum: 1 errorMessage: minimum: must be greater than or equal to 1 required: - percent_decrease - if: allOf: - type: object properties: no_data: not: enum: - true - type: object properties: percent_decrease: anyOf: - enum: - 0 - type: "null" - type: object properties: threshold_bytes: anyOf: - enum: - 0 - type: "null" then: properties: percent_increase: type: number minimum: 1 errorMessage: minimum: must be greater than or equal to 1 required: - percent_increase properties: data_window: type: object additionalProperties: false title: Data period options properties: value: $ref: "#alert_period_value" unit: $ref: "#alert_period_unit" if: properties: unit: title: Data period unit enum: - minute required: - unit then: properties: value: type: number title: Data period value minimum: 5 required: - value required: - value - unit max_daily_notifications: $ref: "#alert_max_notifications" no_data: type: boolean percent_decrease: $ref: "#alert_percent_decrease" percent_increase: $ref: "#alert_percent_increase" threshold_bytes: $ref: "#alert_threshold_bytes" title: $ref: "#alert_title" traffic_type: $ref: "#alert_traffic_type" alert_channels: $ref: "#alert_channels" required: - traffic_type - alert_channels required: true parameters: - schema: type: string format: uuid in: path name: pipeline_id required: true responses: "200": description: Default Response /pipeline/with-linked-components: post: summary: Initialize a pipeline with provided components. The links for all components have to be determined by the caller. tags: - Pipeline Management description: Populates and publishes a new pipeline as specified. requestBody: content: application/json: schema: type: object additionalProperties: false properties: origin: type: string enum: - ui - system default: system pipeline: oneOf: - type: "null" - type: object properties: id: type: string format: uuid title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name sources: type: array minItems: 1 items: type: object allOf: - if: properties: type: enum: - http deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: HTTP description: Receive data from incoming HTTP requests additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 decoding: oneOf: - enum: - bytes - enum: - json - enum: - ndjson - enum: - auto ui: display_type: list errorMessage: must be equal to one of the allowed values description: The decoding method for converting frames into data events. title: Decoding Method default: auto required: - decoding - if: properties: type: enum: - splunk-hec deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Splunk HEC description: Receive Splunk logs additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - s3 deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS S3 description: Collect data from object contents in AWS S3 additionalProperties: false properties: sqs_queue_url: type: string title: SQS Queue URL description: The URL of a AWS SQS queue configured to receive S3 bucket notifications for the desired S3 buckets format: uri errorMessage: format: Must be a valid URI auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key compression: type: string title: Compression description: The compression format of the S3 objects enum: - auto - gzip - none - zstd default: auto region: type: string title: Region minLength: 1 description: The name of the source AWS region. required: - sqs_queue_url - auth - compression - region - if: properties: type: enum: - sqs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS SQS description: Collect messages from AWS SQS additionalProperties: false properties: queue_url: type: string title: Queue URL description: The URL of an AWS SQS queue format: uri errorMessage: format: Must be a valid URI maxLength: 128 auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id maxLength: 128 secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret maxLength: 128 required: - access_key_id - secret_access_key region: type: string title: Region minLength: 1 description: The name of the source AWS region. maxLength: 128 required: - queue_url - auth - region - if: properties: type: enum: - demo-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Demo Logs description: Generate log messages for testing and demonstration. additionalProperties: type: array title: Log Lines description: A list of user-provided log lines that will be sent through the pipeline infinitely. A maximum of 25 lines is allowed. The lines can be either plain text or ndjson. If ndjson is detected, the encoding for the source will be set to JSON, and lines that are not valid will show up as a visible component error. Otherwise, text encoding will be used. minItems: 1 maxItems: 25 items: type: string minLength: 1 maxLength: 4096 title: Line description: A single line of log data, either stringified JSON or plain text. ui: display_type: user-provided-lines allOf: - if: properties: format: enum: - user_provided required: - format then: properties: lines: type: array title: Log Lines description: A list of user-provided log lines that will be sent through the pipeline infinitely. A maximum of 25 lines is allowed. The lines can be either plain text or ndjson. If ndjson is detected, the encoding for the source will be set to JSON, and lines that are not valid will show up as a visible component error. Otherwise, text encoding will be used. minItems: 1 maxItems: 25 items: type: string minLength: 1 maxLength: 4096 title: Line description: A single line of log data, either stringified JSON or plain text. ui: display_type: user-provided-lines required: - lines - allOf: - errorMessage: "`lines` is only valid when `format` is `user_provided`" if: not: type: object properties: format: enum: - user_provided required: - format then: not: type: object properties: lines: type: array title: Log Lines description: A list of user-provided log lines that will be sent through the pipeline infinitely. A maximum of 25 lines is allowed. The lines can be either plain text or ndjson. If ndjson is detected, the encoding for the source will be set to JSON, and lines that are not valid will show up as a visible component error. Otherwise, text encoding will be used. minItems: 1 maxItems: 25 items: type: string minLength: 1 maxLength: 4096 title: Line description: A single line of log data, either stringified JSON or plain text. ui: display_type: user-provided-lines required: - lines properties: interval_ms: type: number title: Interval description: The amount of time to pause between each demo log. Fractions of a second are accepted to increase throughput. e.g. 500 ms == 120 lines per minute default: 1000 minimum: 500 maximum: 3600000 ui: display_type: milliseconds-to-human-readable format: title: Format description: The format of the log messages. All of the data is generated. ui: display_type: list errorMessage: must be one of the available types of demo data oneOf: - label: HTTP JSON description: HTTP request data in JSON format. enum: - json - label: Sensor JSON description: Data from environment sensors in JSON format. enum: - env_sensor - label: Financial JSON description: Financial data in JSON format. enum: - financial - label: Nginx Requests description: Nginx logs in text format. enum: - nginx - label: Apache Requests description: Apache request logs in text format. enum: - apache_common - label: Apache Errors description: Apache error logs in text format. enum: - apache_error - label: BSD Syslog description: Mock BSD syslog entries in rfc3164 text format. enum: - bsd_syslog - label: Syslog description: Mock syslog entries in rfc5424 text format. enum: - syslog - label: HTTP Metrics description: HTTP-based metrics in the Mezmo metrics JSON format. enum: - http_metrics - label: Generic Metrics description: Generic metrics in the Mezmo metrics JSON format enum: - generic_metrics - label: User-Provided Logs description: Custom logs provided by the user. These can either be plain text or serialized JSON. enum: - user_provided - label: Infrastructure Logs description: Infrastructure logs consisting of sysdig, nginx, HDFS and spark logs enum: - infrastructure - label: Kubernetes Logs description: Kubernetes logs in JSON format enum: - kubernetes default: json required: - format - if: properties: type: enum: - kinesis-firehose deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Kinesis Firehose description: Receive Kinesis Firehose data additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 format: type: string enum: - json - text default: json title: Data Format description: This specifies what the data format will be after it is base64 decoded. If it is JSON, it will be automatically parsed. required: - format - if: properties: type: enum: - fluent deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Fluent (Fluentd, Fluent Bit) description: Receive data from Fluentd or Fluent Bit additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 decoding: type: string description: The decoding method for converting frames into data events. enum: - bytes - json - ndjson title: Decoding Method default: json required: - decoding - if: properties: type: enum: - log-analysis deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Log Analysis description: Receive data directly from your Mezmo Log Analysis account additionalProperties: false - if: properties: type: enum: - log-analysis-ingestion deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Log Analysis Ingestion description: Receive unparsed source data directly from your Mezmo Log Analysis account. additionalProperties: type: string - if: properties: type: enum: - logstash deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Logstash description: Receive Logstash data additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 format: type: string enum: - json - text default: json title: Data Format description: The format of the logstash data required: - format - if: properties: type: enum: - mezmo-agent deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Agent description: Receive logs from the Mezmo Agent additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - mezmo-datadog-source deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Datadog Agent description: Send logs and metrics data directly from an installed datadog agent additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - pipeline-data-restoration deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Pipeline Data Restoration description: Receive data directly from your Mezmo Pipline Data Restoration additionalProperties: false - if: properties: type: enum: - webhook deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Webhook (WebSub) description: Receive data from incoming webhooks using the WebSub protocol. additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - kafka deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Kafka Source description: Collect events from Kafka compatible brokers additionalProperties: type: string properties: brokers: type: array title: Brokers uniqueItems: true minItems: 1 description: An array of key/val objects for each host/port of a Kafka broker. items: type: object title: Host/Port description: The host and port for the Kafka broker. additionalProperties: false properties: host: title: Host description: The hostname or IP address of the broker. anyOf: - type: string format: ipv4 errorMessage: format: Must be a valid IPv4 address - type: string format: hostname maxLength: 512 errorMessage: format: Must be a valid hostname port: type: number minimum: 1 maximum: 65535 title: Port description: The port that the broker listens on. required: - host - port topics: type: array title: Topics items: type: string title: Topic minLength: 1 maxLength: 256 description: The Kafka topic name. minItems: 1 description: List of Kafka topics names to read events from. group_id: type: string title: Consumer Group minLength: 1 description: The consumer group name to be used to consume events from Kafka. tls_enabled: type: boolean title: TLS enabled description: Whether or not to require TLS for outgoing connections. default: true sasl_enabled: type: boolean title: SASL/SCRAM Enabled description: Enable SASL/SCRAM authentication with Kafka default: true decoding_codec: type: string enum: - bytes - json title: Decoding description: Configures how events are decoded from raw bytes default: bytes if: properties: sasl_enabled: enum: - true then: properties: sasl_mechanism: type: string enum: - SCRAM-SHA-512 - SCRAM-SHA-256 - PLAIN title: SASL description: The Kafka SASL/SCRAM mechanism default: PLAIN sasl_username: type: string title: Username description: The SASL/SCRAM username minLength: 1 sasl_password: type: string title: Password description: The SASL/SCRAM username ui: display_type: secret minLength: 1 required: - sasl_mechanism - sasl_username - sasl_password required: - brokers - topics - group_id - decoding_codec - if: properties: type: enum: - azure-event-hub deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Azure Event Hub Source description: Collect events from Azure Event Hub using Azure Kafka Endpoint additionalProperties: false properties: connection_string: type: string title: Connection String minLength: 1 maxLength: 512 description: The Connection String as it appears in hub consumer SAS Policy ui: display_type: secret namespace: type: string title: Namespace minLength: 1 maxLength: 256 description: The Event Hub Namespace topics: type: array title: Event Hub Names items: type: string title: Name minLength: 1 maxLength: 256 description: Azure Event Hub instance name. minItems: 1 description: The list of Azure Event Hub name(s) to read events from. group_id: type: string title: Consumer Group minLength: 1 maxLength: 256 description: The consumer group name that this consumer belongs to. decoding_codec: type: string enum: - bytes - json title: Decoding description: Configures how events are decoded from raw bytes default: bytes required: - connection_string - namespace - topics - group_id - decoding_codec - if: properties: type: enum: - prometheus-remote-write deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Prometheus Remote Write description: Receive prometheus remote write protobuf data from HTTP requests additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - open-telemetry-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry Metrics description: Receive metrics from OpenTelemetry over HTTP additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - open-telemetry-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry Logs description: Receive logs from OpenTelemetry over HTTP additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 - if: properties: type: enum: - open-telemetry-traces deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry Traces description: Receive traces from OpenTelemetry over HTTP additionalProperties: false properties: capture_metadata: type: boolean title: Capture Metadata description: >- Enable the inclusion of all http headers and query string parameters that were sent from the source. ui: feature_flag: pipeline-capture-metadata display_type: hidden default: true auto_parse: type: object title: Configure automated parsing additionalProperties: false ui: display_type: hidden properties: enabled: type: boolean default: false enabled_event_types: type: array items: type: string enum: - HTTPD_COMBINEDLOG - HTTPD_COMMONLOG - HTTPD_ERRORLOG - SYSLOG5424LINE - SYSLOGLINE - SYSLOGPAMSESSION - CRONLOG - MONGO3_LOG - NAGIOSLOGLINE - POSTGRESQL - RAILS3 - RUBY_LOGGER - REDISLOG - REDISMONLOG - S3_ACCESS_LOG - ELB_ACCESS_LOG - CLOUDFRONT_ACCESS_LOG - CATALINALOG - TOMCATLOG - SQUID3 - BIND9 - HAPROXYTCP - HAPROXYHTTP - BACULA_LOGLINE - BRO_HTTP - BRO_DNS - BRO_CONN - BRO_FILES - NETSCREENSESSIONLOG - CISCO_TAGGED_SYSLOG - CISCOFW104001 - CISCOFW104002 - CISCOFW104003 - CISCOFW104004 - CISCOFW105003 - CISCOFW105004 - CISCOFW105005 - CISCOFW105008 - CISCOFW105009 - CISCOFW106001 - CISCOFW106006_106007_106010 - CISCOFW106014 - CISCOFW106015 - CISCOFW106021 - CISCOFW106023 - CISCOFW106100_2_3 - CISCOFW106100 - CISCOFW304001 - CISCOFW110002 - CISCOFW302010 - CISCOFW302013_302014_302015_302016 - CISCOFW302020_302021 - CISCOFW305011 - CISCOFW313001_313004_313008 - CISCOFW313005 - CISCOFW321001 - CISCOFW402117 - CISCOFW402119 - CISCOFW419001 - CISCOFW419002 - CISCOFW500004 - CISCOFW602303_602304 - CISCOFW710001_710002_710003_710005_710006 - CISCOFW713172 - CISCOFW733100 - SHOREWALL - SFW2 properties: title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description type: type: string enum: - http - splunk-hec - s3 - sqs - demo-logs - kinesis-firehose - fluent - log-analysis - log-analysis-ingestion - logstash - mezmo-agent - mezmo-datadog-source - pipeline-data-restoration - webhook - kafka - azure-event-hub - prometheus-remote-write - open-telemetry-metrics - open-telemetry-logs - open-telemetry-traces id: type: string pattern: ^[a-zA-Z0-9-_]+$ user_config: type: object required: - id - user_config transforms: type: array minItems: 1 items: type: object allOf: - if: properties: type: enum: - filter deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Filter description: Define condition(s) to include or exclude events from the pipeline additionalProperties: false properties: action: title: Action description: How to handle events matching this criteria oneOf: - label: Allow events matching this criteria enum: - allow - label: Drop events matching this criteria enum: - drop default: allow ui: display_type: radio-group conditional: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine required: - action - conditional - if: properties: type: enum: - sample deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sample description: Sample data at a given rate, retaining only a subset of data events for further processing. additionalProperties: false properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 events will be forwarded and the rest will be dropped minimum: 2 maximum: 10000 default: 10 always_include: title: Always Include description: Certain events may always want to be included (not sampled), regardless of the sample rate. Events matching this criteria will always show up in the results. additionalProperties: type: - string - number dependencies: field: - str_operator str_operator: - field allOf: - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: errorMessage: pattern: Value must be a number type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 pattern: ^\s*-?(\d*\.)?\d+\s*$ required: - value - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix - percent_change_greater - percent_change_greater_or_equal - percent_change_less - percent_change_less_or_equal - value_change_greater - value_change_greater_or_equal - value_change_less - value_change_less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 required: - value - if: title: Operator properties: str_operator: type: string enum: - contains - equal - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix ui: display_type: list-operators title: Operator required: - str_operator then: title: Case sensitive properties: case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive comparison? default: true properties: field: type: string title: Field description: The JSON field to use in a condition to always include in sampling pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - is_metric - is_array - is_boolean - is_empty - is_null - is_number - is_object - is_string - exists - ends_with - starts_with - contains_token_term - contains_token_prefix - regex_match ui: display_type: list-operators title: Operator description: The comparison operator negate: type: boolean default: false title: Negate description: Negate the comparison expression required: - rate - if: properties: type: enum: - drop-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Remove Fields description: Remove fields additionalProperties: false properties: fields: type: array title: Fields description: A list of fields to be removed items: type: string title: Field pattern: ^(?:metadata|\.|(?:metadata|message)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields). Note that dropping an event's message or timestamp is not allowed minItems: 1 required: - fields - if: properties: type: enum: - parse deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Parse description: Parse a specified field using the chosen parser additionalProperties: type: object errorMessage: properties: target_field: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) allOf: - if: properties: parser: enum: - parse_key_value then: properties: options: $ref: "#key-value-parser-options" - if: properties: parser: enum: - parse_timestamp then: properties: options: $ref: "#timestamp-parser-options" required: - options - if: properties: parser: enum: - parse_grok then: properties: options: $ref: "#grok-pattern-parser-options" required: - options - if: properties: parser: enum: - parse_regex then: properties: options: allOf: - $ref: "#regex-parser-options" ui: display_type: regex-pattern required: - options - if: properties: parser: enum: - parse_apache_log then: properties: options: $ref: "#apache-parser-options" required: - options - if: properties: parser: enum: - parse_nginx_log then: properties: options: $ref: "#nginx-parser-options" required: - options - if: properties: parser: enum: - parse_cef then: properties: options: $ref: "#cef-parser-options" - if: properties: parser: enum: - parse_csv then: properties: options: $ref: "#csv-parser-options" properties: field: type: string title: Field description: The JSON field whose value should be parsed. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: oneOf: - type: "null" - type: string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference title: Target field description: The field into which the parsed value should be inserted. Leave blank to insert the parsed data into the original field. default: null parser: oneOf: - label: AWS Cloudfront Access Log group: Line Parsers ui: {} enum: - aws_cloudfront_access_log - label: AWS Cloudwatch Log group: Line Parsers ui: {} enum: - parse_aws_cloudwatch_log_subscription_message - label: AWS Load Balancer Log group: Line Parsers ui: {} enum: - parse_aws_alb_log - label: AWS S3 Access Log group: Line Parsers ui: {} enum: - aws_s3_access_log - label: Apache Log group: Line Parsers ui: {} enum: - parse_apache_log - label: CSV group: Line Parsers ui: {} enum: - parse_csv - label: Common Event Format (CEF) Log group: Line Parsers ui: {} enum: - parse_cef - label: Common Log group: Line Parsers ui: {} enum: - parse_common_log - label: Elasticsearch Log group: Line Parsers ui: {} enum: - elasticsearch_log - label: Golang Log group: Line Parsers ui: {} enum: - golang_log - label: Google Log group: Line Parsers ui: {} enum: - parse_glog - label: Grok Pattern group: Line Parsers ui: {} enum: - parse_grok - label: Integer group: Field Parsers ui: {} enum: - parse_int - label: JSON group: Line Parsers ui: {} enum: - parse_json - label: Key/Value group: Line Parsers ui: {} enum: - parse_key_value - label: Kubernetes Log group: Line Parsers ui: {} enum: - parse_klog - label: Linux Authorization group: Line Parsers ui: {} enum: - parse_linux_authorization - label: Mongo Log group: Line Parsers ui: {} enum: - mongo_log - label: Mongo3 Log group: Line Parsers ui: {} enum: - mongo3_log - label: Nginx Log group: Line Parsers ui: {} enum: - parse_nginx_log - label: PSQL Log group: Line Parsers ui: {} enum: - psql_log - label: Query String group: Field Parsers ui: {} enum: - parse_query_string - label: Regular Expression group: Line Parsers ui: feature_flag: pipeline-parser-regex enum: - parse_regex - label: Syslog group: Line Parsers ui: {} enum: - parse_syslog - label: Timestamp group: Field Parsers ui: {} enum: - parse_timestamp - label: Tokens group: Field Parsers ui: {} enum: - parse_tokens - label: URL group: Field Parsers ui: {} enum: - parse_url - label: User Agent group: Field Parsers ui: {} enum: - parse_user_agent - label: VPC Flow Log group: Line Parsers ui: {} description: Parse VPC logs using default format for fields from v2 to v4 enum: - parse_aws_vpc_flow_log ui: display_type: list errorMessage: must be one of allowed parsers title: Parser description: The kind of parser to use against the input value from "field". required: - field - parser - if: properties: type: enum: - parse-sequentially deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Parse Sequentially description: Parse fields using one or more parsers in order, sending out the event on the first successful match. additionalProperties: false errorMessage: properties: target_field: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) properties: field: type: string title: Field description: The field whose value should be parsed. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: oneOf: - type: "null" - type: string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference title: Target field description: The field into which the parsed value should be inserted. Leave blank to insert the parsed data into the original field. default: null parsers: type: array title: List of parsers description: The list of parsers to use in order against the input valuefrom "field", short-circuiting on the first successful match. items: type: object allOf: - if: properties: parser: enum: - parse_key_value then: properties: options: $ref: "#key-value-parser-options" - if: properties: parser: enum: - parse_timestamp then: properties: options: $ref: "#timestamp-parser-options" required: - options - if: properties: parser: enum: - parse_grok then: properties: options: $ref: "#grok-pattern-parser-options" required: - options - if: properties: parser: enum: - parse_regex then: properties: options: allOf: - $ref: "#regex-parser-options" ui: display_type: regex-pattern required: - options - if: properties: parser: enum: - parse_apache_log then: properties: options: $ref: "#apache-parser-options" required: - options - if: properties: parser: enum: - parse_nginx_log then: properties: options: $ref: "#nginx-parser-options" required: - options - if: properties: parser: enum: - parse_cef then: properties: options: $ref: "#cef-parser-options" - if: properties: parser: enum: - parse_csv then: properties: options: $ref: "#csv-parser-options" properties: label: oneOf: - type: "null" - type: string maxLength: 20 title: Title description: An arbitrary name you choose to identify the results of this processor for use when connecting them to other components. parser: oneOf: - label: AWS Cloudfront Access Log group: Line Parsers ui: {} enum: - aws_cloudfront_access_log - label: AWS Cloudwatch Log group: Line Parsers ui: {} enum: - parse_aws_cloudwatch_log_subscription_message - label: AWS Load Balancer Log group: Line Parsers ui: {} enum: - parse_aws_alb_log - label: AWS S3 Access Log group: Line Parsers ui: {} enum: - aws_s3_access_log - label: Apache Log group: Line Parsers ui: {} enum: - parse_apache_log - label: CSV group: Line Parsers ui: {} enum: - parse_csv - label: Common Event Format (CEF) Log group: Line Parsers ui: {} enum: - parse_cef - label: Common Log group: Line Parsers ui: {} enum: - parse_common_log - label: Elasticsearch Log group: Line Parsers ui: {} enum: - elasticsearch_log - label: Golang Log group: Line Parsers ui: {} enum: - golang_log - label: Google Log group: Line Parsers ui: {} enum: - parse_glog - label: Grok Pattern group: Line Parsers ui: {} enum: - parse_grok - label: Integer group: Field Parsers ui: {} enum: - parse_int - label: JSON group: Line Parsers ui: {} enum: - parse_json - label: Key/Value group: Line Parsers ui: {} enum: - parse_key_value - label: Kubernetes Log group: Line Parsers ui: {} enum: - parse_klog - label: Linux Authorization group: Line Parsers ui: {} enum: - parse_linux_authorization - label: Mongo Log group: Line Parsers ui: {} enum: - mongo_log - label: Mongo3 Log group: Line Parsers ui: {} enum: - mongo3_log - label: Nginx Log group: Line Parsers ui: {} enum: - parse_nginx_log - label: PSQL Log group: Line Parsers ui: {} enum: - psql_log - label: Query String group: Field Parsers ui: {} enum: - parse_query_string - label: Regular Expression group: Line Parsers ui: feature_flag: pipeline-parser-regex enum: - parse_regex - label: Syslog group: Line Parsers ui: {} enum: - parse_syslog - label: Timestamp group: Field Parsers ui: {} enum: - parse_timestamp - label: Tokens group: Field Parsers ui: {} enum: - parse_tokens - label: URL group: Field Parsers ui: {} enum: - parse_url - label: User Agent group: Field Parsers ui: {} enum: - parse_user_agent - label: VPC Flow Log group: Line Parsers ui: {} description: Parse VPC logs using default format for fields from v2 to v4 enum: - parse_aws_vpc_flow_log ui: display_type: list errorMessage: must be one of allowed parsers title: Parser description: The kind of parser to use against the input value from "field". _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. default: "" required: - parser - _output_name additionalProperties: type: object minItems: 1 ui: display_type: multi-output-map unique_parser_and_options: true errorMessage: uniqueItemProperties: parser can not be defined more than once required: - field - parsers - if: properties: type: enum: - route deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Route description: Route data based on whether it matches (or doesn't match) logical comparisons additionalProperties: false properties: conditionals: type: array minItems: 1 title: Conditionals description: A list of complete conditionals, including expressions and label items: type: object additionalProperties: false title: Conditional description: A list of conditions and a label properties: conditional: $ref: "#condition-group" label: type: string minLength: 1 maxLength: 255 title: Label description: An arbitrary name you choose to identify the results of this processor for use when connecting them to other components. _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. default: "" required: - conditional - label - _output_name required: - conditionals - if: properties: type: enum: - stringify deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Stringify description: Stringify messages as JSON additionalProperties: false - if: properties: type: enum: - compact-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Compact Fields description: Remove empty values from a list of fields additionalProperties: false properties: fields: type: array title: Fields description: A list of fields to remove empty values from items: type: string title: Field description: Field to remove empty value from pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference minItems: 1 options: type: object additionalProperties: false title: Compact Options description: Additional options for removing empty values from fields properties: compact_array: type: boolean title: Compact Array description: Remove empty arrays from a field default: true compact_object: type: boolean title: Compact Object description: Remove empty objects from a field default: true required: - fields - if: properties: type: enum: - flatten-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Flatten Fields description: Flattens the object or array value of a field into a single level representation additionalProperties: false properties: fields: type: array title: Fields description: Define specific nested objects you wish to flatten. This will not flatten the entire object, just the specified fields. Leave blank to flatten the entire object items: type: string title: Field description: Field to flatten pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference options: type: object additionalProperties: false title: Flatten Options description: Additional options for flattening fields properties: delimiter: type: string default: _ title: Delimiter description: When flattening, the nested field labels will be appended to the higher level field names along with a delimiter to allow understanding of the original structure. The default is an underscore (_) to ensure no collisions with the . notation used for field identifiers minLength: 1 maxLength: 1 - if: properties: type: enum: - dedupe deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Dedupe description: Dedupe (remove duplicates) from the data stream additionalProperties: false properties: number_of_events: type: number description: Number of events to compare across title: Number of Events minimum: 2 maximum: 5000 default: 5000 comparison_type: type: string enum: - Ignore - Match description: "Match: Only compares across the fields which are specified by the user.Ignore: Compares everything but the fields specified by the user." title: Comparison Type default: Match fields: type: array title: Fields description: A list of fields on which to base deduping items: type: string title: Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference minItems: 1 required: - number_of_events - comparison_type - fields - if: properties: type: enum: - encrypt-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Encrypt Field description: Encrypts the value of the provided field additionalProperties: false allOf: - if: properties: algorithm: enum: - AES-256-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key properties: field: type: string title: Field description: Field to encrypt. The value of the field must be a primitive (string, number, boolean) pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference algorithm: oneOf: - label: AES-256-CFB (key = 32 characters, iv = 16 characters) description: AES-256-CFB (key = 32 characters, iv = 16 characters) enum: - AES-256-CFB - label: AES-192-CFB (key = 24 characters, iv = 16 characters) description: AES-192-CFB (key = 24 characters, iv = 16 characters) enum: - AES-192-CFB - label: AES-128-CFB (key = 16 characters, iv = 16 characters) description: AES-128-CFB (key = 16 characters, iv = 16 characters) enum: - AES-128-CFB - label: AES-256-OFB (key = 32 characters, iv = 16 characters) description: AES-256-OFB (key = 32 characters, iv = 16 characters) enum: - AES-256-OFB - label: AES-192-OFB (key = 24 characters, iv = 16 characters) description: AES-192-OFB (key = 24 characters, iv = 16 characters) enum: - AES-192-OFB - label: AES-128-OFB (key = 16 characters, iv = 16 characters) description: AES-128-OFB (key = 16 characters, iv = 16 characters) enum: - AES-128-OFB - label: AES-256-CTR (key = 32 characters, iv = 16 characters) description: AES-256-CTR (key = 32 characters, iv = 16 characters) enum: - AES-256-CTR - label: AES-192-CTR (key = 24 characters, iv = 16 characters) description: AES-192-CTR (key = 24 characters, iv = 16 characters) enum: - AES-192-CTR - label: AES-128-CTR (key = 16 characters, iv = 16 characters) description: AES-128-CTR (key = 16 characters, iv = 16 characters) enum: - AES-128-CTR - label: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-PKCS7 - label: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-PKCS7 - label: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-PKCS7 - label: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ANSIX923 - label: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ANSIX923 - label: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ANSIX923 - label: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO7816 - label: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO7816 - label: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO7816 - label: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO10126 - label: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO10126 - label: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO10126 ui: display_type: list title: Encryption algorithm description: The encryption algorithm to use key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 32 iv_field: type: string title: Initialization vector (IV) field description: The field in which to store the generated initialization vector, IV. Each encrypted value will have a unique IV pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference encode_raw_bytes: type: boolean title: Encode encrypted field and IV as Base 64 text description: Encode the encrypted value and generated initialization vector as Base64 text default: true required: - field - algorithm - key - iv_field - if: properties: type: enum: - decrypt-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Decrypt Field additionalProperties: false description: Decrypts the value of the provided field allOf: - if: properties: algorithm: enum: - AES-256-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key properties: field: type: string title: Field description: Field to decrypt. The value of the field must be a string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference algorithm: oneOf: - label: AES-256-CFB (key = 32 characters, iv = 16 characters) description: AES-256-CFB (key = 32 characters, iv = 16 characters) enum: - AES-256-CFB - label: AES-192-CFB (key = 24 characters, iv = 16 characters) description: AES-192-CFB (key = 24 characters, iv = 16 characters) enum: - AES-192-CFB - label: AES-128-CFB (key = 16 characters, iv = 16 characters) description: AES-128-CFB (key = 16 characters, iv = 16 characters) enum: - AES-128-CFB - label: AES-256-OFB (key = 32 characters, iv = 16 characters) description: AES-256-OFB (key = 32 characters, iv = 16 characters) enum: - AES-256-OFB - label: AES-192-OFB (key = 24 characters, iv = 16 characters) description: AES-192-OFB (key = 24 characters, iv = 16 characters) enum: - AES-192-OFB - label: AES-128-OFB (key = 16 characters, iv = 16 characters) description: AES-128-OFB (key = 16 characters, iv = 16 characters) enum: - AES-128-OFB - label: AES-256-CTR (key = 32 characters, iv = 16 characters) description: AES-256-CTR (key = 32 characters, iv = 16 characters) enum: - AES-256-CTR - label: AES-192-CTR (key = 24 characters, iv = 16 characters) description: AES-192-CTR (key = 24 characters, iv = 16 characters) enum: - AES-192-CTR - label: AES-128-CTR (key = 16 characters, iv = 16 characters) description: AES-128-CTR (key = 16 characters, iv = 16 characters) enum: - AES-128-CTR - label: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-PKCS7 - label: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-PKCS7 - label: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-PKCS7 - label: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ANSIX923 - label: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ANSIX923 - label: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ANSIX923 - label: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO7816 - label: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO7816 - label: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO7816 - label: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO10126 - label: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO10126 - label: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO10126 ui: display_type: list title: Decryption algorithm description: The algorithm with which the data was encrypted key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 32 iv_field: type: string title: Initialization vector (IV) field description: The field from which to read the initialization vector, IV pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference decode_raw_bytes: type: boolean title: Decode the encrypted value and IV as Base 64 text description: Decode the encrypted value and initialization vector as Base64 text default: true required: - field - algorithm - key - iv_field - if: properties: type: enum: - js-script deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Script Execution description: Use JavaScript to reshape and transform your data. You can combine multiple actions like filtering, dropping, mapping, and casting inside of a single js script. additionalProperties: true properties: script: type: string title: Script ui: display_type: code-editor options: language: js description: The JavaScript representation of the event transformation. minLength: 1 maxLength: 10000 default: > // Modify the event using a subset of the JavaScript language. // The function must return the modified event function processEvent(message, metadata, timestamp, annotations) { return message } - if: properties: type: enum: - unroll deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Unroll description: Takes an array of events and emits them all as individual events. additionalProperties: false properties: field: type: string title: Field description: The JSON field name that contains an array of events pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference values_only: type: boolean title: Values only? description: When enabled, the values from the specified array field will be emitted as new events. Otherwise, the original event will be duplicated for each value in the array field, with the unrolled value present in the field specified. default: true required: - field - if: properties: type: enum: - vrl deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: VRL description: Modify your observability data using Vector Remap Language (VRL) additionalProperties: false properties: source: type: string title: VRL Source description: VRL code to execute for each event ui: display_type: code-editor options: language: vrl minLength: 1 default: .message = .message required: - source - if: properties: type: enum: - reduce deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Reduce description: Combine multiple events over time into one based on a set of criteria additionalProperties: false properties: duration_ms: type: number maximum: 7200000 title: Duration description: The amount of time (in milliseconds) to allow streaming events to accumulate into a single "reduced" event. The process repeats indefinitely, or until an "ends when" condition is satisfied. ui: display_type: milliseconds-to-human-readable default: 30000 max_events: type: number minimum: 1 maximum: 100000 title: Max Events description: The maximum number of events that can be included in a time window (specified by Duration above). The reduce operation will stop once it has reached this number of events, regardless of whether the Duration seconds have elapsed. group_by: type: array minItems: 1 title: Group By Field Path items: type: string title: Field description: The dot-notated property path errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ description: Before reducing, group events based on matching data from each of these field paths. Supports nesting via dot-notation. date_formats: type: array title: Date Formats description: Describes which root-level properties are dates, and their expected format. Dot-notation is supported, but nested field lookup paths will be an error. minItems: 1 items: type: object additionalProperties: false properties: field: type: string title: Date Field Path description: Specifies a root-level path property that contains a date value. maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ format: type: string title: Date Format description: The template describing the date format format: strftime_datetime errorMessage: must be a valid strftime datetime format maxLength: 200 required: - field - format merge_strategies: type: array title: Merge Strategy per Field minItems: 1 description: Specify merge strategies for individual root-level properties. Dot-notation is supported, but nested field lookup paths will be an error. items: type: object additionalProperties: false properties: field: type: string title: Field Path description: This is a root-level path property to apply a merge strategy to its value maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ strategy: oneOf: - label: Array description: Append each value onto an array enum: - array - label: Smallest array description: Keep the smallest array within each reduce time window enum: - shortest_array - label: Largest array description: Keep the largest array within each reduce time window enum: - longest_array - label: Unique Array description: All unique values go into a flattened array enum: - flat_unique - label: Join with space description: Concatenate into a single space-delimited string enum: - concat - label: Join with newline description: Concatenate into a single newline-delimited string enum: - concat_newline - label: Join description: Concatenate into a single string without using a delimiter enum: - concat_raw - label: Keep first description: Keep only the first value and discard the rest enum: - discard - label: Keep last description: Keep only the last non-null value enum: - retain - label: Min value description: Keep the minimum numeric value within each reduce time window enum: - min - label: Max value description: Keep the maximum numeric value within each reduce time window enum: - max - label: Sum description: Add numeric values together enum: - sum ui: display_type: list errorMessage: must be one of the allowed merge strategies title: Merge Strategy description: The merge strategy to be used for the specified property required: - field - strategy flush_condition: title: Event Flush Condition description: Force accumulated event reduction to flush the result when a conditional expression evaluates to true on an inbound event. allOf: - type: object properties: when: type: string enum: - none - starts_when - ends_when default: none ui: display_type: list errorMessage: "" title: Flush When required: - when - if: properties: when: not: enum: - none then: properties: conditional: allOf: - $ref: "#condition-group" ui: display_type: rules-engine required: - conditional - if: properties: type: enum: - map-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Map Fields description: Maps data from one field to another, either by moving or copying. additionalProperties: false properties: mappings: type: array title: Mappings description: A list of field mappings. Mappings are applied in the order they are defined. minItems: 1 ui: display_type: sortable items: type: object properties: source_field: type: string title: Source field description: The field to copy data from pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: type: string title: Target field description: The field to copy data into pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference drop_source: type: boolean title: Drop source? description: When enabled, the source field is dropped after the data is copied to the target field. Otherwise, it is preserved. default: false overwrite_target: type: boolean title: Overwrite target? description: When enabled, any existing data in the target field is overwritten. Otherwise, the target field will be preserved and this mapping will have no effect. default: false required: - source_field - target_field - drop_source - overwrite_target if: properties: drop_source: enum: - true required: - drop_source then: properties: source_field: type: string title: Source field description: The field to copy data from pattern: ^(?:metadata|\.|(?:metadata|message)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields). Note that dropping an event's message or timestamp is not allowed required: - source_field required: - mappings - if: properties: type: enum: - metrics-tag-cardinality-limit deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Tag Cardinality Limit (Metrics) description: Limits the cardinality of metric events by either dropping events or tags that exceed a specified value limit additionalProperties: false allOf: - if: title: Mode properties: mode: type: string enum: - exact then: title: Value Limit properties: value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags minimum: 1 maximum: 500 required: - value_limit - if: title: Mode properties: mode: type: string enum: - probabilistic then: title: Value Limit properties: value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags minimum: 1 maximum: 5000 required: - value_limit properties: tags: type: array title: Tags description: A list of tags to apply cardinality limits. If none are provided, all tags will be considered. items: type: string title: Tag minLength: 1 maxLength: 100 minItems: 0 maxItems: 10 exclude_tags: type: array title: Exclude Tags description: A list of tags to explicitly exclude from cardinality limits. items: type: string title: Tag minLength: 1 maxLength: 100 minItems: 0 maxItems: 10 action: type: string title: Action description: The action to take when a tag's cardinality exceeds the value limit enum: - drop_tag - drop_event default: drop_event value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags mode: type: string title: Mode description: The method to used to reduce tag value cardinality enum: - exact - probabilistic default: exact - if: properties: type: enum: - aggregate deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Aggregate description: Aggregates multiple events into a single event using either a tumbling interval window or a sliding interval window. additionalProperties: false properties: group_by: type: array minItems: 1 maxItems: 100 title: Group By Field Path description: When aggregating, group events based on matching values from each of these field paths. Supports nesting via dot-notation. items: type: string title: Field description: The dot-notated property path errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ ui: default_hint: - .name - .namespace - .tags evaluate: additionalProperties: type: string title: Evaluate description: How events should be aggregated within an interval. allOf: - if: properties: operation: enum: - CUSTOM then: properties: script: type: string title: Script ui: display_type: code-editor options: language: js description: The JavaScript representation of an event aggregation. minLength: 1 maxLength: 2048 default: > // Perform custom aggregation of the current event into the accumulated event. // The accumulated event is persisted across the entire aggregation window. // The function must return the accumulated event, which will be emitted when // the aggregation window elapses. function aggregateEvent(accum, event, metadata, timestamp, annotations) { return accum } required: - script properties: operation: title: Operation oneOf: - label: add description: Adds new values together if they are of the same type. For counters and gauges the values are added together. For sets, this is a union of the elements. For distribution, the samples are concatenated. For histograms, values in the same bucket will be added together. enum: - DEFAULT - label: sum description: Values are added together. Works for the GAUGE and COUNT metric types. enum: - SUM - label: minimum description: Get the minimum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MIN - label: maximum description: Get the maximum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MAX - label: average description: Values are averaged together. Works for the GAUGE and COUNT metric types. enum: - AVG - label: set intersection description: The aggregate produces a set of unique values. enum: - SET_INTERSECTION - label: distribution concatenation description: The aggregate produces a list of values combined together. enum: - DIST_CONCAT - label: custom description: Use a custom script to perform aggregation. enum: - CUSTOM ui: display_type: list required: - operation window: default: {} additionalProperties: type: integer title: Window Type description: Sliding windows can overlap, whereas tumbling windows are disjoint. For example, a tumbling window has a fixed time span and any events that fall within the "interval" will be used in the aggregate. Whereas in a sliding window, the aggregation occurs every "window duration" seconds after an event is encountered. allOf: - if: properties: type: enum: - sliding required: - type then: properties: window_min: type: integer title: Slide Duration (seconds) description: The amount of time to wait before creating a new sliding window for future events. minimum: 1 maximum: 200 default: 5 required: - window_min properties: type: type: string title: Type enum: - tumbling - sliding default: tumbling interval: type: integer title: Interval (seconds) description: The interval over which events are aggregated in seconds. default: 5 minimum: 1 maximum: 90000 required: - type - interval event_timestamp: type: string title: Event Timestamp description: The path to a field on the event that contains a epoch timestamp value. If an event does not have a timestamp field, events will be associated to the wall clock value when the event is processed. default: timestamp maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ conditional: ui: display_type: rules-engine title: Event Flush Condition description: Force accumulated event reduction to flush the result when a conditional expression evaluates to true on an inbound event. allOf: - $ref: "#condition-group" required: - evaluate - window - if: properties: type: enum: - event-to-metric deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Event to Metric description: Allows conversion between arbitrary events and a Metric. additionalProperties: false properties: metric_name: type: string title: Metric Name description: The machine name of the metric to emit. minLength: 1 maxLength: 128 pattern: ^[a-zA-Z][a-zA-Z0-9_:]*$ errorMessage: must be a valid metric name metric_kind: title: Kind description: The kind of metric to emit, Absolute or Incremental. Absolute metrics represent a complete value, and will generally replace an existing value for the metric in the target destination. Incremental metrics represent an additive value which is aggregated in the target destination to produce a new value. oneOf: - label: Incremental enum: - incremental - label: Absolute enum: - absolute default: incremental ui: display_type: radio-group metric_type: title: Type oneOf: - label: Counter enum: - counter - label: Sum enum: - sum - label: Gauge enum: - gauge default: counter ui: display_type: list value: title: Value description: The value of this metric. additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value then: title: Value properties: value: type: number title: Value description: The value. required: - value properties: value_type: title: Value type description: Either a value from an event field, or a new value input. oneOf: - label: Value from event field enum: - field - label: New value enum: - value default: field ui: display_type: radio-group value: type: - number - string required: - value_type namespace: title: Namespace description: The namespace for this metric. additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 128 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit the namespace. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: type: - string - "null" required: - value_type tags: type: array maxItems: 10 title: Tags description: A set of tags (also called labels) to apply to the metric event. items: type: object title: Tag description: A tag to apply to the metric event. allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value of this tag. Note that fields with highly-variable values will result in high-cardinality metrics, which may impact storage or cost in downstream destinations. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference fallback: type: string title: Fallback value description: The value to use for this tag when the field value is missing. minLength: 1 maxLength: 512 required: - value - if: properties: value_type: enum: - value then: title: Value properties: value: type: string title: Value description: The value for this tag. minLength: 1 maxLength: 128 required: - value properties: name: type: string title: Name description: The tag name pattern: ^[a-zA-Z][a-zA-Z0-9_]*$ minLength: 1 maxLength: 128 errorMessage: must be a valid metric tag name value_type: title: Value type description: Either a value from an event field, or a new value input. oneOf: - label: Value from event field enum: - field - label: New value enum: - value default: field ui: display_type: radio-group value: type: string required: - name - value_type - value additionalProperties: type: string required: - metric_name - metric_kind - metric_type - value - namespace - if: properties: type: enum: - clustering deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Clustering description: Finds clusters of similar values additionalProperties: false properties: cluster_field: type: string title: Field description: Field to cluster values on pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference default: .message similarity_threshold: type: number title: Similarity threshold description: A threshold from 0.0 (not similar) to 1.0 (exactly similar) that determines how close two values need to be to be consider as being part of the same cluster default: 0.4 minimum: 0 maximum: 1 max_clusters: type: integer title: Maximum cluster description: Maximum number of clusters kept cached by the transform default: 1000 minimum: 1 maximum: 2000 max_node_depth: type: integer title: Maximum tree depth description: Maximum depth of the prefix tree default: 4 minimum: 2 maximum: 8 max_children: type: integer title: Maximum children default: 100 minimum: 2 maximum: 200 description: Maximum number of children per node of the prefix tree required: - cluster_field - if: properties: type: enum: - redact deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Redact description: Detect and anonymize sensitive data in logs and metrics additionalProperties: false properties: field: type: string title: Field description: Detect patterns in the specified field. If the field is empty, the processor will look for patterns across all the fields. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 256 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference mask_patterns: type: array title: Redact Patterns description: List of PII patterns to detect and mask minItems: 1 maxItems: 20 items: type: object additionalProperties: type: string title: Redact Pattern description: A PII pattern to detect and/or anonymize allOf: - if: properties: action: enum: - Replace required: - action then: properties: replacement: type: string title: Replacement description: "The text to replace the detected pattern. Example: 000-00-0000" minLength: 1 maxLength: 512 required: - replacement - if: properties: action: enum: - Hash required: - action then: properties: algorithm: title: Algorithm description: Hash and encode (base16 or hex) matched PII data using the specified algorithm oneOf: - label: sha2 (sha-512_256) enum: - sha2 - label: sha3 (sha3-512) enum: - sha3 ui: display_type: list required: - algorithm - if: properties: name: enum: - custom required: - name then: properties: custom_options: allOf: - type: object additionalProperties: false properties: pattern: type: string title: Pattern description: The custom pattern minLength: 1 maxLength: 4096 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching default: true multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively default: false match_newline: type: boolean title: Dot all mode description: Dot matches newline character default: false crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" default: false ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines default: false required: - pattern title: Replace Expression description: The regular expression used to find replacements. Named capture groups are supported, e.g. `(?P<key>\w+):(?P<value>.+)`. ui: display_type: regex-pattern required: - custom_options properties: name: title: Pattern description: The PII pattern to match oneOf: - label: Credit card number enum: - credit_card - label: Email address enum: - email_address - label: IPv4 address enum: - ipv4_address - label: US or Canada phone number enum: - us_canada_phone_number - label: US social security number enum: - us_social_security_number - label: Custom pattern ui: feature_flag: pipeline-parser-regex enum: - custom ui: display_type: list action: type: string title: Action description: Action to take on the detected PII data enum: - Hash - Replace - Detect Only default: Replace required: - name - action required: - mask_patterns - if: properties: type: enum: - throttle deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Throttle description: Throttle (rate-limit) events passing through this component additionalProperties: false properties: threshold: type: number minimum: 1 maximum: 10000000 title: Limit description: The number of events to allow over the time window. If a Key Field is specified, the limit will be applied to events for each unique value of that field. default: 1 window_ms: type: number minimum: 1000 maximum: 2592000000 title: Window description: The time window over which the configured limit is applied ui: display_type: milliseconds-to-human-readable default: 5000 key_field: type: string title: Key Field description: When configured, the value from this field will be used to group events into separate buckets for throttling. If left blank, or if the event does not contain a value in this field, the event will be throttled as part of the default bucket. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference exclude: allOf: - $ref: "#condition-group" title: Exclude description: When configured, events matching this condition will not be throttled. ui: display_type: rules-engine required: - threshold - window_ms - if: properties: type: enum: - set-timestamp deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Set Timestamp description: Parse a list of fields using a chosen time format. First match will override the default timestamp of the event. additionalProperties: false properties: parsers: type: array title: List of Fields and Parsers description: The list of fields and parsers to use in order of priority, short-circuiting on the first successful match. ui: display_type: multi-output-map options: sortable: false items: type: object properties: field: type: string title: Field description: The field whose value will be parsed by the chosen date pattern pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference options: $ref: "#timestamp-parser-options" required: - field - options minItems: 1 required: - parsers - if: properties: type: enum: - data-profiler deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Data Profiler description: Profile the data sent through your pipeline. The system will generate annotations such as event type and total bytes to be used downstream. It will also normalize the app/host/level/line fields before sending the event downstream. A profile of the data including total, events, event size, log composition, and message templates will be generated as well. additionalProperties: false properties: app_fields: type: array title: App Path(s) description: A list of paths to look for the app value minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: App description: The path to the app maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .app - .application - .container host_fields: type: array title: Host Path(s) description: A list of paths to look for the host value minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Host description: The path to the host maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .host - .hostname level_fields: type: array title: Level Path(s) description: A list of paths to look for the log level minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Level description: The path to the level maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .level - .log_level line_fields: type: array title: Line Path(s) description: A list of paths to look for the event line minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Line description: The path to the line maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .line - .message label_fields: type: array title: Label Path(s) description: A list of paths to look for the event labels maxItems: 10 uniqueItems: true items: type: string title: Label description: The path to the labels maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .labels field_summary: type: boolean default: false title: Field Summary description: If enabled the profiler will attempt to summarize the values in fields of events passing through the node. ui: display_type: hidden ff_default_value: pipeline-node-transform-field-summary required: - app_fields - host_fields - level_fields - line_fields - if: properties: type: enum: - replace deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Replace description: Replace parts of a string using a regular expression or static string additionalProperties: type: object allOf: - if: properties: replacement_type: enum: - regex required: - replacement_type then: properties: options: allOf: - type: object additionalProperties: false properties: pattern: type: string minLength: 1 maxLength: 4096 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching default: true multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively default: false match_newline: type: boolean title: Dot all mode description: Dot matches newline character default: false crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" default: false ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines default: false required: - pattern title: Replace Expression description: The regular expression used to find replacements. Named capture groups are supported, e.g. `(?P<key>\w+):(?P<value>.+)`. ui: display_type: regex-pattern replace_with: type: string minLength: 1 maxLength: 4096 title: Replacement String description: The replacement string can be static or optionally contain capture group references. Capture groups can be positional or named. For example, to switch the position of two strings in the source string `onetwo`, the pattern could be `(?Pone)(?P.+)`, and the replacement would be `$two$one`. To do the same thing with positional capture groups, a pattern of `(\w{3})(\w{3})`, and a replacement of `$2$1` would have the same result.

To render a literal dollar sign `$` in the replacement, escape it with `$$`. So, if you had a capture group named `price`, to prepend a dollar sign to the replacement, you would use `$$$price`. required: - options - replace_with - if: properties: replacement_type: enum: - string required: - replacement_type then: properties: string_pattern: type: string title: String Pattern minLength: 1 maxLength: 4096 description: A static, case-sensitive string, which will be searched and replaced. replace_with: type: string minLength: 1 maxLength: 4096 title: Replacement String description: The static string used in each replacement. required: - string_pattern - replace_with properties: source_field: type: string title: Source Field description: The field that contains a string for replacement to operate on. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference max_times: type: number title: Maximum Replacements description: The maximum number of replacements to be made. If not specified, all occurrences will be replaced. minimum: 1 replacement_type: title: Replacement Type description: Specifies how the source string will be searched for replacement. ui: display_type: list errorMessage: must be one of the available pattern types oneOf: - label: String description: Replace a specified string one or more times. enum: - string - label: Regular Expression description: Replace using a regular expression. enum: - regex required: - source_field - replacement_type - if: properties: type: enum: - trace-sampling deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Trace Sampling description: Performs sampling of traces per a given rate using the Trace ID as the unique identifier. If a trace is rate selected, all spans of the trace are forwarded. If the Trace ID cannot be found, the event is dropped. allOf: - if: properties: sample_type: enum: - head then: properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 unique traces will be forwarded and all others will be dropped default: 10 minimum: 2 maximum: 100000 parent_span_id_field: type: "null" conditionals: type: "null" required: - rate - if: properties: sample_type: enum: - tail then: properties: parent_span_id_field: type: string default: .parent_span_id title: Parent Span ID description: The field to use as the parent span identifier pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference conditionals: type: array minItems: 1 title: Conditionals description: A list of complete conditionals, including expressions and a sample rate to maintain items: type: object additionalProperties: false title: Conditional description: A list of conditions and a sample rate to maintain properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 unique traces will be forwarded and all others will be dropped. A `rate = 1` means keep all traces that match the condition. minimum: 1 maximum: 100000 default: 10 conditional: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. required: - rate - conditional required: - parent_span_id_field - conditionals properties: sample_type: type: string description: The type of trace sampling to apply. enum: - head title: Sample Type default: head trace_id_field: type: string default: .trace_id title: Trace ID description: The field to use as the identifier of a trace. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - sample_type - trace_id_field properties: title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description type: type: string enum: - filter - sample - drop-fields - parse - parse-sequentially - route - stringify - compact-fields - flatten-fields - dedupe - encrypt-fields - decrypt-fields - js-script - unroll - vrl - reduce - map-fields - metrics-tag-cardinality-limit - aggregate - event-to-metric - clustering - redact - throttle - set-timestamp - data-profiler - replace - trace-sampling id: type: string pattern: ^[a-zA-Z0-9-_]+$ inputs: type: array minItems: 1 items: type: string user_config: type: object required: - id - inputs - user_config sinks: type: array minItems: 1 items: type: object additionalProperties: false allOf: - if: properties: type: enum: - mezmo deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Log Analysis description: Send log data to Mezmo Log Analysis additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true mezmo_host: type: string title: Mezmo Host description: The host for your Log Analysis environment default: null minLength: 1 ingestion_key: type: string title: Ingestion Key description: Ingestion key ui: display_type: mezmo-ingestion-key minLength: 1 query: type: object title: Query Parameters default: hostname: mezmo properties: hostname: type: string title: Hostname description: Hostname string or template to extract event values to form a hostname. Depending on the type of ingestion, this value is usually set from other places so the default value is often sufficient. If the value in the event is null, empty, or doesn't exist, when using a template then it will be set to "mezmo" default: mezmo minLength: 1 ui: display_type: data-template tags: type: array title: Tags description: List of tag strings or templates to attach to logs. Template must resolve to either an array of strings, or a string. Commas are treated as tag delimiters. items: type: string title: Tag description: Tag string or template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template ip: type: string title: IP description: IP address template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template mac: type: string title: MAC description: MAC address template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template message: title: Log Construction Scheme default: scheme: Explicit field selection allOf: - if: properties: scheme: enum: - Explicit field selection then: properties: line: type: string title: Line description: Template or field reference to use as the log line. Field reference can point to an object. minLength: 1 maxLength: 1024 meta_field: type: string title: Meta Field description: Field containing the meta object for the log pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference maxLength: 512 app: type: string title: App description: App name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template file: type: string title: File description: File name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template timestamp_field: type: string title: Timestamp description: Field containing the timestamp for the log pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference maxLength: 512 env: type: string title: Env description: Environment name template to attach to logs minLength: 1 maxLength: 512 ui: display_type: data-template properties: scheme: type: string title: Scheme description: How to construct the log message. enum: - Explicit field selection - Message pass-through default: Explicit field selection required: - scheme required: - mezmo_host - ingestion_key - if: properties: type: enum: - mezmo-archive deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Archive Destination description: Store logs for future imports into mezmo pipeline or log analysis. additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true destination: title: Destination description: "" allOf: - if: properties: archive_provider: enum: - Azure then: properties: container_name: type: string title: Container Name minLength: 1 description: The name of the container for blob storage. connection_string: type: string title: Connection String minLength: 1 description: A connection string for the account that contains an access key required: - container_name - connection_string - if: properties: archive_provider: enum: - S3 then: properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret bucket: type: string title: Bucket pattern: (?!(^xn--|.+-s3alias$))^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ description: The S3 bucket name. Do not include a leading s3:// or a trailing / region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - access_key_id - secret_access_key - bucket - region properties: archive_provider: type: string title: Archive Provider description: The storage destination for your data. enum: - Azure - S3 default: Azure required: - archive_provider required: - destination - if: properties: type: enum: - http deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: HTTP description: Send data to a HTTP endpoint additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI to make HTTP requests to. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none auth: title: Authentication description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 description: The basic authentication user password: type: string title: Password minLength: 1 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none headers: type: array title: HTTP Headers description: An array of key/value objects for the headers to be sent with the request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 description: The value of the header. required: - header_name - header_value advanced_options: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-http-advanced-options title: Advanced Options description: Settings available for fine-tuning the destination behavior. additionalProperties: false dependencies: payload_prefix: - payload_suffix payload_suffix: - payload_prefix properties: max_bytes: type: integer title: Payload Size description: The maximum number of uncompressed bytes when batching data to the destination. minimum: 1 maximum: 2097152 timeout_secs: type: integer title: Timeout Seconds description: The number of seconds before a destination write timeout. minimum: 5 method: type: string enum: - post - put - patch - delete - get - head - options - trace title: HTTP Method description: The HTTP method to use for the destination. payload_prefix: type: string title: Payload Prefix description: Add a prefix to the payload. Only used for serialized JSON chunks. This option also requires "Payload Suffix" to form a valid JSON string. examples: - '{"event":"' - '{"extra_prop": true, "our_thing":"' minLength: 1 payload_suffix: type: string title: Payload Suffix description: Used in combination with "Payload Prefix" to form valid JSON from the payload. examples: - "}" - '"extra_prop": true }' minLength: 1 proxy: type: object additionalProperties: false title: Proxy Settings description: Configure proxy settings for the destination. properties: enabled: type: boolean title: Enabled description: Turns proxying on/off default: false endpoint: type: string minLength: 1 title: HTTP(S) Endpoint description: HTTP or HTTPS Endpoint to use for traffic. format: uri errorMessage: format: Must be a valid URI hosts_bypass_proxy: type: array minItems: 1 title: Hosts to Bypass description: A list of hosts to bypass proxying. Can be specified as a domain name, IP address, or CIDR block. Wildcards are supported as a dot (.) in domain names, or as a star (*) to match all hosts. examples: - 93.184.216.34 - ignorehost.example.com - .ignoreall.example.com - 93.184.216.0/24 items: type: string title: Host description: Domain name, IP address, or CIDR block. minLength: 1 tls_protocols: type: array title: TLS Protocols description: A list of ALPN protocols to use during TLS negotiation. They are attempted in the order they appear. items: type: string title: Protocol description: An ALPN protocol name minLength: 1 minItems: 1 rate_limiting: type: object additionalProperties: false title: Rate Limiting description: Settings for controlling rate limiting to the destination. properties: request_limit: type: integer minimum: 1 title: Request Limit description: The max number of requests allowed within the specified "Duration Seconds" setting. duration_secs: type: integer default: 1 title: Duration Seconds description: The window of time used to apply "Request Limit." minimum: 1 required: - uri - encoding - compression - if: properties: type: enum: - cloudwatch-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Cloudwatch Logs description: Publish log events to AWS Cloudwatch Logs additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none group_name: type: string title: Group Name minLength: 1 description: The name of the log group for the targeted log stream. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the targeted log stream. required: - auth - encoding - compression - group_name - region - stream_name - if: properties: type: enum: - cloudwatch-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Cloudwatch Metrics description: Send metric events to AWS Cloudwatch Metrics additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none namespace: type: string title: Namespace minLength: 1 description: Name for the container that will isolate metrics from one another. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - auth - compression - namespace - region - if: properties: type: enum: - sqs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Simple Queue Service (SQS) description: Publish log events to AWS Simple Queue Service additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text sqs_queue_url: type: string title: SQS Queue URL format: url errorMessage: format: Must be a valid URL description: The URL of the SQS queue that the data is publishing to. region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. required: - auth - encoding - sqs_queue_url - region - if: properties: type: enum: - notification-channel deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Notification Channel description: Send a Message to a 3rd Party Service. additionalProperties: false properties: service: title: Service description: Configure a service to receive the payload. Currently supports Slack, PagerDuty, Webhook or Mezmo (Log Analysis). Where allowed, messagetemplating can be used. To use a template, surround field names with double curly braces, e.g., {{.my_field}} or {{."my-object".thing}}. See [documentation](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) for more examples. additionalProperties: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 allOf: - if: properties: name: enum: - slack required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The text to be sent as a Slack message. required: - uri - message_text - if: properties: name: enum: - pager_duty required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 summary: type: string title: Summary description: Summarize the alert details for PagerDuty. minLength: 1 ui: display_type: data-template severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. source: type: string title: Source description: The source of the alert. minLength: 1 ui: display_type: data-template routing_key: type: string title: Routing Key description: The service integration routing key. minLength: 1 ui: display_type: secret event_action: type: string enum: - trigger - acknowledge - resolve title: Event Action description: The event action associated with the alert. required: - uri - summary - severity - source - routing_key - event_action - if: properties: name: enum: - webhook required: - name then: properties: uri: type: string title: URI format: uri errorMessage: format: Must be a valid URI description: The full URI where the payload is sent. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. minLength: 1 method: type: string title: HTTP Method description: The HTTP method to use for the request. enum: - post - put - patch - delete - get - head - options - trace ui: default_hint: post message_text: type: string title: Message minLength: 1 ui: display_type: data-template options: multiline: true description: The Webhook payload to send. This can be a text string or stringified JSON. If the message can be parsed as JSON, it will be sent as such. headers: type: array title: HTTP Headers description: A list of key/value pairs to be included in the HTTP request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 maxLength: 1024 description: The value of the header. required: - header_name - header_value maxItems: 20 auth: title: Authentication Options description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 description: The basic authentication user password: type: string title: Password minLength: 1 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: Choose basic or token-based authentication. default: none required: - uri - message_text - if: properties: name: enum: - log_analysis required: - name then: properties: severity: type: string enum: - INFO - WARNING - ERROR - CRITICAL title: Severity description: The severity level of the alert. default: INFO subject: type: string title: Subject description: A string to be used as the subject of the alert. minLength: 1 maxLength: 200 ui: display_type: data-template body: type: string title: Body description: A string to be used as the alert message body. minLength: 1 maxLength: 1024 ui: display_type: data-template ingestion_key: type: string title: Ingestion Key description: The key required to communicate to Log Analysis for sending the alert. ui: display_type: mezmo-ingestion-key minLength: 1 required: - severity - subject - body - ingestion_key properties: name: title: Name description: The name of the service that will receive the notification. ui: display_type: list errorMessage: must be a valid service name oneOf: - label: Slack enum: - slack - label: PagerDuty enum: - pager_duty - label: Webhook enum: - webhook - label: Mezmo Log Analysis enum: - log_analysis required: - name throttling: type: object title: Throttling description: Only allow a certain number of notifications to be sent within a given time window. properties: window_secs: type: integer title: Window Seconds description: The time frame during which the number of notifications (set by the Threshold) is permitted. default: 60 minimum: 1 threshold: type: integer title: Threshold description: 'The maximum number of notifications allowed over the given time window set by "Window Seconds" (default: 1).' default: 1 minimum: 1 required: - service - throttling - if: properties: type: enum: - azure-blob-storage deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Azure Blob Storage description: Publish log events to Azure Blob Storage additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none container_name: type: string title: Container Name minLength: 1 description: The name of the container for blob storage. connection_string: type: string title: Connection String minLength: 1 description: A connection string for the account that contains an access key prefix: type: string title: Prefix minLength: 1 description: A prefix to be applied to all object keys ui: display_type: data-template file_consolidation: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-azure-blob-file-consolidation title: File Consolidation description: This sink writes many small files out to azure blob storage. Enabling this process will allow the automatic consolidation of these small files into larger files of your choosing. This process will enable upon deployment and run on the chosen interval from `Processing Interval` creating files named `merged_[timestamp].log` where `timestamp` is the time since epoch when the actual file was created. The process will recursively access all files under the `Base Path` to handle merging sub-directory logging structures. additionalProperties: false properties: enabled: type: boolean default: false title: Enabled description: Toggles whether the process is enabled. process_every_seconds: type: integer title: Processing Interval (seconds) description: How often to run the consolidation process. default: 600 minimum: 300 maximum: 3600 requested_size_bytes: type: integer title: Requested File Size (bytes) description: The requested size of the consolidated files. default: 500000000 minimum: 50000000 maximum: 10000000000 base_path: type: string title: Base Path default: "" maxLength: 512 description: The path from the container to begin recursivelylooking for files to merge. A blank value indicates root.Merged files will only contain data from the respectivefolder. required: - encoding - compression - container_name - connection_string - if: properties: type: enum: - kinesis-firehose deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Kinesis Data Firehose description: Publish log events to AWS Kinesis Data Firehose additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the Kinesis Firehose stream required: - auth - encoding - compression - region - stream_name - if: properties: type: enum: - kinesis-streams deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS Kinesis Streams description: Publish log events to AWS Kinesis Streams additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none partition_key_field: type: string title: Partition Key Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ description: The field in the log whose value is used as Kinesis' partition key. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. stream_name: type: string title: Stream Name minLength: 1 description: The name of the targeted log stream. required: - auth - encoding - compression - region - stream_name - if: properties: type: enum: - elasticsearch deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Elasticsearch description: Publish log events to Elasticsearch additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none auth: title: ElasticSearch Auth Strategy description: ElasticSearch authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - aws then: properties: access_key_id: type: string title: Access Key ID minLength: 1 maxLength: 1024 description: AWS access key ID ui: display_type: secret secret_access_key: type: string title: Secrete Access Key minLength: 1 maxLength: 1024 description: AWS secrete access key ui: display_type: secret region: type: string title: Region minLength: 1 maxLength: 128 description: The name of the AWS region that is targeted. required: - access_key_id - secret_access_key - region properties: strategy: type: string title: Strategy description: The ES authentication strategy to use enum: - basic - aws required: - strategy endpoints: type: array title: Endpoints uniqueItems: true minItems: 1 description: An array of ElasticSearch endpoints items: type: string title: URL format: url errorMessage: format: Must be a valid URL description: The full Elasticsearch URL to send data to. minLength: 1 maxLength: 1024 pipeline: type: string title: Pipeline minLength: 1 description: Name of an ES ingest pipeline to include. index: type: string minLength: 1 maxLength: 512 title: Custom Index description: Index to use when writing ES events (default = mezmo-%Y.%m.%d) default: mezmo-%Y.%m.%d required: - compression - auth - endpoints - if: properties: type: enum: - gcp-cloud-monitoring deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud Monitoring description: Publish metrics events to GCP Cloud Monitoring additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. resource_type: type: string title: Resource Type minLength: 1 description: The `monitored-resource` type as defined in Monitoring. resource_labels: type: array title: Resource Labels description: An array of key/val objects describing labels of the Monitoring resource. items: type: object title: Label/Value description: A key/val object describing a label name and its value. additionalProperties: false properties: label_name: type: string title: Label Name minLength: 1 description: The name of the label. label_value: type: string title: Label Value minLength: 1 description: The value of the label. required: - label_name - label_value required: - credentials_json - project_id - resource_type - if: properties: type: enum: - gcp-cloud-storage deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud Storage description: Publish log events to GCP Cloud Storage additionalProperties: false properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file encoding: type: string description: Dictates how the data will be serialized before storing. enum: - json - ndjson - text title: Encoding default: text bucket: type: string title: Bucket minLength: 1 description: The name of the bucket in GCP where the data will be stored. compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none bucket_prefix: type: string title: Bucket Prefix minLength: 1 description: The prefix applied to the bucket name, giving the appearance of having directories. required: - credentials_json - encoding - bucket - compression - if: properties: type: enum: - gcp-cloud-operations deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud Operations description: Publish log events to GCP Cloud Operations additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file log_id: type: string title: Log ID minLength: 1 description: Concise reference for the log stream name. project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. resource_type: type: string title: Resource Type minLength: 1 description: The `monitored-resource` type as defined in Monitoring. resource_labels: type: array title: Resource Labels description: An array of key/val objects describing labels of the Monitoring resource. items: type: object title: Label/Value description: A key/val object describing a label name and its value. additionalProperties: false properties: label_name: type: string title: Label Name minLength: 1 description: The name of the label. label_value: type: string title: Label Value minLength: 1 description: The value of the label. required: - label_name - label_value required: - credentials_json - log_id - project_id - resource_type - if: properties: type: enum: - gcp-cloud-pubsub deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: GCP Cloud PubSub description: Publish log events to GCP Cloud PubSub additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true credentials_json: type: string title: JSON Credentials minLength: 1 description: JSON Credentials ui: display_type: secret-file encoding: type: string description: Dictates how the data will be serialized before storing. enum: - json - text title: Encoding default: text project_id: type: string title: Project ID minLength: 1 description: The Project ID as defined in Google Cloud. topic: type: string title: Topic minLength: 1 description: The name of the topic to publish messages to. required: - credentials_json - encoding - project_id - topic - if: properties: type: enum: - honeycomb-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Honeycomb Logs description: Send log data to Honeycomb additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Honeycomb API key description: Your Honeycomb API key minLength: 1 maxLength: 512 ui: display_type: secret dataset: type: string title: Dataset description: The name of the targeted dataset. minLength: 1 maxLength: 512 required: - api_key - dataset - if: properties: type: enum: - indexed-search deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Mezmo Indexed Search description: Publish log events to Mezmo's data store for Indexed Search capabilities additionalProperties: true properties: search_field: type: string title: Full Text Search Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: .line maxLength: 256 description: If your data reaches this stage as a plain-text string, the entire payload will be indexed for full-text searching. If your data reaches this stage as an object, the field you enter here will be used as the full-text search field. Nested fields are allowed. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference timestamp_field: type: string title: Timestamp Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: timestamp maxLength: 256 description: If your data reaches this stage as an object, this field represents the timestamp value to be cataloged for searching. If the field does not exists the current time will be used. We accept UNIX timestamps (in milliseconds), ISO8601/RFC3389, and RFC2822 formats. errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - search_field - timestamp_field - if: properties: type: enum: - kafka deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Kafka description: Publish log events to Kafka additionalProperties: type: string allOf: - if: properties: sasl_enabled: enum: - true then: properties: sasl_mechanism: type: string enum: - SCRAM-SHA-512 - SCRAM-SHA-256 - PLAIN title: SASL description: The Kafka SASL/SCRAM mechanism default: SCRAM-SHA-512 sasl_username: type: string title: Username description: The SASL/SCRAM username minLength: 1 sasl_password: type: string title: Password description: The SASL/SCRAM username ui: display_type: secret minLength: 1 required: - sasl_mechanism - sasl_username - sasl_password - if: properties: tls_enabled: enum: - true then: properties: tls_ca_certificate: type: string title: TLS CA Certificate Chain minLength: 1 description: The CA certificate chain in the PEM format ui: display_type: secret-file format: x509_certificate errorMessage: format: Must be a valid PEM or DER certificate tls_verify_certificate: type: boolean title: TLS Verify Certificate description: Verify the TLS certificates default: true required: - tls_verify_certificate properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - lz4 - snappy - zstd - none title: Compression default: none event_key_field: type: string title: Event Key Field minLength: 1 description: The field in the log whose value is used as Kafka's event key. brokers: type: array title: Brokers uniqueItems: true minItems: 1 description: An array of key/val objects for each host/port of a Kafka broker. items: type: object title: Host/Port description: The host and port for the Kafka broker. additionalProperties: false properties: host: title: Host description: The hostname or IP address of the broker. anyOf: - type: string format: ipv4 errorMessage: format: Must be a valid IPv4 address - type: string format: hostname errorMessage: format: Must be a valid hostname port: type: number title: Port description: The port that the broker listens on. required: - host - port topic: type: string title: Topic minLength: 1 description: The name of the topic to publish to. group_id: type: string title: Group ID minLength: 1 description: The consumer group identifier sasl_enabled: type: boolean title: SASL/SCRAM Enabled description: Enable SASL/SCRAM authentication with Kafka default: false tls_enabled: type: boolean title: Use TLS description: Enable TLS during connections to the remote default: false required: - encoding - compression - brokers - topic - if: properties: type: enum: - loki deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Loki description: Publish log events to Loki additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: title: Loki Auth Strategy description: Loki authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token (passed as-is) ui: display_type: secret required: - token properties: strategy: type: string title: Strategy description: The authentication strategy to use (only basic supported) enum: - basic - bearer required: - strategy encoding: type: object title: Encoding description: Configures how event are encoded properties: codec: type: string title: Codec enum: - json - text required: - codec endpoint: type: string title: Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Loki base URL path: type: string title: Path minLength: 1 maxLength: 512 description: The path appended to the Loki base URL(defaults to /loki/api/v1/push) default: /loki/api/v1/push labels: type: array title: Loki Labels description: Key/Value pair used to describe Loki data items: type: object title: Labels description: Label name and values additionalProperties: false properties: label_name: type: string title: Label Name description: The name of the label minLength: 1 maxLength: 512 label_value: type: string title: Label Value description: The value of the label minLength: 1 maxLength: 512 required: - label_name - label_value required: - auth - encoding - endpoint - labels - if: properties: type: enum: - new-relic deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: New Relic description: Publish to New Relic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true account_id: type: string title: New Relic Account ID description: Account ID for your New Relic account minLength: 1 maxLength: 1024 ui: display_type: secret api: type: string title: API description: New Relic API endpoint (events not supported) enum: - logs - metrics default: logs license_key: type: string title: New Relic License Key description: License key for your New Relic account minLength: 1 maxLength: 1024 ui: display_type: secret required: - account_id - api - license_key - if: properties: type: enum: - prometheus-remote-write deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Prometheus Remote Write description: Publish metrics to Prometheus Remote Write endpoint additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Endpoint format: uri errorMessage: format: Must be a valid URI description: "The full URI to make HTTP requests to. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI. Example: http://example.org:8080/api/v1/push" minLength: 1 maxLength: 2000 auth: title: Authentication description: Configures HTTP authentication allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none required: - strategy required: - endpoint - if: properties: type: enum: - pulsar deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Pulsar description: Publish log events to Apache Pulsar additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text url: type: string title: URL format: url errorMessage: format: Must be a valid URL description: The full Pulsar URL to send data to. topic: type: string title: Topic minLength: 1 description: The name of the Pulsar topic to publish to. required: - encoding - url - topic - if: properties: type: enum: - redis deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Redis description: Publish log events to Redis additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true encoding: type: string description: The encoding to apply to the data. enum: - json - text title: Encoding default: text url: type: string title: URL format: uri errorMessage: format: Must be a valid URL description: The full Redis URL to send data to. key: type: string title: Key minLength: 1 description: The name of the Redis key to write data to. required: - encoding - url - key - if: properties: type: enum: - splunk-hec-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Splunk HTTP Event Collector description: Publish log events to Splunk HTTP Event Collector additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none url: type: string title: Endpoint format: url errorMessage: format: Must be a valid URL description: The base URL for the Splunk instance. The collector path, such as `/services/collector/events`, will be automatically inferred from the destination's configuration. token: type: string title: Token minLength: 1 ui: display_type: secret description: The default token to authenticate to Splunk HEC. tls_verify_certificate: type: boolean title: Verify TLS Certificate description: If unchecked, the TLS certificate of the remote host will not be verified. **This is insecure**, but may be useful for testing or when evaluating Splunk in a trial instance. default: true host_field: type: string title: Host Field description: The field that contains the hostname to include in the event. default: metadata.host timestamp_field: type: string title: Timestamp Field description: The field that contains the timestamp to include in the event. default: metadata.time pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: feature_flag: pipeline-splunk-sink-v2 display_type: field-reference source: title: Source description: The source of events sent to this sink. This is typically the filename the logs originated from. Use the field path ".metadata.source" to use the upstream source value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - "null" required: - value_type sourcetype: title: Source Type description: The sourcetype of events sent to this sink. Use the field path ".metadata.sourcetype" to use the upstream sourcetype value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - "null" required: - value_type index: title: Index description: The name of the index to send events to. Use the field path ".metadata.index" to use the upstream index value from a HEC log source additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 512 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit value. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: ui: feature_flag: pipeline-splunk-sink-v2 type: - string - "null" required: - value_type required: - compression - url - token - source - sourcetype - index - if: properties: type: enum: - sumo-logic-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sumo Logic Logs description: Publish log data to Sumo Logic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Sumo Logic Collector URL that receives the data ui: display_type: secret compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none category: type: string minLength: 1 maxLength: 512 title: Category description: Category to use when publishing data to Sumo Logic (default = mezmo_default_category) default: mezmo_default_category required: - endpoint - compression - if: properties: type: enum: - sumo-logic-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sumo Logic Metrics description: Publish metrics data to Sumo Logic additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The Sumo Logic Collector URL that receives the data ui: display_type: secret compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none category: type: string minLength: 1 maxLength: 512 title: Category description: Category to use when publishing data to Sumo Logic (default = mezmo_default_category) default: mezmo_default_category required: - endpoint - compression - if: properties: type: enum: - vector deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Vector description: Publish log events to Vector additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none vector_ip: type: string title: Vector IP format: ipv4 description: The IP portion of the Vector host vector_port: type: number title: Vector Port description: The port that the Vector host listens on. version: type: string enum: - "1" - "2" title: Version default: "2" description: The version of the Vector sink which must match the source version. required: - compression - vector_ip - vector_port - version - if: properties: type: enum: - s3 deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: AWS S3 description: Publish events as objects in AWS S3 additionalProperties: false errorMessage: properties: bucket: must be a valid bucket name tags: must be a valid tag identifier properties: batch_timeout_secs: type: integer title: Batch Timeout (seconds) description: The maximum amount of time, in seconds, events will be buffered before being flushed to the destination enum: - 30 - 60 - 90 - 120 - 300 default: 300 ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true auth: type: object title: AWS Authentication description: Configures AWS authentication additionalProperties: false properties: access_key_id: type: string title: Access Key ID minLength: 1 description: The AWS access key id secret_access_key: type: string title: Secret Access Key minLength: 1 description: The AWS secret access key ui: display_type: secret required: - access_key_id - secret_access_key bucket: type: string title: Bucket pattern: (?!(^xn--|.+-s3alias$))^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ description: The S3 bucket name. Do not include a leading s3:// or a trailing / prefix: type: string title: Prefix description: A prefix to apply to all object key names. default: / ui: display_type: data-template tags: type: array maxItems: 10 title: Tags description: A set of tags (key/value pairs) to apply to the object in S3. items: type: object properties: key: type: string title: Name description: The tag name pattern: ^[A-Za-z0-9_.:=+@/\-\s]*$ minLength: 1 maxLength: 128 value: type: string title: Value description: The tag value default: "" pattern: ^[A-Za-z0-9_.:=+@/\-\s]*$ minLength: 0 maxLength: 256 required: - key - value encoding: type: string description: The encoding to apply to the data. enum: - json - ndjson - text title: Encoding default: text compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: none region: type: string title: Region minLength: 1 description: The name of the AWS region that is targeted. file_consolidation: type: object ui: display_type: collapsable feature_flag: pipeline-node-sink-s3-file-consolidation title: File Consolidation description: "This sink writes many small files out to S3. Enabling this process will allow the automatic consolidation of these small files into larger files of your choosing. This process will enable upon deployment and run on the chosen interval from `Processing Interval` creating files named `merged_[timestamp].log` where `timestamp` is the time since epoch. *NOTE*: only `text` and `ndjson` files written by the sink are consolidated." additionalProperties: false properties: enabled: type: boolean default: false title: Enabled description: Toggles whether the process is enabled. process_every_seconds: type: integer title: Processing Interval (seconds) description: How often to run the consolidation process. default: 600 minimum: 300 maximum: 3600 requested_size_bytes: type: integer title: Requested File Size (bytes) description: The size of the requested files. default: 500000000 minimum: 50000000 maximum: 10000000000 base_path: type: string title: Base Path default: "" maxLength: 512 description: The path from the container to begin recursivelylooking for files to merge. A blank value indicates root.Merged files will only contain data from the respectivefolder. required: - auth - bucket - prefix - encoding - compression - region - if: properties: type: enum: - blackhole deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Blackhole description: Drop data arriving at this destination additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true - if: properties: type: enum: - datadog-logs deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Datadog Logs description: Publish log events to Datadog additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Datadog API Key description: Datadog log application API key minLength: 1 ui: display_type: secret compression: type: string description: The compression strategy used on the encoded data prior to sending. enum: - gzip - none title: Compression default: gzip site: type: string description: The Datadog site (region) to send logs to. enum: - us1 - us3 - us5 - eu1 title: Datadog Site default: us1 required: - api_key - compression - site - if: properties: type: enum: - datadog-metrics deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Datadog Metrics description: Publish metrics events to Datadog. Sending custom metrics to Datadog may result in extra charges on your Datadog bill. Please go to https://docs.datadoghq.com/metrics/custom_metrics/ for more information on Datadog custom metrics pricing. additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true api_key: type: string title: Datadog API Key description: Datadog metrics application API key minLength: 1 maxLength: 512 ui: display_type: secret site: type: string description: The Datadog site (region) to send metrics to. enum: - us1 - us3 - us5 - eu1 title: Datadog Site default: us1 required: - api_key - site - if: properties: type: enum: - clickhouse deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Clickhouse description: Deliver log data to the ClickHouse database additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string minLength: 1 maxLength: 2048 title: Endpoint description: Endpoint of the Clickhouse server format: uri errorMessage: format: Must be a valid URI auth: title: Auth Strategy description: Authentication strategy allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 2048 description: The bearer token to use for authentication ui: display_type: secret required: - token properties: strategy: type: string title: Strategy description: The ES authentication strategy to use enum: - basic - bearer required: - strategy table: type: string title: Table description: The table that data is inserted into pattern: ^[a-zA-Z0-9_]+$ errorMessage: Invalid table name ui: display_type: data-template database: type: string title: Database description: The database that contains the table that data is inserted into pattern: ^[a-zA-Z0-9_]+$ errorMessage: Invalid database name ui: display_type: data-template compression: type: string enum: - gzip - none - snappy - zlib - zstd title: Compression description: The compression algorithm to use default: gzip skip_unknown_fields: type: boolean title: Skip Unknown Fields description: Allows ClickHouse to discard fields not present in the table schema default: false required: - endpoint - auth - table - if: properties: type: enum: - opentelemetry deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: OpenTelemetry description: Publish data to OpenTelemetry additionalProperties: false properties: ack_enabled: type: boolean title: End-to-end Acknowledgement description: Sets end-to-end acknowledgement on the destination default: true endpoint: type: string title: Collector Endpoint format: url errorMessage: format: Must be a valid URL minLength: 1 maxLength: 512 description: The OpenTelemetry Collector URL that receives the data auth: title: Authentication description: Configures HTTP authentication additionalProperties: type: string allOf: - if: properties: strategy: enum: - basic then: properties: user: type: string title: User minLength: 1 maxLength: 512 description: The basic authentication user password: type: string title: Password minLength: 1 maxLength: 512 description: The basic authentication password ui: display_type: secret required: - user - password - if: properties: strategy: enum: - bearer then: properties: token: type: string title: Token minLength: 1 maxLength: 1024 description: The bearer token ui: display_type: secret required: - token properties: strategy: type: string enum: - basic - bearer - none title: Strategy description: The authentication strategy to use default: none headers: type: array title: HTTP Headers description: An array of key/value objects for the headers to be sent with the request. items: type: object title: Headers description: A key/value object describing a header name and its value. additionalProperties: false properties: header_name: type: string title: Header Name minLength: 1 description: The name of the header. header_value: type: string title: Header Value minLength: 1 description: The value of the header. required: - header_name - header_value compression: type: string title: Compression description: The compression strategy (defaults to none). enum: - gzip - none default: none required: - endpoint - compression properties: title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description type: type: string enum: - mezmo - mezmo-archive - http - cloudwatch-logs - cloudwatch-metrics - sqs - notification-channel - azure-blob-storage - kinesis-firehose - kinesis-streams - elasticsearch - gcp-cloud-monitoring - gcp-cloud-storage - gcp-cloud-operations - gcp-cloud-pubsub - honeycomb-logs - indexed-search - kafka - loki - new-relic - prometheus-remote-write - pulsar - redis - splunk-hec-logs - sumo-logic-logs - sumo-logic-metrics - vector - s3 - blackhole - datadog-logs - datadog-metrics - clickhouse - opentelemetry inputs: type: array minItems: 1 items: type: string user_config: type: object required: - inputs - user_config required: - sources - transforms - sinks required: true responses: "200": description: Default Response /pipeline/ai/expression: post: summary: Use AI to generate Regular Expressions tags: - Experimental description: Generates one or more regular expression that will parse the given log samples requestBody: content: application/json: schema: type: object additionalProperties: false properties: samples: type: array description: At least 1 line to pass to the AI model for pattern generation. If the lines are not all from the same application, it will generate an expression for each identified app. It accepts a maximum of 5 lines. items: type: string description: A log line minLength: 1 maxLength: 4096 minItems: 1 maxItems: 5 required: - samples required: true responses: "200": description: Default Response /pipeline/ai/auto-parse: post: summary: Use AI to automatically parse log text into JSON tags: - Experimental description: Generates a parsed JSON object for each textual log line provided. requestBody: content: application/json: schema: type: object additionalProperties: false properties: samples: type: array description: At least 1 line to pass to the AI model for JSON generation. It accepts a maximum of 25 lines. items: type: string description: A log line minLength: 1 maxLength: 4096 minItems: 1 maxItems: 25 required: - samples required: true responses: "200": description: Default Response /pipeline/classifier/analysis: get: summary: Fetch classifier information tags: - Data Classification & Profiling description: Retrieve information about a classification of a shared source parameters: - schema: type: string enum: - in_progress - complete in: query name: status required: false description: Fetch the classifier analyses having this status responses: "200": description: Default Response /pipeline/classifier/analysis/{analysis_id}: patch: summary: Update analysis options tags: - Data Classification & Profiling description: Reset classification and profiling results requestBody: content: application/json: schema: type: object additionalProperties: false properties: status: type: string enum: - in_progress - complete description: The new status of the classifier analysis required: - status required: true parameters: - schema: type: boolean default: false in: query name: restart_analysis required: false description: Reset the current event count so that classification can begin when 'status' is set to "in_progress". - schema: type: boolean default: false in: query name: reset_profile required: false description: Reset the profiling data associated with this analysis. - schema: type: boolean default: false in: query name: create_new_classifier_analysis required: false description: Create a new classifier analysis record to replace the current record. Must be used in conjunction with a "complete" status - schema: type: string format: uuid in: path name: analysis_id required: true description: The id of the classifier analysis to update responses: "200": description: Default Response /pipeline/classifier/{analysis_id}/fields: get: summary: Get Field Summary for an analysis tags: - Data Classification & Profiling description: Get Field Summary for an analysis parameters: - schema: type: string format: uuid in: path name: analysis_id required: true description: The classifier analysis to get fields of responses: "200": description: Default Response /pipeline/gateway-route/{gateway_route_id}/access-key/{id}: delete: summary: Delete access key tags: - Shared Sources description: Delete an access key parameters: - schema: type: string format: uuid in: path name: gateway_route_id required: true - schema: type: string format: uuid in: path name: id required: true responses: "200": description: Default Response /pipeline/gateway-route/{gateway_route_id}/access-key: post: summary: Create access key tags: - Shared Sources description: Create an access key associated with this gateway route requestBody: content: application/json: schema: type: object allOf: - if: title: Type properties: type: type: string enum: - user_defined required: - type then: title: Key properties: key: type: string description: The access key to use required: - key properties: type: type: string enum: - generated - user_defined title: type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ errorMessage: invalid title name key: type: string minLength: 1 maxLength: 512 required: - type - title required: true parameters: - schema: type: string format: uuid in: path name: gateway_route_id required: true responses: "200": description: Default Response /pipeline/gateway-route/{gateway_route_id}/signing-key/{id}: delete: summary: Delete signing key tags: - Shared Sources description: Removes a signing key attached to a gateway route parameters: - schema: type: string format: uuid in: path name: gateway_route_id required: true - schema: type: string format: uuid in: path name: id required: true responses: "200": description: Default Response /pipeline/gateway-route/{gateway_route_id}/signing-key: put: summary: Add signing key tags: - Shared Sources description: Adds a signing key to a gateway route requestBody: content: application/json: schema: type: object additionalProperties: false properties: key: type: string minLength: 1 maxLength: 512 required: - key required: true parameters: - schema: type: string format: uuid in: path name: gateway_route_id required: true responses: "200": description: Default Response /pipeline/metric/health: get: summary: Fetch pipeline health metrics tags: - Metrics description: Query and fetch health information for the current account. parameters: - schema: type: string format: uuid in: query name: pipeline_id required: false description: Limit health metrics to the given pipeline id. If not defined, the metrics will include all pipelines for the account. - schema: type: boolean default: true in: query name: include_logs required: false description: Fetch previous log messages for transforms. responses: "200": description: Default Response /pipeline/metric/usage: get: summary: Fetch account usage metics tags: - Metrics description: Query and fetch usage metrics for the current account. parameters: - schema: type: string format: date-time in: query name: from required: true description: Define the beginning of the window - schema: type: string format: date-time in: query name: to required: true description: Define the end of the window - schema: type: integer minimum: 300 in: query name: bucket required: true description: Size of the interval represented by each returned metric result - schema: type: boolean default: true in: query name: include_deleted required: false description: Include Pipelines and nodes that have been deleted - schema: type: string format: uuid in: query name: pipeline_id required: false description: Limit usage metrics to the given pipeline id. If not defined, the metrics will include all pipelines for the account. - schema: type: string enum: - ingress - egress in: query name: traffic required: false description: Limit usage metrics to a single traffic direction. If not defined, the metrics for both directions will be included. responses: "200": description: Default Response /pipeline/{container_id}/source/{source_id}/sample/{sample_id}: delete: summary: Delete data sample tags: - Sampling and Simulation description: Remove a data sample from the pipeline parameters: - schema: type: string title: Container ID format: uuid in: path name: container_id required: true description: The ID of the container entity, either a Pipeline or Processor Group - schema: title: Source ID oneOf: - type: string format: uuid - enum: - transform-group-input-source-id in: path name: source_id required: true description: The ID of the source to retrieve samples for - schema: type: string title: Sample ID format: uuid in: path name: sample_id required: true description: The ID of the sample responses: "200": description: Default Response get: summary: Fetch data sample tags: - Sampling and Simulation description: Returns a data sample parameters: - schema: type: string title: Container ID format: uuid in: path name: container_id required: true description: The ID of the container entity, either a Pipeline or Processor Group - schema: title: Source ID oneOf: - type: string format: uuid - enum: - transform-group-input-source-id in: path name: source_id required: true description: The ID of the source to retrieve samples for - schema: type: string title: Sample ID format: uuid in: path name: sample_id required: true description: The ID of the sample responses: "200": description: Default Response put: summary: Update data sample tags: - Sampling and Simulation description: Updates an existing data sample requestBody: content: application/json: schema: type: object properties: title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ errorMessage: invalid title name - type: "null" parameters: - schema: type: string title: Container ID format: uuid in: path name: container_id required: true description: The ID of the container entity, either a Pipeline or Processor Group - schema: title: Source ID oneOf: - type: string format: uuid - enum: - transform-group-input-source-id in: path name: source_id required: true description: The ID of the source to retrieve samples for - schema: type: string title: Sample ID format: uuid in: path name: sample_id required: true description: The ID of the sample responses: "200": description: Default Response /pipeline/{container_id}/source/{source_id}/sample: get: summary: List data samples tags: - Sampling and Simulation description: Returns a list of data samples that exist in the pipeline parameters: - schema: type: string title: Container ID format: uuid in: path name: container_id required: true description: The ID of the container entity, either a Pipeline or Processor Group - schema: title: Source ID oneOf: - type: string format: uuid - enum: - transform-group-input-source-id in: path name: source_id required: true description: The ID of the source to retrieve samples for responses: "200": description: Default Response post: summary: Create data sample tags: - Sampling and Simulation description: Create sample data for use in simulation requestBody: content: application/json: schema: type: object additionalProperties: false properties: title: oneOf: - type: string minLength: 1 maxLength: 512 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid title name version: title: Sample Version default: v2 enum: - v2 container_type: type: string enum: - pipeline - transform_group title: Container Type description: Discriminator to identify the type of container referenced by the container_id default: pipeline data: type: array title: Sample data minItems: 1 maxItems: 100 items: $ref: "#sample-envelope" required: - data required: true parameters: - schema: type: string title: Container ID format: uuid in: path name: container_id required: true description: The ID of the container entity, either a Pipeline or Processor Group - schema: title: Source ID oneOf: - type: string format: uuid - enum: - transform-group-input-source-id in: path name: source_id required: true description: The ID of the source to retrieve samples for responses: "200": description: Default Response /pipeline/account/local-deploy/config: get: summary: Get client config tags: - Edge - Local Deploy description: Gets client configuration for Edge pipelines parameters: - schema: type: string in: query name: deployment_group required: false description: The deployment group for which to construct configuration responses: "200": description: Default Response post: summary: Post client config tags: - Edge - Local Deploy description: Posts deployment configuration+liveness for Edge pipelines, returning pipeline config requestBody: content: application/json: schema: type: object properties: edge_id: type: string pattern: ^[a-zA-Z0-9-.]{1,253}$ replica: type: string pattern: ^[a-zA-Z0-9-.]{1,253}$ namespace: type: string pattern: ^[a-zA-Z0-9-.]{1,253}$ name: type: string pattern: ^[a-zA-Z0-9-.]{1,253}$ deployment_group: type: string pattern: ^[a-zA-Z0-9-_]{1,256}$ ports: type: array items: type: integer version: type: string chart_version: type: string vector_version: type: string uptime_sec: type: integer responses: "200": description: Default Response /pipeline/account/local-deploy/log: post: summary: Fetch logs from Edge tags: - Edge - Local Deploy description: Receive logs from an Edge pipeline requestBody: content: application/json: schema: type: array items: type: object additionalProperties: false properties: pipeline_id: type: string format: uuid component_id: type: string format: uuid message: type: string minLength: 1 maxLength: 16384 level: type: string enum: - DEBUG - INFO - WARN - ERROR last_seen_at: type: string format: date-time required: - pipeline_id - component_id - message - level - last_seen_at minItems: 1 parameters: - schema: type: string minLength: 1 in: query name: edge_id required: false description: Logs for specific edge_id responses: "200": description: Default Response /pipeline/account/local-deploy/tasks: get: summary: Poll tasks tags: - Edge - Local Deploy description: Poll and claim available local deployment tasks for a pipeline parameters: - schema: type: integer default: 10 minimum: 1 maximum: 100 in: query name: limit required: false description: The maximum number of tasks to return - schema: type: string minLength: 1 in: query name: edge_id required: false description: Specific edge_id instance tap responses: "200": description: Default Response /pipeline/classifier/{analysis_id}/fields/{field_name}: get: summary: Get Field Summary values for a field in an analysis tags: - Data Classification & Profiling description: Get Field Summary values for a field in an analysis parameters: - schema: type: string format: uuid in: path name: analysis_id required: true description: The classifier analysis to get values of - schema: type: string in: path name: field_name required: true description: The name of the field to get values of responses: "200": description: Default Response /pipeline/metric/usage/profile: get: summary: Fetch usage metrics tags: - Data Classification & Profiling description: Query and fetch usage metrics from data profiling. parameters: - schema: type: string format: date-time in: query name: from required: true description: A date/time string that defines the beginning of the window - schema: type: string format: date-time in: query name: to required: true description: A date/time string that defines the end of the window - schema: type: string in: query name: id required: true description: The identifier of the classifier analysis - schema: type: string enum: - app - host - level - log_type in: query name: annotation required: true - schema: type: integer default: 20 minimum: 1 maximum: 500 in: query name: limit required: false responses: "200": description: Default Response /pipeline/{pipeline_id}/transform/{pipeline_transform_id}/classifier/analysis: get: tags: - Data Classification & Profiling parameters: - schema: enum: - in_progress - complete in: query name: status required: false description: Fetch the classifier analyses having this status - schema: type: string format: uuid in: path name: pipeline_id required: true - schema: type: string format: uuid in: path name: pipeline_transform_id required: true responses: "200": description: Default Response /pipeline/transform/classifier/analysis: get: tags: - Data Classification & Profiling parameters: - schema: enum: - in_progress - complete in: query name: status required: false description: Fetch the transform analysis having this status - schema: type: string format: uuid in: query name: pipeline_id required: false description: Fetch the processor analyses for the pipeline with the specified ID responses: "200": description: Default Response /pipeline/account/local-deploy/heartbeat/{edge_id}: delete: summary: Remove heartbeat tags: - Edge - Local Deploy description: Removes existing Edge heartbeats parameters: - schema: type: string in: path name: edge_id required: true responses: "200": description: Default Response /pipeline/account/local-deploy/heartbeat: get: summary: List all heartbeats tags: - Edge - Local Deploy description: List all Edge check-ins for the account responses: "200": description: Default Response /pipeline/account/local-deploy/metric/usage: post: summary: Get usage metrics tags: - Edge - Local Deploy description: Receive usage metrics from an Edge pipeline requestBody: content: application/json: schema: type: array items: type: object additionalProperties: false properties: event_ts: type: integer minimum: 0 pipeline_id: type: string format: uuid component_id: type: string minLength: 1 maxLength: 1024 processor: type: string minLength: 1 maxLength: 256 total_count: type: integer minimum: 0 total_size: type: integer minimum: 0 required: - event_ts - pipeline_id - component_id - processor - total_count - total_size minItems: 1 parameters: - schema: type: string minLength: 1 in: query name: edge_id required: false description: Metrics for specific edge_id responses: "200": description: Default Response /pipeline/account/local-deploy/tasks/{task_id}/results: post: summary: Add task result tags: - Edge - Local Deploy description: Add results for a given task requestBody: content: application/json: schema: type: object parameters: - schema: type: string format: uuid in: path name: task_id required: true responses: "200": description: Default Response /pipeline/metric/usage/profile/log-clustering: get: summary: Fetch log usage metrics tags: - Metrics description: Query and fetch log clustering usage metrics from data profiling. parameters: - schema: type: string format: date-time in: query name: from required: true description: A date/time string that defines the beginning of the window - schema: type: string format: date-time in: query name: to required: false description: A date/time string that defines the end of the window - schema: type: string in: query name: id required: true description: The identifier of the classifier analysis - schema: type: integer default: 20 minimum: 1 maximum: 200 in: query name: limit required: false responses: "200": description: Default Response /pipeline/metric/usage/profile/log-clustering/samples: get: summary: Fetch log cluster samples tags: - Metrics description: Query and fetch log cluster samples from data profiling. parameters: - schema: type: string in: query name: id required: true description: The identifier of the classifier analysis - schema: type: string in: query name: log_cluster_id required: true description: The cluster identifier - schema: type: string format: date-time in: query name: from required: true description: A date/time string that defines the beginning of the window - schema: type: string format: date-time in: query name: to required: false description: A date/time string that defines the end of the window - schema: type: integer default: 20 minimum: 1 maximum: 200 in: query name: limit required: false responses: "200": description: Default Response /transform-group/{transform_group_id}: delete: tags: - Processor Groups description: Delete a Processor Group parameters: - schema: type: string format: uuid in: path name: transform_group_id required: true responses: "200": description: Default Response get: tags: - Processor Groups description: Get a Processor Group parameters: - schema: type: string format: uuid in: path name: transform_group_id required: true responses: "200": description: Default Response put: tags: - Processor Groups description: Update a Processor Group requestBody: content: application/json: schema: type: object additionalProperties: false properties: title: type: string title: Title description: The title of the Processor Group pattern: ^[^<>"]+$ errorMessage: invalid title maxLength: 512 description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" title: Description description: A description of this Processor Group's behavior or purpose. errorMessage: invalid description required: - title required: true parameters: - schema: type: string format: uuid in: path name: transform_group_id required: true responses: "200": description: Default Response /transform-group: get: tags: - Processor Groups description: List all Processor Groups for the account responses: "200": description: Default Response post: tags: - Processor Groups description: Create a new Processor Group requestBody: content: application/json: schema: type: object additionalProperties: false properties: title: type: string title: Title description: The title of the Processor Group pattern: ^[^<>"]+$ errorMessage: invalid title maxLength: 512 description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" title: Description description: A description of this Processor Group's behavior or purpose. errorMessage: invalid description deploy_type: type: string title: Deploy Type description: The type of deployment this Processor Group will be used in. enum: - saas - local default: saas required: - title required: true responses: "200": description: Default Response /transform-group/{transform_group_id}/publish: post: tags: - Processor Groups description: Publish changes to a Processor Group parameters: - schema: type: boolean default: false in: query name: allow_unconnected_edges required: false - schema: type: string format: uuid in: path name: transform_group_id required: true responses: "200": description: Default Response /transform-group/from-pipeline: post: tags: - Processor Groups description: Create a new Processor Group from a set of Pipeline components requestBody: content: application/json: schema: type: object additionalProperties: false properties: title: type: string title: Title description: The title of the Processor Group pattern: ^[^<>"]+$ errorMessage: invalid title maxLength: 512 description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" title: Description description: A description of this Processor Group's behavior or purpose. errorMessage: invalid description deploy_type: type: string title: Deploy Type description: The type of deployment this Processor Group will be used in. enum: - saas - local default: saas pipeline_id: type: string format: uuid title: Pipeline ID description: The Pipeline ID to copy from. pipeline_transform_ids: type: array items: type: string format: uuid minItems: 1 title: Pipeline Transform IDs description: A list of IDs of transforms to copy into this new group. required: - title - pipeline_id - pipeline_transform_ids required: true responses: "200": description: Default Response /transform-group/{transform_group_id}/revision/{transform_group_revision_lookup}: get: tags: - Processor Groups parameters: - schema: type: string format: uuid in: path name: transform_group_id required: true - schema: oneOf: - type: string format: uuid - enum: - latest in: path name: transform_group_revision_lookup required: true responses: "200": description: Default Response /transform-group/{transform_group_id}/revision: get: tags: - Processor Groups parameters: - schema: type: integer default: 200 minimum: 1 maximum: 1000 in: query name: limit required: false description: The maximum number of revisions to return - schema: type: string format: uuid in: path name: transform_group_id required: true responses: "200": description: Default Response /transform-group/{transform_group_id}/simulate: post: tags: - Sampling and Simulation description: Sends data to a vector instance with a simulated transform group requestBody: content: application/json: schema: type: object properties: sample_id: type: string format: uuid required: - sample_id required: true parameters: - schema: type: string format: uuid in: path name: transform_group_id required: true responses: "200": description: Default Response /transform-group/{transform_group_id}/transform/{transform_group_transform_id}: delete: tags: - Processor Groups description: Delete a Processor from a Processor Group parameters: - schema: type: string format: uuid in: path name: transform_group_id required: true - schema: type: string format: uuid in: path name: transform_group_transform_id required: true responses: "200": description: Default Response get: tags: - Processor Groups description: Get a Processor from a Processor Group parameters: - schema: type: string format: uuid in: path name: transform_group_id required: true - schema: type: string format: uuid in: path name: transform_group_transform_id required: true responses: "200": description: Default Response put: tags: - Processor Groups description: Update a Processor in a Processor Group requestBody: content: application/json: schema: type: object allOf: - if: properties: type: enum: - filter deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Filter description: Define condition(s) to include or exclude events from the pipeline additionalProperties: false properties: action: title: Action description: How to handle events matching this criteria oneOf: - label: Allow events matching this criteria enum: - allow - label: Drop events matching this criteria enum: - drop default: allow ui: display_type: radio-group conditional: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine required: - action - conditional - if: properties: type: enum: - sample deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sample description: Sample data at a given rate, retaining only a subset of data events for further processing. additionalProperties: false properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 events will be forwarded and the rest will be dropped minimum: 2 maximum: 10000 default: 10 always_include: title: Always Include description: Certain events may always want to be included (not sampled), regardless of the sample rate. Events matching this criteria will always show up in the results. additionalProperties: type: - string - number dependencies: field: - str_operator str_operator: - field allOf: - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: errorMessage: pattern: Value must be a number type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 pattern: ^\s*-?(\d*\.)?\d+\s*$ required: - value - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix - percent_change_greater - percent_change_greater_or_equal - percent_change_less - percent_change_less_or_equal - value_change_greater - value_change_greater_or_equal - value_change_less - value_change_less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 required: - value - if: title: Operator properties: str_operator: type: string enum: - contains - equal - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix ui: display_type: list-operators title: Operator required: - str_operator then: title: Case sensitive properties: case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive comparison? default: true properties: field: type: string title: Field description: The JSON field to use in a condition to always include in sampling pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - is_metric - is_array - is_boolean - is_empty - is_null - is_number - is_object - is_string - exists - ends_with - starts_with - contains_token_term - contains_token_prefix - regex_match ui: display_type: list-operators title: Operator description: The comparison operator negate: type: boolean default: false title: Negate description: Negate the comparison expression required: - rate - if: properties: type: enum: - drop-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Remove Fields description: Remove fields additionalProperties: false properties: fields: type: array title: Fields description: A list of fields to be removed items: type: string title: Field pattern: ^(?:metadata|\.|(?:metadata|message)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields). Note that dropping an event's message or timestamp is not allowed minItems: 1 required: - fields - if: properties: type: enum: - parse deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Parse description: Parse a specified field using the chosen parser additionalProperties: type: object errorMessage: properties: target_field: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) allOf: - if: properties: parser: enum: - parse_key_value then: properties: options: $ref: "#key-value-parser-options" - if: properties: parser: enum: - parse_timestamp then: properties: options: $ref: "#timestamp-parser-options" required: - options - if: properties: parser: enum: - parse_grok then: properties: options: $ref: "#grok-pattern-parser-options" required: - options - if: properties: parser: enum: - parse_regex then: properties: options: allOf: - $ref: "#regex-parser-options" ui: display_type: regex-pattern required: - options - if: properties: parser: enum: - parse_apache_log then: properties: options: $ref: "#apache-parser-options" required: - options - if: properties: parser: enum: - parse_nginx_log then: properties: options: $ref: "#nginx-parser-options" required: - options - if: properties: parser: enum: - parse_cef then: properties: options: $ref: "#cef-parser-options" - if: properties: parser: enum: - parse_csv then: properties: options: $ref: "#csv-parser-options" properties: field: type: string title: Field description: The JSON field whose value should be parsed. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: oneOf: - type: "null" - type: string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference title: Target field description: The field into which the parsed value should be inserted. Leave blank to insert the parsed data into the original field. default: null parser: oneOf: - label: AWS Cloudfront Access Log group: Line Parsers ui: {} enum: - aws_cloudfront_access_log - label: AWS Cloudwatch Log group: Line Parsers ui: {} enum: - parse_aws_cloudwatch_log_subscription_message - label: AWS Load Balancer Log group: Line Parsers ui: {} enum: - parse_aws_alb_log - label: AWS S3 Access Log group: Line Parsers ui: {} enum: - aws_s3_access_log - label: Apache Log group: Line Parsers ui: {} enum: - parse_apache_log - label: CSV group: Line Parsers ui: {} enum: - parse_csv - label: Common Event Format (CEF) Log group: Line Parsers ui: {} enum: - parse_cef - label: Common Log group: Line Parsers ui: {} enum: - parse_common_log - label: Elasticsearch Log group: Line Parsers ui: {} enum: - elasticsearch_log - label: Golang Log group: Line Parsers ui: {} enum: - golang_log - label: Google Log group: Line Parsers ui: {} enum: - parse_glog - label: Grok Pattern group: Line Parsers ui: {} enum: - parse_grok - label: Integer group: Field Parsers ui: {} enum: - parse_int - label: JSON group: Line Parsers ui: {} enum: - parse_json - label: Key/Value group: Line Parsers ui: {} enum: - parse_key_value - label: Kubernetes Log group: Line Parsers ui: {} enum: - parse_klog - label: Linux Authorization group: Line Parsers ui: {} enum: - parse_linux_authorization - label: Mongo Log group: Line Parsers ui: {} enum: - mongo_log - label: Mongo3 Log group: Line Parsers ui: {} enum: - mongo3_log - label: Nginx Log group: Line Parsers ui: {} enum: - parse_nginx_log - label: PSQL Log group: Line Parsers ui: {} enum: - psql_log - label: Query String group: Field Parsers ui: {} enum: - parse_query_string - label: Regular Expression group: Line Parsers ui: feature_flag: pipeline-parser-regex enum: - parse_regex - label: Syslog group: Line Parsers ui: {} enum: - parse_syslog - label: Timestamp group: Field Parsers ui: {} enum: - parse_timestamp - label: Tokens group: Field Parsers ui: {} enum: - parse_tokens - label: URL group: Field Parsers ui: {} enum: - parse_url - label: User Agent group: Field Parsers ui: {} enum: - parse_user_agent - label: VPC Flow Log group: Line Parsers ui: {} description: Parse VPC logs using default format for fields from v2 to v4 enum: - parse_aws_vpc_flow_log ui: display_type: list errorMessage: must be one of allowed parsers title: Parser description: The kind of parser to use against the input value from "field". required: - field - parser - if: properties: type: enum: - parse-sequentially deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Parse Sequentially description: Parse fields using one or more parsers in order, sending out the event on the first successful match. additionalProperties: false errorMessage: properties: target_field: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) properties: field: type: string title: Field description: The field whose value should be parsed. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: oneOf: - type: "null" - type: string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference title: Target field description: The field into which the parsed value should be inserted. Leave blank to insert the parsed data into the original field. default: null parsers: type: array title: List of parsers description: The list of parsers to use in order against the input valuefrom "field", short-circuiting on the first successful match. items: type: object allOf: - if: properties: parser: enum: - parse_key_value then: properties: options: $ref: "#key-value-parser-options" - if: properties: parser: enum: - parse_timestamp then: properties: options: $ref: "#timestamp-parser-options" required: - options - if: properties: parser: enum: - parse_grok then: properties: options: $ref: "#grok-pattern-parser-options" required: - options - if: properties: parser: enum: - parse_regex then: properties: options: allOf: - $ref: "#regex-parser-options" ui: display_type: regex-pattern required: - options - if: properties: parser: enum: - parse_apache_log then: properties: options: $ref: "#apache-parser-options" required: - options - if: properties: parser: enum: - parse_nginx_log then: properties: options: $ref: "#nginx-parser-options" required: - options - if: properties: parser: enum: - parse_cef then: properties: options: $ref: "#cef-parser-options" - if: properties: parser: enum: - parse_csv then: properties: options: $ref: "#csv-parser-options" properties: label: oneOf: - type: "null" - type: string maxLength: 20 title: Title description: An arbitrary name you choose to identify the results of this processor for use when connecting them to other components. parser: oneOf: - label: AWS Cloudfront Access Log group: Line Parsers ui: {} enum: - aws_cloudfront_access_log - label: AWS Cloudwatch Log group: Line Parsers ui: {} enum: - parse_aws_cloudwatch_log_subscription_message - label: AWS Load Balancer Log group: Line Parsers ui: {} enum: - parse_aws_alb_log - label: AWS S3 Access Log group: Line Parsers ui: {} enum: - aws_s3_access_log - label: Apache Log group: Line Parsers ui: {} enum: - parse_apache_log - label: CSV group: Line Parsers ui: {} enum: - parse_csv - label: Common Event Format (CEF) Log group: Line Parsers ui: {} enum: - parse_cef - label: Common Log group: Line Parsers ui: {} enum: - parse_common_log - label: Elasticsearch Log group: Line Parsers ui: {} enum: - elasticsearch_log - label: Golang Log group: Line Parsers ui: {} enum: - golang_log - label: Google Log group: Line Parsers ui: {} enum: - parse_glog - label: Grok Pattern group: Line Parsers ui: {} enum: - parse_grok - label: Integer group: Field Parsers ui: {} enum: - parse_int - label: JSON group: Line Parsers ui: {} enum: - parse_json - label: Key/Value group: Line Parsers ui: {} enum: - parse_key_value - label: Kubernetes Log group: Line Parsers ui: {} enum: - parse_klog - label: Linux Authorization group: Line Parsers ui: {} enum: - parse_linux_authorization - label: Mongo Log group: Line Parsers ui: {} enum: - mongo_log - label: Mongo3 Log group: Line Parsers ui: {} enum: - mongo3_log - label: Nginx Log group: Line Parsers ui: {} enum: - parse_nginx_log - label: PSQL Log group: Line Parsers ui: {} enum: - psql_log - label: Query String group: Field Parsers ui: {} enum: - parse_query_string - label: Regular Expression group: Line Parsers ui: feature_flag: pipeline-parser-regex enum: - parse_regex - label: Syslog group: Line Parsers ui: {} enum: - parse_syslog - label: Timestamp group: Field Parsers ui: {} enum: - parse_timestamp - label: Tokens group: Field Parsers ui: {} enum: - parse_tokens - label: URL group: Field Parsers ui: {} enum: - parse_url - label: User Agent group: Field Parsers ui: {} enum: - parse_user_agent - label: VPC Flow Log group: Line Parsers ui: {} description: Parse VPC logs using default format for fields from v2 to v4 enum: - parse_aws_vpc_flow_log ui: display_type: list errorMessage: must be one of allowed parsers title: Parser description: The kind of parser to use against the input value from "field". _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. default: "" required: - parser - _output_name additionalProperties: type: object minItems: 1 ui: display_type: multi-output-map unique_parser_and_options: true errorMessage: uniqueItemProperties: parser can not be defined more than once required: - field - parsers - if: properties: type: enum: - route deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Route description: Route data based on whether it matches (or doesn't match) logical comparisons additionalProperties: false properties: conditionals: type: array minItems: 1 title: Conditionals description: A list of complete conditionals, including expressions and label items: type: object additionalProperties: false title: Conditional description: A list of conditions and a label properties: conditional: $ref: "#condition-group" label: type: string minLength: 1 maxLength: 255 title: Label description: An arbitrary name you choose to identify the results of this processor for use when connecting them to other components. _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. default: "" required: - conditional - label - _output_name required: - conditionals - if: properties: type: enum: - stringify deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Stringify description: Stringify messages as JSON additionalProperties: false - if: properties: type: enum: - compact-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Compact Fields description: Remove empty values from a list of fields additionalProperties: false properties: fields: type: array title: Fields description: A list of fields to remove empty values from items: type: string title: Field description: Field to remove empty value from pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference minItems: 1 options: type: object additionalProperties: false title: Compact Options description: Additional options for removing empty values from fields properties: compact_array: type: boolean title: Compact Array description: Remove empty arrays from a field default: true compact_object: type: boolean title: Compact Object description: Remove empty objects from a field default: true required: - fields - if: properties: type: enum: - flatten-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Flatten Fields description: Flattens the object or array value of a field into a single level representation additionalProperties: false properties: fields: type: array title: Fields description: Define specific nested objects you wish to flatten. This will not flatten the entire object, just the specified fields. Leave blank to flatten the entire object items: type: string title: Field description: Field to flatten pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference options: type: object additionalProperties: false title: Flatten Options description: Additional options for flattening fields properties: delimiter: type: string default: _ title: Delimiter description: When flattening, the nested field labels will be appended to the higher level field names along with a delimiter to allow understanding of the original structure. The default is an underscore (_) to ensure no collisions with the . notation used for field identifiers minLength: 1 maxLength: 1 - if: properties: type: enum: - dedupe deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Dedupe description: Dedupe (remove duplicates) from the data stream additionalProperties: false properties: number_of_events: type: number description: Number of events to compare across title: Number of Events minimum: 2 maximum: 5000 default: 5000 comparison_type: type: string enum: - Ignore - Match description: "Match: Only compares across the fields which are specified by the user.Ignore: Compares everything but the fields specified by the user." title: Comparison Type default: Match fields: type: array title: Fields description: A list of fields on which to base deduping items: type: string title: Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference minItems: 1 required: - number_of_events - comparison_type - fields - if: properties: type: enum: - encrypt-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Encrypt Field description: Encrypts the value of the provided field additionalProperties: false allOf: - if: properties: algorithm: enum: - AES-256-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key properties: field: type: string title: Field description: Field to encrypt. The value of the field must be a primitive (string, number, boolean) pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference algorithm: oneOf: - label: AES-256-CFB (key = 32 characters, iv = 16 characters) description: AES-256-CFB (key = 32 characters, iv = 16 characters) enum: - AES-256-CFB - label: AES-192-CFB (key = 24 characters, iv = 16 characters) description: AES-192-CFB (key = 24 characters, iv = 16 characters) enum: - AES-192-CFB - label: AES-128-CFB (key = 16 characters, iv = 16 characters) description: AES-128-CFB (key = 16 characters, iv = 16 characters) enum: - AES-128-CFB - label: AES-256-OFB (key = 32 characters, iv = 16 characters) description: AES-256-OFB (key = 32 characters, iv = 16 characters) enum: - AES-256-OFB - label: AES-192-OFB (key = 24 characters, iv = 16 characters) description: AES-192-OFB (key = 24 characters, iv = 16 characters) enum: - AES-192-OFB - label: AES-128-OFB (key = 16 characters, iv = 16 characters) description: AES-128-OFB (key = 16 characters, iv = 16 characters) enum: - AES-128-OFB - label: AES-256-CTR (key = 32 characters, iv = 16 characters) description: AES-256-CTR (key = 32 characters, iv = 16 characters) enum: - AES-256-CTR - label: AES-192-CTR (key = 24 characters, iv = 16 characters) description: AES-192-CTR (key = 24 characters, iv = 16 characters) enum: - AES-192-CTR - label: AES-128-CTR (key = 16 characters, iv = 16 characters) description: AES-128-CTR (key = 16 characters, iv = 16 characters) enum: - AES-128-CTR - label: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-PKCS7 - label: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-PKCS7 - label: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-PKCS7 - label: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ANSIX923 - label: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ANSIX923 - label: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ANSIX923 - label: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO7816 - label: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO7816 - label: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO7816 - label: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO10126 - label: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO10126 - label: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO10126 ui: display_type: list title: Encryption algorithm description: The encryption algorithm to use key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 32 iv_field: type: string title: Initialization vector (IV) field description: The field in which to store the generated initialization vector, IV. Each encrypted value will have a unique IV pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference encode_raw_bytes: type: boolean title: Encode encrypted field and IV as Base 64 text description: Encode the encrypted value and generated initialization vector as Base64 text default: true required: - field - algorithm - key - iv_field - if: properties: type: enum: - decrypt-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Decrypt Field additionalProperties: false description: Decrypts the value of the provided field allOf: - if: properties: algorithm: enum: - AES-256-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key properties: field: type: string title: Field description: Field to decrypt. The value of the field must be a string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference algorithm: oneOf: - label: AES-256-CFB (key = 32 characters, iv = 16 characters) description: AES-256-CFB (key = 32 characters, iv = 16 characters) enum: - AES-256-CFB - label: AES-192-CFB (key = 24 characters, iv = 16 characters) description: AES-192-CFB (key = 24 characters, iv = 16 characters) enum: - AES-192-CFB - label: AES-128-CFB (key = 16 characters, iv = 16 characters) description: AES-128-CFB (key = 16 characters, iv = 16 characters) enum: - AES-128-CFB - label: AES-256-OFB (key = 32 characters, iv = 16 characters) description: AES-256-OFB (key = 32 characters, iv = 16 characters) enum: - AES-256-OFB - label: AES-192-OFB (key = 24 characters, iv = 16 characters) description: AES-192-OFB (key = 24 characters, iv = 16 characters) enum: - AES-192-OFB - label: AES-128-OFB (key = 16 characters, iv = 16 characters) description: AES-128-OFB (key = 16 characters, iv = 16 characters) enum: - AES-128-OFB - label: AES-256-CTR (key = 32 characters, iv = 16 characters) description: AES-256-CTR (key = 32 characters, iv = 16 characters) enum: - AES-256-CTR - label: AES-192-CTR (key = 24 characters, iv = 16 characters) description: AES-192-CTR (key = 24 characters, iv = 16 characters) enum: - AES-192-CTR - label: AES-128-CTR (key = 16 characters, iv = 16 characters) description: AES-128-CTR (key = 16 characters, iv = 16 characters) enum: - AES-128-CTR - label: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-PKCS7 - label: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-PKCS7 - label: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-PKCS7 - label: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ANSIX923 - label: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ANSIX923 - label: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ANSIX923 - label: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO7816 - label: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO7816 - label: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO7816 - label: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO10126 - label: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO10126 - label: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO10126 ui: display_type: list title: Decryption algorithm description: The algorithm with which the data was encrypted key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 32 iv_field: type: string title: Initialization vector (IV) field description: The field from which to read the initialization vector, IV pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference decode_raw_bytes: type: boolean title: Decode the encrypted value and IV as Base 64 text description: Decode the encrypted value and initialization vector as Base64 text default: true required: - field - algorithm - key - iv_field - if: properties: type: enum: - js-script deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Script Execution description: Use JavaScript to reshape and transform your data. You can combine multiple actions like filtering, dropping, mapping, and casting inside of a single js script. additionalProperties: true properties: script: type: string title: Script ui: display_type: code-editor options: language: js description: The JavaScript representation of the event transformation. minLength: 1 maxLength: 10000 default: > // Modify the event using a subset of the JavaScript language. // The function must return the modified event function processEvent(message, metadata, timestamp, annotations) { return message } - if: properties: type: enum: - unroll deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Unroll description: Takes an array of events and emits them all as individual events. additionalProperties: false properties: field: type: string title: Field description: The JSON field name that contains an array of events pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference values_only: type: boolean title: Values only? description: When enabled, the values from the specified array field will be emitted as new events. Otherwise, the original event will be duplicated for each value in the array field, with the unrolled value present in the field specified. default: true required: - field - if: properties: type: enum: - vrl deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: VRL description: Modify your observability data using Vector Remap Language (VRL) additionalProperties: false properties: source: type: string title: VRL Source description: VRL code to execute for each event ui: display_type: code-editor options: language: vrl minLength: 1 default: .message = .message required: - source - if: properties: type: enum: - reduce deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Reduce description: Combine multiple events over time into one based on a set of criteria additionalProperties: false properties: duration_ms: type: number maximum: 7200000 title: Duration description: The amount of time (in milliseconds) to allow streaming events to accumulate into a single "reduced" event. The process repeats indefinitely, or until an "ends when" condition is satisfied. ui: display_type: milliseconds-to-human-readable default: 30000 max_events: type: number minimum: 1 maximum: 100000 title: Max Events description: The maximum number of events that can be included in a time window (specified by Duration above). The reduce operation will stop once it has reached this number of events, regardless of whether the Duration seconds have elapsed. group_by: type: array minItems: 1 title: Group By Field Path items: type: string title: Field description: The dot-notated property path errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ description: Before reducing, group events based on matching data from each of these field paths. Supports nesting via dot-notation. date_formats: type: array title: Date Formats description: Describes which root-level properties are dates, and their expected format. Dot-notation is supported, but nested field lookup paths will be an error. minItems: 1 items: type: object additionalProperties: false properties: field: type: string title: Date Field Path description: Specifies a root-level path property that contains a date value. maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ format: type: string title: Date Format description: The template describing the date format format: strftime_datetime errorMessage: must be a valid strftime datetime format maxLength: 200 required: - field - format merge_strategies: type: array title: Merge Strategy per Field minItems: 1 description: Specify merge strategies for individual root-level properties. Dot-notation is supported, but nested field lookup paths will be an error. items: type: object additionalProperties: false properties: field: type: string title: Field Path description: This is a root-level path property to apply a merge strategy to its value maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ strategy: oneOf: - label: Array description: Append each value onto an array enum: - array - label: Smallest array description: Keep the smallest array within each reduce time window enum: - shortest_array - label: Largest array description: Keep the largest array within each reduce time window enum: - longest_array - label: Unique Array description: All unique values go into a flattened array enum: - flat_unique - label: Join with space description: Concatenate into a single space-delimited string enum: - concat - label: Join with newline description: Concatenate into a single newline-delimited string enum: - concat_newline - label: Join description: Concatenate into a single string without using a delimiter enum: - concat_raw - label: Keep first description: Keep only the first value and discard the rest enum: - discard - label: Keep last description: Keep only the last non-null value enum: - retain - label: Min value description: Keep the minimum numeric value within each reduce time window enum: - min - label: Max value description: Keep the maximum numeric value within each reduce time window enum: - max - label: Sum description: Add numeric values together enum: - sum ui: display_type: list errorMessage: must be one of the allowed merge strategies title: Merge Strategy description: The merge strategy to be used for the specified property required: - field - strategy flush_condition: title: Event Flush Condition description: Force accumulated event reduction to flush the result when a conditional expression evaluates to true on an inbound event. allOf: - type: object properties: when: type: string enum: - none - starts_when - ends_when default: none ui: display_type: list errorMessage: "" title: Flush When required: - when - if: properties: when: not: enum: - none then: properties: conditional: allOf: - $ref: "#condition-group" ui: display_type: rules-engine required: - conditional - if: properties: type: enum: - map-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Map Fields description: Maps data from one field to another, either by moving or copying. additionalProperties: false properties: mappings: type: array title: Mappings description: A list of field mappings. Mappings are applied in the order they are defined. minItems: 1 ui: display_type: sortable items: type: object properties: source_field: type: string title: Source field description: The field to copy data from pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: type: string title: Target field description: The field to copy data into pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference drop_source: type: boolean title: Drop source? description: When enabled, the source field is dropped after the data is copied to the target field. Otherwise, it is preserved. default: false overwrite_target: type: boolean title: Overwrite target? description: When enabled, any existing data in the target field is overwritten. Otherwise, the target field will be preserved and this mapping will have no effect. default: false required: - source_field - target_field - drop_source - overwrite_target if: properties: drop_source: enum: - true required: - drop_source then: properties: source_field: type: string title: Source field description: The field to copy data from pattern: ^(?:metadata|\.|(?:metadata|message)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields). Note that dropping an event's message or timestamp is not allowed required: - source_field required: - mappings - if: properties: type: enum: - metrics-tag-cardinality-limit deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Tag Cardinality Limit (Metrics) description: Limits the cardinality of metric events by either dropping events or tags that exceed a specified value limit additionalProperties: false allOf: - if: title: Mode properties: mode: type: string enum: - exact then: title: Value Limit properties: value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags minimum: 1 maximum: 500 required: - value_limit - if: title: Mode properties: mode: type: string enum: - probabilistic then: title: Value Limit properties: value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags minimum: 1 maximum: 5000 required: - value_limit properties: tags: type: array title: Tags description: A list of tags to apply cardinality limits. If none are provided, all tags will be considered. items: type: string title: Tag minLength: 1 maxLength: 100 minItems: 0 maxItems: 10 exclude_tags: type: array title: Exclude Tags description: A list of tags to explicitly exclude from cardinality limits. items: type: string title: Tag minLength: 1 maxLength: 100 minItems: 0 maxItems: 10 action: type: string title: Action description: The action to take when a tag's cardinality exceeds the value limit enum: - drop_tag - drop_event default: drop_event value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags mode: type: string title: Mode description: The method to used to reduce tag value cardinality enum: - exact - probabilistic default: exact - if: properties: type: enum: - aggregate deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Aggregate description: Aggregates multiple events into a single event using either a tumbling interval window or a sliding interval window. additionalProperties: false properties: group_by: type: array minItems: 1 maxItems: 100 title: Group By Field Path description: When aggregating, group events based on matching values from each of these field paths. Supports nesting via dot-notation. items: type: string title: Field description: The dot-notated property path errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ ui: default_hint: - .name - .namespace - .tags evaluate: additionalProperties: type: string title: Evaluate description: How events should be aggregated within an interval. allOf: - if: properties: operation: enum: - CUSTOM then: properties: script: type: string title: Script ui: display_type: code-editor options: language: js description: The JavaScript representation of an event aggregation. minLength: 1 maxLength: 2048 default: > // Perform custom aggregation of the current event into the accumulated event. // The accumulated event is persisted across the entire aggregation window. // The function must return the accumulated event, which will be emitted when // the aggregation window elapses. function aggregateEvent(accum, event, metadata, timestamp, annotations) { return accum } required: - script properties: operation: title: Operation oneOf: - label: add description: Adds new values together if they are of the same type. For counters and gauges the values are added together. For sets, this is a union of the elements. For distribution, the samples are concatenated. For histograms, values in the same bucket will be added together. enum: - DEFAULT - label: sum description: Values are added together. Works for the GAUGE and COUNT metric types. enum: - SUM - label: minimum description: Get the minimum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MIN - label: maximum description: Get the maximum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MAX - label: average description: Values are averaged together. Works for the GAUGE and COUNT metric types. enum: - AVG - label: set intersection description: The aggregate produces a set of unique values. enum: - SET_INTERSECTION - label: distribution concatenation description: The aggregate produces a list of values combined together. enum: - DIST_CONCAT - label: custom description: Use a custom script to perform aggregation. enum: - CUSTOM ui: display_type: list required: - operation window: default: {} additionalProperties: type: integer title: Window Type description: Sliding windows can overlap, whereas tumbling windows are disjoint. For example, a tumbling window has a fixed time span and any events that fall within the "interval" will be used in the aggregate. Whereas in a sliding window, the aggregation occurs every "window duration" seconds after an event is encountered. allOf: - if: properties: type: enum: - sliding required: - type then: properties: window_min: type: integer title: Slide Duration (seconds) description: The amount of time to wait before creating a new sliding window for future events. minimum: 1 maximum: 200 default: 5 required: - window_min properties: type: type: string title: Type enum: - tumbling - sliding default: tumbling interval: type: integer title: Interval (seconds) description: The interval over which events are aggregated in seconds. default: 5 minimum: 1 maximum: 90000 required: - type - interval event_timestamp: type: string title: Event Timestamp description: The path to a field on the event that contains a epoch timestamp value. If an event does not have a timestamp field, events will be associated to the wall clock value when the event is processed. default: timestamp maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ conditional: ui: display_type: rules-engine title: Event Flush Condition description: Force accumulated event reduction to flush the result when a conditional expression evaluates to true on an inbound event. allOf: - $ref: "#condition-group" required: - evaluate - window - if: properties: type: enum: - event-to-metric deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Event to Metric description: Allows conversion between arbitrary events and a Metric. additionalProperties: false properties: metric_name: type: string title: Metric Name description: The machine name of the metric to emit. minLength: 1 maxLength: 128 pattern: ^[a-zA-Z][a-zA-Z0-9_:]*$ errorMessage: must be a valid metric name metric_kind: title: Kind description: The kind of metric to emit, Absolute or Incremental. Absolute metrics represent a complete value, and will generally replace an existing value for the metric in the target destination. Incremental metrics represent an additive value which is aggregated in the target destination to produce a new value. oneOf: - label: Incremental enum: - incremental - label: Absolute enum: - absolute default: incremental ui: display_type: radio-group metric_type: title: Type oneOf: - label: Counter enum: - counter - label: Sum enum: - sum - label: Gauge enum: - gauge default: counter ui: display_type: list value: title: Value description: The value of this metric. additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value then: title: Value properties: value: type: number title: Value description: The value. required: - value properties: value_type: title: Value type description: Either a value from an event field, or a new value input. oneOf: - label: Value from event field enum: - field - label: New value enum: - value default: field ui: display_type: radio-group value: type: - number - string required: - value_type namespace: title: Namespace description: The namespace for this metric. additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 128 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit the namespace. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: type: - string - "null" required: - value_type tags: type: array maxItems: 10 title: Tags description: A set of tags (also called labels) to apply to the metric event. items: type: object title: Tag description: A tag to apply to the metric event. allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value of this tag. Note that fields with highly-variable values will result in high-cardinality metrics, which may impact storage or cost in downstream destinations. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference fallback: type: string title: Fallback value description: The value to use for this tag when the field value is missing. minLength: 1 maxLength: 512 required: - value - if: properties: value_type: enum: - value then: title: Value properties: value: type: string title: Value description: The value for this tag. minLength: 1 maxLength: 128 required: - value properties: name: type: string title: Name description: The tag name pattern: ^[a-zA-Z][a-zA-Z0-9_]*$ minLength: 1 maxLength: 128 errorMessage: must be a valid metric tag name value_type: title: Value type description: Either a value from an event field, or a new value input. oneOf: - label: Value from event field enum: - field - label: New value enum: - value default: field ui: display_type: radio-group value: type: string required: - name - value_type - value additionalProperties: type: string required: - metric_name - metric_kind - metric_type - value - namespace - if: properties: type: enum: - clustering deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Clustering description: Finds clusters of similar values additionalProperties: false properties: cluster_field: type: string title: Field description: Field to cluster values on pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference default: .message similarity_threshold: type: number title: Similarity threshold description: A threshold from 0.0 (not similar) to 1.0 (exactly similar) that determines how close two values need to be to be consider as being part of the same cluster default: 0.4 minimum: 0 maximum: 1 max_clusters: type: integer title: Maximum cluster description: Maximum number of clusters kept cached by the transform default: 1000 minimum: 1 maximum: 2000 max_node_depth: type: integer title: Maximum tree depth description: Maximum depth of the prefix tree default: 4 minimum: 2 maximum: 8 max_children: type: integer title: Maximum children default: 100 minimum: 2 maximum: 200 description: Maximum number of children per node of the prefix tree required: - cluster_field - if: properties: type: enum: - redact deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Redact description: Detect and anonymize sensitive data in logs and metrics additionalProperties: false properties: field: type: string title: Field description: Detect patterns in the specified field. If the field is empty, the processor will look for patterns across all the fields. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 256 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference mask_patterns: type: array title: Redact Patterns description: List of PII patterns to detect and mask minItems: 1 maxItems: 20 items: type: object additionalProperties: type: string title: Redact Pattern description: A PII pattern to detect and/or anonymize allOf: - if: properties: action: enum: - Replace required: - action then: properties: replacement: type: string title: Replacement description: "The text to replace the detected pattern. Example: 000-00-0000" minLength: 1 maxLength: 512 required: - replacement - if: properties: action: enum: - Hash required: - action then: properties: algorithm: title: Algorithm description: Hash and encode (base16 or hex) matched PII data using the specified algorithm oneOf: - label: sha2 (sha-512_256) enum: - sha2 - label: sha3 (sha3-512) enum: - sha3 ui: display_type: list required: - algorithm - if: properties: name: enum: - custom required: - name then: properties: custom_options: allOf: - type: object additionalProperties: false properties: pattern: type: string title: Pattern description: The custom pattern minLength: 1 maxLength: 4096 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching default: true multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively default: false match_newline: type: boolean title: Dot all mode description: Dot matches newline character default: false crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" default: false ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines default: false required: - pattern title: Replace Expression description: The regular expression used to find replacements. Named capture groups are supported, e.g. `(?P<key>\w+):(?P<value>.+)`. ui: display_type: regex-pattern required: - custom_options properties: name: title: Pattern description: The PII pattern to match oneOf: - label: Credit card number enum: - credit_card - label: Email address enum: - email_address - label: IPv4 address enum: - ipv4_address - label: US or Canada phone number enum: - us_canada_phone_number - label: US social security number enum: - us_social_security_number - label: Custom pattern ui: feature_flag: pipeline-parser-regex enum: - custom ui: display_type: list action: type: string title: Action description: Action to take on the detected PII data enum: - Hash - Replace - Detect Only default: Replace required: - name - action required: - mask_patterns - if: properties: type: enum: - throttle deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Throttle description: Throttle (rate-limit) events passing through this component additionalProperties: false properties: threshold: type: number minimum: 1 maximum: 10000000 title: Limit description: The number of events to allow over the time window. If a Key Field is specified, the limit will be applied to events for each unique value of that field. default: 1 window_ms: type: number minimum: 1000 maximum: 2592000000 title: Window description: The time window over which the configured limit is applied ui: display_type: milliseconds-to-human-readable default: 5000 key_field: type: string title: Key Field description: When configured, the value from this field will be used to group events into separate buckets for throttling. If left blank, or if the event does not contain a value in this field, the event will be throttled as part of the default bucket. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference exclude: allOf: - $ref: "#condition-group" title: Exclude description: When configured, events matching this condition will not be throttled. ui: display_type: rules-engine required: - threshold - window_ms - if: properties: type: enum: - set-timestamp deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Set Timestamp description: Parse a list of fields using a chosen time format. First match will override the default timestamp of the event. additionalProperties: false properties: parsers: type: array title: List of Fields and Parsers description: The list of fields and parsers to use in order of priority, short-circuiting on the first successful match. ui: display_type: multi-output-map options: sortable: false items: type: object properties: field: type: string title: Field description: The field whose value will be parsed by the chosen date pattern pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference options: $ref: "#timestamp-parser-options" required: - field - options minItems: 1 required: - parsers - if: properties: type: enum: - data-profiler deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Data Profiler description: Profile the data sent through your pipeline. The system will generate annotations such as event type and total bytes to be used downstream. It will also normalize the app/host/level/line fields before sending the event downstream. A profile of the data including total, events, event size, log composition, and message templates will be generated as well. additionalProperties: false properties: app_fields: type: array title: App Path(s) description: A list of paths to look for the app value minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: App description: The path to the app maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .app - .application - .container host_fields: type: array title: Host Path(s) description: A list of paths to look for the host value minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Host description: The path to the host maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .host - .hostname level_fields: type: array title: Level Path(s) description: A list of paths to look for the log level minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Level description: The path to the level maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .level - .log_level line_fields: type: array title: Line Path(s) description: A list of paths to look for the event line minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Line description: The path to the line maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .line - .message label_fields: type: array title: Label Path(s) description: A list of paths to look for the event labels maxItems: 10 uniqueItems: true items: type: string title: Label description: The path to the labels maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .labels field_summary: type: boolean default: false title: Field Summary description: If enabled the profiler will attempt to summarize the values in fields of events passing through the node. ui: display_type: hidden ff_default_value: pipeline-node-transform-field-summary required: - app_fields - host_fields - level_fields - line_fields - if: properties: type: enum: - replace deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Replace description: Replace parts of a string using a regular expression or static string additionalProperties: type: object allOf: - if: properties: replacement_type: enum: - regex required: - replacement_type then: properties: options: allOf: - type: object additionalProperties: false properties: pattern: type: string minLength: 1 maxLength: 4096 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching default: true multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively default: false match_newline: type: boolean title: Dot all mode description: Dot matches newline character default: false crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" default: false ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines default: false required: - pattern title: Replace Expression description: The regular expression used to find replacements. Named capture groups are supported, e.g. `(?P<key>\w+):(?P<value>.+)`. ui: display_type: regex-pattern replace_with: type: string minLength: 1 maxLength: 4096 title: Replacement String description: The replacement string can be static or optionally contain capture group references. Capture groups can be positional or named. For example, to switch the position of two strings in the source string `onetwo`, the pattern could be `(?Pone)(?P.+)`, and the replacement would be `$two$one`. To do the same thing with positional capture groups, a pattern of `(\w{3})(\w{3})`, and a replacement of `$2$1` would have the same result.

To render a literal dollar sign `$` in the replacement, escape it with `$$`. So, if you had a capture group named `price`, to prepend a dollar sign to the replacement, you would use `$$$price`. required: - options - replace_with - if: properties: replacement_type: enum: - string required: - replacement_type then: properties: string_pattern: type: string title: String Pattern minLength: 1 maxLength: 4096 description: A static, case-sensitive string, which will be searched and replaced. replace_with: type: string minLength: 1 maxLength: 4096 title: Replacement String description: The static string used in each replacement. required: - string_pattern - replace_with properties: source_field: type: string title: Source Field description: The field that contains a string for replacement to operate on. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference max_times: type: number title: Maximum Replacements description: The maximum number of replacements to be made. If not specified, all occurrences will be replaced. minimum: 1 replacement_type: title: Replacement Type description: Specifies how the source string will be searched for replacement. ui: display_type: list errorMessage: must be one of the available pattern types oneOf: - label: String description: Replace a specified string one or more times. enum: - string - label: Regular Expression description: Replace using a regular expression. enum: - regex required: - source_field - replacement_type - if: properties: type: enum: - trace-sampling deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Trace Sampling description: Performs sampling of traces per a given rate using the Trace ID as the unique identifier. If a trace is rate selected, all spans of the trace are forwarded. If the Trace ID cannot be found, the event is dropped. allOf: - if: properties: sample_type: enum: - head then: properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 unique traces will be forwarded and all others will be dropped default: 10 minimum: 2 maximum: 100000 parent_span_id_field: type: "null" conditionals: type: "null" required: - rate - if: properties: sample_type: enum: - tail then: properties: parent_span_id_field: type: string default: .parent_span_id title: Parent Span ID description: The field to use as the parent span identifier pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference conditionals: type: array minItems: 1 title: Conditionals description: A list of complete conditionals, including expressions and a sample rate to maintain items: type: object additionalProperties: false title: Conditional description: A list of conditions and a sample rate to maintain properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 unique traces will be forwarded and all others will be dropped. A `rate = 1` means keep all traces that match the condition. minimum: 1 maximum: 100000 default: 10 conditional: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. required: - rate - conditional required: - parent_span_id_field - conditionals properties: sample_type: type: string description: The type of trace sampling to apply. enum: - head title: Sample Type default: head trace_id_field: type: string default: .trace_id title: Trace ID description: The field to use as the identifier of a trace. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - sample_type - trace_id_field properties: type: type: string enum: - filter - sample - drop-fields - parse - parse-sequentially - route - stringify - compact-fields - flatten-fields - dedupe - encrypt-fields - decrypt-fields - js-script - unroll - vrl - reduce - map-fields - metrics-tag-cardinality-limit - aggregate - event-to-metric - clustering - redact - throttle - set-timestamp - data-profiler - replace - trace-sampling deploy_type: type: string enum: - saas - local default: saas title: type: string pattern: ^[^<>"]+$ errorMessage: invalid title maxLength: 512 description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description user_config: type: object generation_id: type: integer inputs: type: array items: type: string description: A list of inputs for the given transform. Typically this is empty when creating a new transform via the UI, but is leveraged by the terraform provider when managing pipeline state. required: - type - title - user_config - generation_id required: true parameters: - schema: type: string format: uuid in: path name: transform_group_id required: true - schema: type: string format: uuid in: path name: transform_group_transform_id required: true responses: "200": description: Default Response /transform-group/{transform_group_id}/transform: post: tags: - Processor Groups description: Create a Processor in a Processor Group requestBody: content: application/json: schema: type: object allOf: - if: properties: type: enum: - filter deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Filter description: Define condition(s) to include or exclude events from the pipeline additionalProperties: false properties: action: title: Action description: How to handle events matching this criteria oneOf: - label: Allow events matching this criteria enum: - allow - label: Drop events matching this criteria enum: - drop default: allow ui: display_type: radio-group conditional: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine required: - action - conditional - if: properties: type: enum: - sample deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Sample description: Sample data at a given rate, retaining only a subset of data events for further processing. additionalProperties: false properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 events will be forwarded and the rest will be dropped minimum: 2 maximum: 10000 default: 10 always_include: title: Always Include description: Certain events may always want to be included (not sampled), regardless of the sample rate. Events matching this criteria will always show up in the results. additionalProperties: type: - string - number dependencies: field: - str_operator str_operator: - field allOf: - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: errorMessage: pattern: Value must be a number type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 pattern: ^\s*-?(\d*\.)?\d+\s*$ required: - value - if: title: Operator properties: str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix - percent_change_greater - percent_change_greater_or_equal - percent_change_less - percent_change_less_or_equal - value_change_greater - value_change_greater_or_equal - value_change_less - value_change_less_or_equal ui: display_type: list-operators title: Operator description: The comparison operator required: - str_operator then: title: Value properties: value: type: - string - number title: Value description: The value used in the expression, ie the "operand" minLength: 1 required: - value - if: title: Operator properties: str_operator: type: string enum: - contains - equal - ends_with - starts_with - drain_matches - contains_token_term - contains_token_prefix ui: display_type: list-operators title: Operator required: - str_operator then: title: Case sensitive properties: case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive comparison? default: true properties: field: type: string title: Field description: The JSON field to use in a condition to always include in sampling pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference str_operator: type: string enum: - greater - greater_or_equal - less - less_or_equal - equal - contains - is_ip_in_cidr_range - is_metric - is_array - is_boolean - is_empty - is_null - is_number - is_object - is_string - exists - ends_with - starts_with - contains_token_term - contains_token_prefix - regex_match ui: display_type: list-operators title: Operator description: The comparison operator negate: type: boolean default: false title: Negate description: Negate the comparison expression required: - rate - if: properties: type: enum: - drop-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Remove Fields description: Remove fields additionalProperties: false properties: fields: type: array title: Fields description: A list of fields to be removed items: type: string title: Field pattern: ^(?:metadata|\.|(?:metadata|message)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields). Note that dropping an event's message or timestamp is not allowed minItems: 1 required: - fields - if: properties: type: enum: - parse deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Parse description: Parse a specified field using the chosen parser additionalProperties: type: object errorMessage: properties: target_field: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) allOf: - if: properties: parser: enum: - parse_key_value then: properties: options: $ref: "#key-value-parser-options" - if: properties: parser: enum: - parse_timestamp then: properties: options: $ref: "#timestamp-parser-options" required: - options - if: properties: parser: enum: - parse_grok then: properties: options: $ref: "#grok-pattern-parser-options" required: - options - if: properties: parser: enum: - parse_regex then: properties: options: allOf: - $ref: "#regex-parser-options" ui: display_type: regex-pattern required: - options - if: properties: parser: enum: - parse_apache_log then: properties: options: $ref: "#apache-parser-options" required: - options - if: properties: parser: enum: - parse_nginx_log then: properties: options: $ref: "#nginx-parser-options" required: - options - if: properties: parser: enum: - parse_cef then: properties: options: $ref: "#cef-parser-options" - if: properties: parser: enum: - parse_csv then: properties: options: $ref: "#csv-parser-options" properties: field: type: string title: Field description: The JSON field whose value should be parsed. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: oneOf: - type: "null" - type: string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference title: Target field description: The field into which the parsed value should be inserted. Leave blank to insert the parsed data into the original field. default: null parser: oneOf: - label: AWS Cloudfront Access Log group: Line Parsers ui: {} enum: - aws_cloudfront_access_log - label: AWS Cloudwatch Log group: Line Parsers ui: {} enum: - parse_aws_cloudwatch_log_subscription_message - label: AWS Load Balancer Log group: Line Parsers ui: {} enum: - parse_aws_alb_log - label: AWS S3 Access Log group: Line Parsers ui: {} enum: - aws_s3_access_log - label: Apache Log group: Line Parsers ui: {} enum: - parse_apache_log - label: CSV group: Line Parsers ui: {} enum: - parse_csv - label: Common Event Format (CEF) Log group: Line Parsers ui: {} enum: - parse_cef - label: Common Log group: Line Parsers ui: {} enum: - parse_common_log - label: Elasticsearch Log group: Line Parsers ui: {} enum: - elasticsearch_log - label: Golang Log group: Line Parsers ui: {} enum: - golang_log - label: Google Log group: Line Parsers ui: {} enum: - parse_glog - label: Grok Pattern group: Line Parsers ui: {} enum: - parse_grok - label: Integer group: Field Parsers ui: {} enum: - parse_int - label: JSON group: Line Parsers ui: {} enum: - parse_json - label: Key/Value group: Line Parsers ui: {} enum: - parse_key_value - label: Kubernetes Log group: Line Parsers ui: {} enum: - parse_klog - label: Linux Authorization group: Line Parsers ui: {} enum: - parse_linux_authorization - label: Mongo Log group: Line Parsers ui: {} enum: - mongo_log - label: Mongo3 Log group: Line Parsers ui: {} enum: - mongo3_log - label: Nginx Log group: Line Parsers ui: {} enum: - parse_nginx_log - label: PSQL Log group: Line Parsers ui: {} enum: - psql_log - label: Query String group: Field Parsers ui: {} enum: - parse_query_string - label: Regular Expression group: Line Parsers ui: feature_flag: pipeline-parser-regex enum: - parse_regex - label: Syslog group: Line Parsers ui: {} enum: - parse_syslog - label: Timestamp group: Field Parsers ui: {} enum: - parse_timestamp - label: Tokens group: Field Parsers ui: {} enum: - parse_tokens - label: URL group: Field Parsers ui: {} enum: - parse_url - label: User Agent group: Field Parsers ui: {} enum: - parse_user_agent - label: VPC Flow Log group: Line Parsers ui: {} description: Parse VPC logs using default format for fields from v2 to v4 enum: - parse_aws_vpc_flow_log ui: display_type: list errorMessage: must be one of allowed parsers title: Parser description: The kind of parser to use against the input value from "field". required: - field - parser - if: properties: type: enum: - parse-sequentially deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Parse Sequentially description: Parse fields using one or more parsers in order, sending out the event on the first successful match. additionalProperties: false errorMessage: properties: target_field: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) properties: field: type: string title: Field description: The field whose value should be parsed. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: oneOf: - type: "null" - type: string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference title: Target field description: The field into which the parsed value should be inserted. Leave blank to insert the parsed data into the original field. default: null parsers: type: array title: List of parsers description: The list of parsers to use in order against the input valuefrom "field", short-circuiting on the first successful match. items: type: object allOf: - if: properties: parser: enum: - parse_key_value then: properties: options: $ref: "#key-value-parser-options" - if: properties: parser: enum: - parse_timestamp then: properties: options: $ref: "#timestamp-parser-options" required: - options - if: properties: parser: enum: - parse_grok then: properties: options: $ref: "#grok-pattern-parser-options" required: - options - if: properties: parser: enum: - parse_regex then: properties: options: allOf: - $ref: "#regex-parser-options" ui: display_type: regex-pattern required: - options - if: properties: parser: enum: - parse_apache_log then: properties: options: $ref: "#apache-parser-options" required: - options - if: properties: parser: enum: - parse_nginx_log then: properties: options: $ref: "#nginx-parser-options" required: - options - if: properties: parser: enum: - parse_cef then: properties: options: $ref: "#cef-parser-options" - if: properties: parser: enum: - parse_csv then: properties: options: $ref: "#csv-parser-options" properties: label: oneOf: - type: "null" - type: string maxLength: 20 title: Title description: An arbitrary name you choose to identify the results of this processor for use when connecting them to other components. parser: oneOf: - label: AWS Cloudfront Access Log group: Line Parsers ui: {} enum: - aws_cloudfront_access_log - label: AWS Cloudwatch Log group: Line Parsers ui: {} enum: - parse_aws_cloudwatch_log_subscription_message - label: AWS Load Balancer Log group: Line Parsers ui: {} enum: - parse_aws_alb_log - label: AWS S3 Access Log group: Line Parsers ui: {} enum: - aws_s3_access_log - label: Apache Log group: Line Parsers ui: {} enum: - parse_apache_log - label: CSV group: Line Parsers ui: {} enum: - parse_csv - label: Common Event Format (CEF) Log group: Line Parsers ui: {} enum: - parse_cef - label: Common Log group: Line Parsers ui: {} enum: - parse_common_log - label: Elasticsearch Log group: Line Parsers ui: {} enum: - elasticsearch_log - label: Golang Log group: Line Parsers ui: {} enum: - golang_log - label: Google Log group: Line Parsers ui: {} enum: - parse_glog - label: Grok Pattern group: Line Parsers ui: {} enum: - parse_grok - label: Integer group: Field Parsers ui: {} enum: - parse_int - label: JSON group: Line Parsers ui: {} enum: - parse_json - label: Key/Value group: Line Parsers ui: {} enum: - parse_key_value - label: Kubernetes Log group: Line Parsers ui: {} enum: - parse_klog - label: Linux Authorization group: Line Parsers ui: {} enum: - parse_linux_authorization - label: Mongo Log group: Line Parsers ui: {} enum: - mongo_log - label: Mongo3 Log group: Line Parsers ui: {} enum: - mongo3_log - label: Nginx Log group: Line Parsers ui: {} enum: - parse_nginx_log - label: PSQL Log group: Line Parsers ui: {} enum: - psql_log - label: Query String group: Field Parsers ui: {} enum: - parse_query_string - label: Regular Expression group: Line Parsers ui: feature_flag: pipeline-parser-regex enum: - parse_regex - label: Syslog group: Line Parsers ui: {} enum: - parse_syslog - label: Timestamp group: Field Parsers ui: {} enum: - parse_timestamp - label: Tokens group: Field Parsers ui: {} enum: - parse_tokens - label: URL group: Field Parsers ui: {} enum: - parse_url - label: User Agent group: Field Parsers ui: {} enum: - parse_user_agent - label: VPC Flow Log group: Line Parsers ui: {} description: Parse VPC logs using default format for fields from v2 to v4 enum: - parse_aws_vpc_flow_log ui: display_type: list errorMessage: must be one of allowed parsers title: Parser description: The kind of parser to use against the input value from "field". _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. default: "" required: - parser - _output_name additionalProperties: type: object minItems: 1 ui: display_type: multi-output-map unique_parser_and_options: true errorMessage: uniqueItemProperties: parser can not be defined more than once required: - field - parsers - if: properties: type: enum: - route deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Route description: Route data based on whether it matches (or doesn't match) logical comparisons additionalProperties: false properties: conditionals: type: array minItems: 1 title: Conditionals description: A list of complete conditionals, including expressions and label items: type: object additionalProperties: false title: Conditional description: A list of conditions and a label properties: conditional: $ref: "#condition-group" label: type: string minLength: 1 maxLength: 255 title: Label description: An arbitrary name you choose to identify the results of this processor for use when connecting them to other components. _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. default: "" required: - conditional - label - _output_name required: - conditionals - if: properties: type: enum: - stringify deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Stringify description: Stringify messages as JSON additionalProperties: false - if: properties: type: enum: - compact-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Compact Fields description: Remove empty values from a list of fields additionalProperties: false properties: fields: type: array title: Fields description: A list of fields to remove empty values from items: type: string title: Field description: Field to remove empty value from pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference minItems: 1 options: type: object additionalProperties: false title: Compact Options description: Additional options for removing empty values from fields properties: compact_array: type: boolean title: Compact Array description: Remove empty arrays from a field default: true compact_object: type: boolean title: Compact Object description: Remove empty objects from a field default: true required: - fields - if: properties: type: enum: - flatten-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Flatten Fields description: Flattens the object or array value of a field into a single level representation additionalProperties: false properties: fields: type: array title: Fields description: Define specific nested objects you wish to flatten. This will not flatten the entire object, just the specified fields. Leave blank to flatten the entire object items: type: string title: Field description: Field to flatten pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference options: type: object additionalProperties: false title: Flatten Options description: Additional options for flattening fields properties: delimiter: type: string default: _ title: Delimiter description: When flattening, the nested field labels will be appended to the higher level field names along with a delimiter to allow understanding of the original structure. The default is an underscore (_) to ensure no collisions with the . notation used for field identifiers minLength: 1 maxLength: 1 - if: properties: type: enum: - dedupe deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Dedupe description: Dedupe (remove duplicates) from the data stream additionalProperties: false properties: number_of_events: type: number description: Number of events to compare across title: Number of Events minimum: 2 maximum: 5000 default: 5000 comparison_type: type: string enum: - Ignore - Match description: "Match: Only compares across the fields which are specified by the user.Ignore: Compares everything but the fields specified by the user." title: Comparison Type default: Match fields: type: array title: Fields description: A list of fields on which to base deduping items: type: string title: Field pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference minItems: 1 required: - number_of_events - comparison_type - fields - if: properties: type: enum: - encrypt-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Encrypt Field description: Encrypts the value of the provided field additionalProperties: false allOf: - if: properties: algorithm: enum: - AES-256-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-OFB required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CTR required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 16 required: - key properties: field: type: string title: Field description: Field to encrypt. The value of the field must be a primitive (string, number, boolean) pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference algorithm: oneOf: - label: AES-256-CFB (key = 32 characters, iv = 16 characters) description: AES-256-CFB (key = 32 characters, iv = 16 characters) enum: - AES-256-CFB - label: AES-192-CFB (key = 24 characters, iv = 16 characters) description: AES-192-CFB (key = 24 characters, iv = 16 characters) enum: - AES-192-CFB - label: AES-128-CFB (key = 16 characters, iv = 16 characters) description: AES-128-CFB (key = 16 characters, iv = 16 characters) enum: - AES-128-CFB - label: AES-256-OFB (key = 32 characters, iv = 16 characters) description: AES-256-OFB (key = 32 characters, iv = 16 characters) enum: - AES-256-OFB - label: AES-192-OFB (key = 24 characters, iv = 16 characters) description: AES-192-OFB (key = 24 characters, iv = 16 characters) enum: - AES-192-OFB - label: AES-128-OFB (key = 16 characters, iv = 16 characters) description: AES-128-OFB (key = 16 characters, iv = 16 characters) enum: - AES-128-OFB - label: AES-256-CTR (key = 32 characters, iv = 16 characters) description: AES-256-CTR (key = 32 characters, iv = 16 characters) enum: - AES-256-CTR - label: AES-192-CTR (key = 24 characters, iv = 16 characters) description: AES-192-CTR (key = 24 characters, iv = 16 characters) enum: - AES-192-CTR - label: AES-128-CTR (key = 16 characters, iv = 16 characters) description: AES-128-CTR (key = 16 characters, iv = 16 characters) enum: - AES-128-CTR - label: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-PKCS7 - label: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-PKCS7 - label: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-PKCS7 - label: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ANSIX923 - label: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ANSIX923 - label: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ANSIX923 - label: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO7816 - label: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO7816 - label: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO7816 - label: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO10126 - label: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO10126 - label: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO10126 ui: display_type: list title: Encryption algorithm description: The encryption algorithm to use key: type: string title: Encryption key description: The encryption key minLength: 16 maxLength: 32 iv_field: type: string title: Initialization vector (IV) field description: The field in which to store the generated initialization vector, IV. Each encrypted value will have a unique IV pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference encode_raw_bytes: type: boolean title: Encode encrypted field and IV as Base 64 text description: Encode the encrypted value and generated initialization vector as Base64 text default: true required: - field - algorithm - key - iv_field - if: properties: type: enum: - decrypt-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Decrypt Field additionalProperties: false description: Decrypts the value of the provided field allOf: - if: properties: algorithm: enum: - AES-256-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-OFB required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CTR required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-PKCS7 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ANSIX923 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO7816 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key - if: properties: algorithm: enum: - AES-256-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 32 maxLength: 32 required: - key - if: properties: algorithm: enum: - AES-192-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 24 maxLength: 24 required: - key - if: properties: algorithm: enum: - AES-128-CBC-ISO10126 required: - algorithm then: properties: key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 16 required: - key properties: field: type: string title: Field description: Field to decrypt. The value of the field must be a string pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference algorithm: oneOf: - label: AES-256-CFB (key = 32 characters, iv = 16 characters) description: AES-256-CFB (key = 32 characters, iv = 16 characters) enum: - AES-256-CFB - label: AES-192-CFB (key = 24 characters, iv = 16 characters) description: AES-192-CFB (key = 24 characters, iv = 16 characters) enum: - AES-192-CFB - label: AES-128-CFB (key = 16 characters, iv = 16 characters) description: AES-128-CFB (key = 16 characters, iv = 16 characters) enum: - AES-128-CFB - label: AES-256-OFB (key = 32 characters, iv = 16 characters) description: AES-256-OFB (key = 32 characters, iv = 16 characters) enum: - AES-256-OFB - label: AES-192-OFB (key = 24 characters, iv = 16 characters) description: AES-192-OFB (key = 24 characters, iv = 16 characters) enum: - AES-192-OFB - label: AES-128-OFB (key = 16 characters, iv = 16 characters) description: AES-128-OFB (key = 16 characters, iv = 16 characters) enum: - AES-128-OFB - label: AES-256-CTR (key = 32 characters, iv = 16 characters) description: AES-256-CTR (key = 32 characters, iv = 16 characters) enum: - AES-256-CTR - label: AES-192-CTR (key = 24 characters, iv = 16 characters) description: AES-192-CTR (key = 24 characters, iv = 16 characters) enum: - AES-192-CTR - label: AES-128-CTR (key = 16 characters, iv = 16 characters) description: AES-128-CTR (key = 16 characters, iv = 16 characters) enum: - AES-128-CTR - label: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-PKCS7 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-PKCS7 - label: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-PKCS7 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-PKCS7 - label: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-PKCS7 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-PKCS7 - label: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ANSIX923 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ANSIX923 - label: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ANSIX923 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ANSIX923 - label: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ANSIX923 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ANSIX923 - label: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO7816 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO7816 - label: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO7816 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO7816 - label: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO7816 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO7816 - label: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) description: AES-256-CBC-ISO10126 (key = 32 characters, iv = 16 characters) enum: - AES-256-CBC-ISO10126 - label: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) description: AES-192-CBC-ISO10126 (key = 24 characters, iv = 16 characters) enum: - AES-192-CBC-ISO10126 - label: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) description: AES-128-CBC-ISO10126 (key = 16 characters, iv = 16 characters) enum: - AES-128-CBC-ISO10126 ui: display_type: list title: Decryption algorithm description: The algorithm with which the data was encrypted key: type: string title: Decryption key description: The key/secret used to encrypt the value minLength: 16 maxLength: 32 iv_field: type: string title: Initialization vector (IV) field description: The field from which to read the initialization vector, IV pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference decode_raw_bytes: type: boolean title: Decode the encrypted value and IV as Base 64 text description: Decode the encrypted value and initialization vector as Base64 text default: true required: - field - algorithm - key - iv_field - if: properties: type: enum: - js-script deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Script Execution description: Use JavaScript to reshape and transform your data. You can combine multiple actions like filtering, dropping, mapping, and casting inside of a single js script. additionalProperties: true properties: script: type: string title: Script ui: display_type: code-editor options: language: js description: The JavaScript representation of the event transformation. minLength: 1 maxLength: 10000 default: > // Modify the event using a subset of the JavaScript language. // The function must return the modified event function processEvent(message, metadata, timestamp, annotations) { return message } - if: properties: type: enum: - unroll deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Unroll description: Takes an array of events and emits them all as individual events. additionalProperties: false properties: field: type: string title: Field description: The JSON field name that contains an array of events pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference values_only: type: boolean title: Values only? description: When enabled, the values from the specified array field will be emitted as new events. Otherwise, the original event will be duplicated for each value in the array field, with the unrolled value present in the field specified. default: true required: - field - if: properties: type: enum: - vrl deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: VRL description: Modify your observability data using Vector Remap Language (VRL) additionalProperties: false properties: source: type: string title: VRL Source description: VRL code to execute for each event ui: display_type: code-editor options: language: vrl minLength: 1 default: .message = .message required: - source - if: properties: type: enum: - reduce deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Reduce description: Combine multiple events over time into one based on a set of criteria additionalProperties: false properties: duration_ms: type: number maximum: 7200000 title: Duration description: The amount of time (in milliseconds) to allow streaming events to accumulate into a single "reduced" event. The process repeats indefinitely, or until an "ends when" condition is satisfied. ui: display_type: milliseconds-to-human-readable default: 30000 max_events: type: number minimum: 1 maximum: 100000 title: Max Events description: The maximum number of events that can be included in a time window (specified by Duration above). The reduce operation will stop once it has reached this number of events, regardless of whether the Duration seconds have elapsed. group_by: type: array minItems: 1 title: Group By Field Path items: type: string title: Field description: The dot-notated property path errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ description: Before reducing, group events based on matching data from each of these field paths. Supports nesting via dot-notation. date_formats: type: array title: Date Formats description: Describes which root-level properties are dates, and their expected format. Dot-notation is supported, but nested field lookup paths will be an error. minItems: 1 items: type: object additionalProperties: false properties: field: type: string title: Date Field Path description: Specifies a root-level path property that contains a date value. maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ format: type: string title: Date Format description: The template describing the date format format: strftime_datetime errorMessage: must be a valid strftime datetime format maxLength: 200 required: - field - format merge_strategies: type: array title: Merge Strategy per Field minItems: 1 description: Specify merge strategies for individual root-level properties. Dot-notation is supported, but nested field lookup paths will be an error. items: type: object additionalProperties: false properties: field: type: string title: Field Path description: This is a root-level path property to apply a merge strategy to its value maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ strategy: oneOf: - label: Array description: Append each value onto an array enum: - array - label: Smallest array description: Keep the smallest array within each reduce time window enum: - shortest_array - label: Largest array description: Keep the largest array within each reduce time window enum: - longest_array - label: Unique Array description: All unique values go into a flattened array enum: - flat_unique - label: Join with space description: Concatenate into a single space-delimited string enum: - concat - label: Join with newline description: Concatenate into a single newline-delimited string enum: - concat_newline - label: Join description: Concatenate into a single string without using a delimiter enum: - concat_raw - label: Keep first description: Keep only the first value and discard the rest enum: - discard - label: Keep last description: Keep only the last non-null value enum: - retain - label: Min value description: Keep the minimum numeric value within each reduce time window enum: - min - label: Max value description: Keep the maximum numeric value within each reduce time window enum: - max - label: Sum description: Add numeric values together enum: - sum ui: display_type: list errorMessage: must be one of the allowed merge strategies title: Merge Strategy description: The merge strategy to be used for the specified property required: - field - strategy flush_condition: title: Event Flush Condition description: Force accumulated event reduction to flush the result when a conditional expression evaluates to true on an inbound event. allOf: - type: object properties: when: type: string enum: - none - starts_when - ends_when default: none ui: display_type: list errorMessage: "" title: Flush When required: - when - if: properties: when: not: enum: - none then: properties: conditional: allOf: - $ref: "#condition-group" ui: display_type: rules-engine required: - conditional - if: properties: type: enum: - map-fields deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Map Fields description: Maps data from one field to another, either by moving or copying. additionalProperties: false properties: mappings: type: array title: Mappings description: A list of field mappings. Mappings are applied in the order they are defined. minItems: 1 ui: display_type: sortable items: type: object properties: source_field: type: string title: Source field description: The field to copy data from pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference target_field: type: string title: Target field description: The field to copy data into pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference drop_source: type: boolean title: Drop source? description: When enabled, the source field is dropped after the data is copied to the target field. Otherwise, it is preserved. default: false overwrite_target: type: boolean title: Overwrite target? description: When enabled, any existing data in the target field is overwritten. Otherwise, the target field will be preserved and this mapping will have no effect. default: false required: - source_field - target_field - drop_source - overwrite_target if: properties: drop_source: enum: - true required: - drop_source then: properties: source_field: type: string title: Source field description: The field to copy data from pattern: ^(?:metadata|\.|(?:metadata|message)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ minLength: 1 errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields). Note that dropping an event's message or timestamp is not allowed required: - source_field required: - mappings - if: properties: type: enum: - metrics-tag-cardinality-limit deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Tag Cardinality Limit (Metrics) description: Limits the cardinality of metric events by either dropping events or tags that exceed a specified value limit additionalProperties: false allOf: - if: title: Mode properties: mode: type: string enum: - exact then: title: Value Limit properties: value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags minimum: 1 maximum: 500 required: - value_limit - if: title: Mode properties: mode: type: string enum: - probabilistic then: title: Value Limit properties: value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags minimum: 1 maximum: 5000 required: - value_limit properties: tags: type: array title: Tags description: A list of tags to apply cardinality limits. If none are provided, all tags will be considered. items: type: string title: Tag minLength: 1 maxLength: 100 minItems: 0 maxItems: 10 exclude_tags: type: array title: Exclude Tags description: A list of tags to explicitly exclude from cardinality limits. items: type: string title: Tag minLength: 1 maxLength: 100 minItems: 0 maxItems: 10 action: type: string title: Action description: The action to take when a tag's cardinality exceeds the value limit enum: - drop_tag - drop_event default: drop_event value_limit: type: integer title: Value Limit description: Maximum number of unique values for tags mode: type: string title: Mode description: The method to used to reduce tag value cardinality enum: - exact - probabilistic default: exact - if: properties: type: enum: - aggregate deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Aggregate description: Aggregates multiple events into a single event using either a tumbling interval window or a sliding interval window. additionalProperties: false properties: group_by: type: array minItems: 1 maxItems: 100 title: Group By Field Path description: When aggregating, group events based on matching values from each of these field paths. Supports nesting via dot-notation. items: type: string title: Field description: The dot-notated property path errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ ui: default_hint: - .name - .namespace - .tags evaluate: additionalProperties: type: string title: Evaluate description: How events should be aggregated within an interval. allOf: - if: properties: operation: enum: - CUSTOM then: properties: script: type: string title: Script ui: display_type: code-editor options: language: js description: The JavaScript representation of an event aggregation. minLength: 1 maxLength: 2048 default: > // Perform custom aggregation of the current event into the accumulated event. // The accumulated event is persisted across the entire aggregation window. // The function must return the accumulated event, which will be emitted when // the aggregation window elapses. function aggregateEvent(accum, event, metadata, timestamp, annotations) { return accum } required: - script properties: operation: title: Operation oneOf: - label: add description: Adds new values together if they are of the same type. For counters and gauges the values are added together. For sets, this is a union of the elements. For distribution, the samples are concatenated. For histograms, values in the same bucket will be added together. enum: - DEFAULT - label: sum description: Values are added together. Works for the GAUGE and COUNT metric types. enum: - SUM - label: minimum description: Get the minimum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MIN - label: maximum description: Get the maximum value for a series of events. Works for the GAUGE and COUNT metric types. enum: - MAX - label: average description: Values are averaged together. Works for the GAUGE and COUNT metric types. enum: - AVG - label: set intersection description: The aggregate produces a set of unique values. enum: - SET_INTERSECTION - label: distribution concatenation description: The aggregate produces a list of values combined together. enum: - DIST_CONCAT - label: custom description: Use a custom script to perform aggregation. enum: - CUSTOM ui: display_type: list required: - operation window: default: {} additionalProperties: type: integer title: Window Type description: Sliding windows can overlap, whereas tumbling windows are disjoint. For example, a tumbling window has a fixed time span and any events that fall within the "interval" will be used in the aggregate. Whereas in a sliding window, the aggregation occurs every "window duration" seconds after an event is encountered. allOf: - if: properties: type: enum: - sliding required: - type then: properties: window_min: type: integer title: Slide Duration (seconds) description: The amount of time to wait before creating a new sliding window for future events. minimum: 1 maximum: 200 default: 5 required: - window_min properties: type: type: string title: Type enum: - tumbling - sliding default: tumbling interval: type: integer title: Interval (seconds) description: The interval over which events are aggregated in seconds. default: 5 minimum: 1 maximum: 90000 required: - type - interval event_timestamp: type: string title: Event Timestamp description: The path to a field on the event that contains a epoch timestamp value. If an event does not have a timestamp field, events will be associated to the wall clock value when the event is processed. default: timestamp maxLength: 200 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ conditional: ui: display_type: rules-engine title: Event Flush Condition description: Force accumulated event reduction to flush the result when a conditional expression evaluates to true on an inbound event. allOf: - $ref: "#condition-group" required: - evaluate - window - if: properties: type: enum: - event-to-metric deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Event to Metric description: Allows conversion between arbitrary events and a Metric. additionalProperties: false properties: metric_name: type: string title: Metric Name description: The machine name of the metric to emit. minLength: 1 maxLength: 128 pattern: ^[a-zA-Z][a-zA-Z0-9_:]*$ errorMessage: must be a valid metric name metric_kind: title: Kind description: The kind of metric to emit, Absolute or Incremental. Absolute metrics represent a complete value, and will generally replace an existing value for the metric in the target destination. Incremental metrics represent an additive value which is aggregated in the target destination to produce a new value. oneOf: - label: Incremental enum: - incremental - label: Absolute enum: - absolute default: incremental ui: display_type: radio-group metric_type: title: Type oneOf: - label: Counter enum: - counter - label: Sum enum: - sum - label: Gauge enum: - gauge default: counter ui: display_type: list value: title: Value description: The value of this metric. additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value then: title: Value properties: value: type: number title: Value description: The value. required: - value properties: value_type: title: Value type description: Either a value from an event field, or a new value input. oneOf: - label: Value from event field enum: - field - label: New value enum: - value default: field ui: display_type: radio-group value: type: - number - string required: - value_type namespace: title: Namespace description: The namespace for this metric. additionalProperties: false allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - value - if: properties: value_type: enum: - value required: - value_type then: title: Value properties: value: type: string title: Value description: The value. minLength: 1 maxLength: 128 required: - value properties: value_type: title: Value type description: Either a value from an event field, a new value input, or none to omit the namespace. oneOf: - label: Value from event field enum: - field - label: New value enum: - value - label: None enum: - none default: none ui: display_type: radio-group value: type: - string - "null" required: - value_type tags: type: array maxItems: 10 title: Tags description: A set of tags (also called labels) to apply to the metric event. items: type: object title: Tag description: A tag to apply to the metric event. allOf: - if: properties: value_type: enum: - field then: title: Value properties: value: type: string title: Field value description: The field to use as the value of this tag. Note that fields with highly-variable values will result in high-cardinality metrics, which may impact storage or cost in downstream destinations. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference fallback: type: string title: Fallback value description: The value to use for this tag when the field value is missing. minLength: 1 maxLength: 512 required: - value - if: properties: value_type: enum: - value then: title: Value properties: value: type: string title: Value description: The value for this tag. minLength: 1 maxLength: 128 required: - value properties: name: type: string title: Name description: The tag name pattern: ^[a-zA-Z][a-zA-Z0-9_]*$ minLength: 1 maxLength: 128 errorMessage: must be a valid metric tag name value_type: title: Value type description: Either a value from an event field, or a new value input. oneOf: - label: Value from event field enum: - field - label: New value enum: - value default: field ui: display_type: radio-group value: type: string required: - name - value_type - value additionalProperties: type: string required: - metric_name - metric_kind - metric_type - value - namespace - if: properties: type: enum: - clustering deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Clustering description: Finds clusters of similar values additionalProperties: false properties: cluster_field: type: string title: Field description: Field to cluster values on pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference default: .message similarity_threshold: type: number title: Similarity threshold description: A threshold from 0.0 (not similar) to 1.0 (exactly similar) that determines how close two values need to be to be consider as being part of the same cluster default: 0.4 minimum: 0 maximum: 1 max_clusters: type: integer title: Maximum cluster description: Maximum number of clusters kept cached by the transform default: 1000 minimum: 1 maximum: 2000 max_node_depth: type: integer title: Maximum tree depth description: Maximum depth of the prefix tree default: 4 minimum: 2 maximum: 8 max_children: type: integer title: Maximum children default: 100 minimum: 2 maximum: 200 description: Maximum number of children per node of the prefix tree required: - cluster_field - if: properties: type: enum: - redact deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Redact description: Detect and anonymize sensitive data in logs and metrics additionalProperties: false properties: field: type: string title: Field description: Detect patterns in the specified field. If the field is empty, the processor will look for patterns across all the fields. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 256 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference mask_patterns: type: array title: Redact Patterns description: List of PII patterns to detect and mask minItems: 1 maxItems: 20 items: type: object additionalProperties: type: string title: Redact Pattern description: A PII pattern to detect and/or anonymize allOf: - if: properties: action: enum: - Replace required: - action then: properties: replacement: type: string title: Replacement description: "The text to replace the detected pattern. Example: 000-00-0000" minLength: 1 maxLength: 512 required: - replacement - if: properties: action: enum: - Hash required: - action then: properties: algorithm: title: Algorithm description: Hash and encode (base16 or hex) matched PII data using the specified algorithm oneOf: - label: sha2 (sha-512_256) enum: - sha2 - label: sha3 (sha3-512) enum: - sha3 ui: display_type: list required: - algorithm - if: properties: name: enum: - custom required: - name then: properties: custom_options: allOf: - type: object additionalProperties: false properties: pattern: type: string title: Pattern description: The custom pattern minLength: 1 maxLength: 4096 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching default: true multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively default: false match_newline: type: boolean title: Dot all mode description: Dot matches newline character default: false crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" default: false ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines default: false required: - pattern title: Replace Expression description: The regular expression used to find replacements. Named capture groups are supported, e.g. `(?P<key>\w+):(?P<value>.+)`. ui: display_type: regex-pattern required: - custom_options properties: name: title: Pattern description: The PII pattern to match oneOf: - label: Credit card number enum: - credit_card - label: Email address enum: - email_address - label: IPv4 address enum: - ipv4_address - label: US or Canada phone number enum: - us_canada_phone_number - label: US social security number enum: - us_social_security_number - label: Custom pattern ui: feature_flag: pipeline-parser-regex enum: - custom ui: display_type: list action: type: string title: Action description: Action to take on the detected PII data enum: - Hash - Replace - Detect Only default: Replace required: - name - action required: - mask_patterns - if: properties: type: enum: - throttle deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Throttle description: Throttle (rate-limit) events passing through this component additionalProperties: false properties: threshold: type: number minimum: 1 maximum: 10000000 title: Limit description: The number of events to allow over the time window. If a Key Field is specified, the limit will be applied to events for each unique value of that field. default: 1 window_ms: type: number minimum: 1000 maximum: 2592000000 title: Window description: The time window over which the configured limit is applied ui: display_type: milliseconds-to-human-readable default: 5000 key_field: type: string title: Key Field description: When configured, the value from this field will be used to group events into separate buckets for throttling. If left blank, or if the event does not contain a value in this field, the event will be throttled as part of the default bucket. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference exclude: allOf: - $ref: "#condition-group" title: Exclude description: When configured, events matching this condition will not be throttled. ui: display_type: rules-engine required: - threshold - window_ms - if: properties: type: enum: - set-timestamp deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Set Timestamp description: Parse a list of fields using a chosen time format. First match will override the default timestamp of the event. additionalProperties: false properties: parsers: type: array title: List of Fields and Parsers description: The list of fields and parsers to use in order of priority, short-circuiting on the first successful match. ui: display_type: multi-output-map options: sortable: false items: type: object properties: field: type: string title: Field description: The field whose value will be parsed by the chosen date pattern pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . maxLength: 512 errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference options: $ref: "#timestamp-parser-options" required: - field - options minItems: 1 required: - parsers - if: properties: type: enum: - data-profiler deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: type: object title: Data Profiler description: Profile the data sent through your pipeline. The system will generate annotations such as event type and total bytes to be used downstream. It will also normalize the app/host/level/line fields before sending the event downstream. A profile of the data including total, events, event size, log composition, and message templates will be generated as well. additionalProperties: false properties: app_fields: type: array title: App Path(s) description: A list of paths to look for the app value minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: App description: The path to the app maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .app - .application - .container host_fields: type: array title: Host Path(s) description: A list of paths to look for the host value minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Host description: The path to the host maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .host - .hostname level_fields: type: array title: Level Path(s) description: A list of paths to look for the log level minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Level description: The path to the level maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .level - .log_level line_fields: type: array title: Line Path(s) description: A list of paths to look for the event line minItems: 1 maxItems: 10 uniqueItems: true items: type: string title: Line description: The path to the line maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .line - .message label_fields: type: array title: Label Path(s) description: A list of paths to look for the event labels maxItems: 10 uniqueItems: true items: type: string title: Label description: The path to the labels maxLength: 512 pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference ui: default_hint: - .labels field_summary: type: boolean default: false title: Field Summary description: If enabled the profiler will attempt to summarize the values in fields of events passing through the node. ui: display_type: hidden ff_default_value: pipeline-node-transform-field-summary required: - app_fields - host_fields - level_fields - line_fields - if: properties: type: enum: - replace deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Replace description: Replace parts of a string using a regular expression or static string additionalProperties: type: object allOf: - if: properties: replacement_type: enum: - regex required: - replacement_type then: properties: options: allOf: - type: object additionalProperties: false properties: pattern: type: string minLength: 1 maxLength: 4096 case_sensitive: type: boolean title: Case sensitive description: Perform case sensitive matching default: true multiline: type: boolean title: Multiline mode description: ^ and $ match beginning and end of line respectively default: false match_newline: type: boolean title: Dot all mode description: Dot matches newline character default: false crlf_newline: type: boolean title: Carriage return (CRLF) mode description: "When multiline mode is enabled, newline character is interpreted as \r\n" default: false ignore_whitespace: type: boolean title: Verbose mode description: Ignore whitespace and allow comment lines default: false required: - pattern title: Replace Expression description: The regular expression used to find replacements. Named capture groups are supported, e.g. `(?P<key>\w+):(?P<value>.+)`. ui: display_type: regex-pattern replace_with: type: string minLength: 1 maxLength: 4096 title: Replacement String description: The replacement string can be static or optionally contain capture group references. Capture groups can be positional or named. For example, to switch the position of two strings in the source string `onetwo`, the pattern could be `(?Pone)(?P.+)`, and the replacement would be `$two$one`. To do the same thing with positional capture groups, a pattern of `(\w{3})(\w{3})`, and a replacement of `$2$1` would have the same result.

To render a literal dollar sign `$` in the replacement, escape it with `$$`. So, if you had a capture group named `price`, to prepend a dollar sign to the replacement, you would use `$$$price`. required: - options - replace_with - if: properties: replacement_type: enum: - string required: - replacement_type then: properties: string_pattern: type: string title: String Pattern minLength: 1 maxLength: 4096 description: A static, case-sensitive string, which will be searched and replaced. replace_with: type: string minLength: 1 maxLength: 4096 title: Replacement String description: The static string used in each replacement. required: - string_pattern - replace_with properties: source_field: type: string title: Source Field description: The field that contains a string for replacement to operate on. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ default: . errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference max_times: type: number title: Maximum Replacements description: The maximum number of replacements to be made. If not specified, all occurrences will be replaced. minimum: 1 replacement_type: title: Replacement Type description: Specifies how the source string will be searched for replacement. ui: display_type: list errorMessage: must be one of the available pattern types oneOf: - label: String description: Replace a specified string one or more times. enum: - string - label: Regular Expression description: Replace using a regular expression. enum: - regex required: - source_field - replacement_type - if: properties: type: enum: - trace-sampling deploy_type: anyOf: - enum: - saas - type: "null" required: - type then: properties: user_config: title: Trace Sampling description: Performs sampling of traces per a given rate using the Trace ID as the unique identifier. If a trace is rate selected, all spans of the trace are forwarded. If the Trace ID cannot be found, the event is dropped. allOf: - if: properties: sample_type: enum: - head then: properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 unique traces will be forwarded and all others will be dropped default: 10 minimum: 2 maximum: 100000 parent_span_id_field: type: "null" conditionals: type: "null" required: - rate - if: properties: sample_type: enum: - tail then: properties: parent_span_id_field: type: string default: .parent_span_id title: Parent Span ID description: The field to use as the parent span identifier pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference conditionals: type: array minItems: 1 title: Conditionals description: A list of complete conditionals, including expressions and a sample rate to maintain items: type: object additionalProperties: false title: Conditional description: A list of conditions and a sample rate to maintain properties: rate: type: integer title: Rate description: The rate at which events will be forwarded, expressed as 1/N. For example, `rate = 10` means 1 out of every 10 unique traces will be forwarded and all others will be dropped. A `rate = 1` means keep all traces that match the condition. minimum: 1 maximum: 100000 default: 10 conditional: allOf: - $ref: "#condition-group" title: Conditional description: A group of expressions joined together by an and/or operator. ui: display_type: rules-engine _output_name: type: string title: "[INTERNAL]" description: Internal-only field that holds a static id for the output_name. This way, labels for outputs can be updated independent of the id. required: - rate - conditional required: - parent_span_id_field - conditionals properties: sample_type: type: string description: The type of trace sampling to apply. enum: - head title: Sample Type default: head trace_id_field: type: string default: .trace_id title: Trace ID description: The field to use as the identifier of a trace. pattern: ^(?:timestamp|annotations|metadata|message|\.|(?:metadata|message|annotations)?(?:\.\w+\w*(?:\[\d+\])?|(?:\."(?:.+?)+"(:?\[\d+\])?))+)$ errorMessage: pattern: Must be a valid data access syntax. [Syntax Reference](https://docs.mezmo.com/telemetry-pipelines/syntax-for-editing-pipeline-component-configuration-values#data-fields) ui: display_type: field-reference required: - sample_type - trace_id_field properties: type: type: string enum: - filter - sample - drop-fields - parse - parse-sequentially - route - stringify - compact-fields - flatten-fields - dedupe - encrypt-fields - decrypt-fields - js-script - unroll - vrl - reduce - map-fields - metrics-tag-cardinality-limit - aggregate - event-to-metric - clustering - redact - throttle - set-timestamp - data-profiler - replace - trace-sampling deploy_type: type: string enum: - saas - local default: saas title: type: string pattern: ^[^<>"]+$ errorMessage: invalid title maxLength: 512 description: oneOf: - type: string minLength: 1 pattern: ^[^<>"]+$ - type: "null" errorMessage: invalid description user_config: type: object inputs: type: array items: type: string description: A list of inputs for the given transform. Typically this is empty when creating a new transform via the UI, but is leveraged by the terraform provider when managing pipeline state. required: - type - title - user_config required: true parameters: - schema: type: string format: uuid in: path name: transform_group_id required: true responses: "200": description: Default Response servers: - url: https://api.mezmo.com/v3 security: - PipelineServiceKey: [] tags: - name: Internal description: Routes not accessible by outside users - name: Experimental description: Experimental routes, not intended for public consumption - name: Account description: Perform account level actions - name: Alerting description: API to control alerts on pipelines - name: Data Classification & Profiling description: API for classifying and profiling data - name: Edge - Local Deploy description: User routes that are available to use with local Edge instances - name: Incident Mode description: Change or read state of pipeline (incident mode on/off) - name: Log Volume Reduction description: CRUD operations for log volume reduction - name: Metrics description: Usage metrics for pipelines - name: Pipeline Export description: Export pipeline to terraform - name: Pipeline Management description: CRUD Operations for pipeline - name: Pipeline Nodes description: CRUD operations for sources, processors, and destinations - name: Processor Groups description: CRUD operations for Processor Groups - name: Shared Sources description: Create and manage sources for use across multiple pipelines - name: Sampling and Simulation description: Create samples of data and run them through a simulated pipeline without affecting your published pipeline externalDocs: url: https://docs.mezmo.com/pipeline-api