# Introduction Source: https://docs.mezmo.com/OptTutorial/1-intro Workshop overview for using the OpenTelemetry Demo and Mezmo to understand and optimize telemetry data for both MTTR and cost. *** ## Getting Started Outside our [Telemetry Pipeline docs](/telemetry-pipelines), if you run into any issues or have feedback on either the workshop or Pipeline, please reach out to us at [support@mezmo.com](mailto:support@mezmo.com). Before beginning, you will need the following * A Mezmo account, [sign up for a trial here](https://www.mezmo.com/sign-up-mezmo-platform). * [Docker](https://www.docker.com/get-started/) ## Overview In this workshop, we will use the [OpenTelemetry Demo](https://github.com/open-telemetry/opentelemetry-demo) to explore how to understand and optimize telemetry data to both improve its value in identifying and resolving application issues and reduce the cost associated with maintaining that data. To accomplish this we will: * Create a OpenTelemetry Log, Metric and Trace [Shared Sources](/telemetry-pipelines/shared-sources) in Mezmo * [Configure OpenTelemetry collector](https://github.com/braxtonj/opentelemetry-demo/mezmo-otel-config-extras.yml) with Mezmo Shared Source credentials * Explore the OpenTelemetry Logs via [Data Profiling](/telemetry-pipelines/data-profiling) * Send log data to [Mezmo Log Analysis](/docs) * [Aggregate](/telemetry-pipelines/reduce-processor) specific log patterns * [Parse](/telemetry-pipelines/parse-sequentially-processor) custom Apache data * [Aggregate OpenTelemetry Metrics](/telemetry-pipelines/aggregate-processor) to lower fidelity * [Sample OpenTelemetry Traces](/telemetry-pipelines/sample-processor) * Configure Pipelines to be [Responsive](/telemetry-pipelines/configure-responsive-pipelines) (i.e., capture full fidelity when in an incident or deployment state) ## Final Results In the end, you are going to build four Pipelines that look like * Log Profiling Pipeline Image * Log Handler Pipeline Image * Metric Handler Pipeline Image * Trace Handler Pipeline Image These pipelines will optimize your OpenTelemetry data by aggregating, better parsing and configuring data flow responsively. By allowing for easy, granular control you can ensure the right data ends up where it belongs. The end result is a system that provides the insight needed, at the fidelity when it's needed, leading to an order of magnitude in savings. # Shared Sources Source: https://docs.mezmo.com/OptTutorial/2-shared-sources Create OpenTelemetry log, metric, and trace Shared Sources in Mezmo and save their access keys for the optimization workshop. *** ## Creating Shared Sources In order to complete this workshop, you will need a Mezmo account with Pipeline enabled. If you do not have an account, you can sign up for a free trial [here](https://www.mezmo.com/sign-up-mezmo-platform). Please feel free to select either of the demo logs upon sign up, we will be connecting new data from a local OpenTelemetry Demo deployment. You can explore [Mezmo Flow](/telemetry-pipelines/about-mezmo-flow) or continue on with the workshop. ## Step 1: Create an OpenTelemetry Log Shared Source Once you have an account, go ahead and [Create a new Shared Source](https://app.mezmo.com/pipelines/shared-sources). Select `OpenTelemetry Logs` and call it what you will, be sure to **save your created Access Key** as you will need this later. Image Note that a [Shared Source](/telemetry-pipelines/shared-sources) is a powerful way to re-use sources within Mezmo without needing to configure new integrations. Organizations can take advantage of them to provide teams the self-service ability to pull in sanitized and normalized telemetry data from any stream needed. Image ## Step 2: Create an OpenTelemetry Metric Shared Source Repeat the prior step but this time select `OpenTelemetry Metrics`. Once again be sure to write that Access Key down for later. Image ## Step 3: Create an OpenTelemetry Trace Shared Source Finally create a `OpenTelemetry Metrics` Shared Source and save that Access Key. Image # Setting Up the OTel Demo App Source: https://docs.mezmo.com/OptTutorial/3-configure-build-demo Clone the OpenTelemetry Demo, configure the collector with your Mezmo Shared Source credentials, and run it via Docker. *** ## Step 1: Clone the OpenTelemetry Demo As we are using a modified version of the OpenTelemetry Demo with expanded logs we will need to: clone the repo, modify the config, and then build from scratch. Run `git clone https://github.com/open-telemetry/opentelemetry-demo` into a folder of your choice. ## Step 2: Configure `src/otel-collector/otelcol-config-extras.yml` with Shared Source Credentials Grab the URL and saved Access Keys from your Log, Metric and Trace Shared Sources (don't worry, if you didn't save the key you can always create a new one). Modify[ src/otel-collector/otelcol-config-extras.yml](https://github.com/open-telemetry/opentelemetry-demo/blob/main/src/otel-collector/otelcol-config-extras.yml) with the proper credentials and save. Note that Log, Metrics and Traces will each have their own unique URL and Access Keys. #### Example OpenTelemetry Extras Configuration YAML ```yaml theme={null} # Example Mezmo Pipeline Configuration for OpenTelemetry Demo exporters: otlphttp/mezmo_otel_logs: endpoint: "YOUR_MEZMO_PIPELINE_SOURCE_URL" headers: Authorization: "YOUR_MEZMO_PIPELINE_AUTH_TOKEN" otlphttp/mezmo_otel_metrics: endpoint: "YOUR_MEZMO_PIPELINE_SOURCE_URL" headers: Authorization: "OUR_MEZMO_PIPELINE_AUTH_TOKEN" otlphttp/mezmo_otel_traces: endpoint: "YOUR_MEZMO_PIPELINE_SOURCE_URL" headers: Authorization: "OUR_MEZMO_PIPELINE_AUTH_TOKEN" service: pipelines: logs: exporters: [otlphttp/mezmo_otel_logs, opensearch, debug] metrics: exporters: [otlphttp/mezmo_otel_metrics, otlphttp/prometheus, debug] traces: exporters: [otlphttp/mezmo_otel_traces, otlp, debug, spanmetrics] ``` ## Step 3: Run the OpenTelmetry Demo via Docker From the top level of the repo, run the following command `docker compose up --force-recreate --remove-orphans --detach` You will now have a running OpenTelemetry Demo instance up and running at `http://localhost:8080`. To learn more about the demo's capabilities, check out their [extensive docs](https://opentelemetry.io/docs/demo/docker-deployment/#verify-the-web-store-and-telemetry). For this workshop, we will be taking advantage of the default output. # Profiling Log Data Source: https://docs.mezmo.com/OptTutorial/4-profile-log-data Build an exploration pipeline with the Mezmo Data Profiler to surface high-volume and unparsed log patterns in the OpenTelemetry Demo. *** ## Why it matters The Mezmo [Data Profiler](/telemetry-pipelines/data-profiling) gives teams real-time visibility into the shape and quality of their telemetry data as it flows through the pipeline. It automatically surfaces schemas, cardinality, and value distributions to highlight noisy, inconsistent, or high-cost fields. With these insights, teams can make informed decisions on filtering, redaction, and enrichment—optimizing both observability costs and the usefulness of their data. The Data Profiler currently works with log data. It categorizes this data by app, host, level, and type, based on how the processor is configured. It will also identify other fields and report on their cardinality. To build a Data Profile report, you'll need a pipeline with the Data Profiler processor in it. Mezmo recommends that this pipeline be separate from pipelines that feed your observability destinations, as the Data Profiler adds additional annotation data that is specific to it and not useful for observability purposes. The Data Profiler consumes 1 million events at a time and builds the report in real time as those events are being consumed. To consume additional events, use the Rerun Analysis button. This will add an additional 1 million lines into the profile. A single profile can contain up to 3 days worth of events before it needs to be reset. In addition, fields from an event can be mapped into Data Profiler fields to customize the report results (see Step 3). Image ## Step 1: Create an "Exploration" Pipeline Create a new Mezmo Pipeline by clicking [New Pipeline](https://app.mezmo.com/pipelines/pipeline/new) in the platform. Give this a name like `Log Explorer` and select `Create a blank pipeline`. Image ## Step 2: Add OpenTelemetry Log Source Click `Add Source` and select your OpenTelemetry Log source from the `Shared Sources` list. Image ## Step 3: Insert Otel to Profile mapping Script In order to fully take advantage of the [Mezmo Data Profiler](/telemetry-pipelines/data-profiling), let's modify the structure of this data to increase the insights the profiler provides. To do this, connect a Script Processor to the Log Source as the first processor and copy in the following code: ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { let line = message let app = metadata.resource.attributes["container.name"] let host = metadata.resource.attributes["container.hostname"] let level = metadata.level if( app == null || app == '' ){ app = metadata.resource["service.name"] } if( app == null || app == '' ){ app = metadata.resource["service_name"] } if( app == null || app == '' ){ app = metadata.scope.name } if( app == null || app == '' ){ app = 'na' } if( host == null || host == '' ){ host = metadata.headers["x-bus-partition-key"] } if( host == null || host == '' ){ host = metadata.attributes["log.file.path"] } if( host == null || host == '' ){ host = 'na' } if( level == null || level == '' ){ level = annotations.level } let new_msg = { "line":line, "app":app, "host":host, "level":level } // Extract metadata to top level fields for( const meta of Object.entries(metadata) ){ let meta_name = 'metadataotel_' + meta[0].toString() let meta_val = meta[1] new_msg[meta_name] = meta_val } return new_msg } ``` The above script simply maps Otel data to the defaults of Mezmo's profiling nodes. Note that the Profiler is completely configurable and thus mapping is not always needed. ## Step 4: Connect a Profiler node Add a `Data Profiler` node connected after the Script processor from Step 3. Give it a name like `Otel Demo Log Exploration` and leave the default configuration. To complete the pipeline, add a Blackhole Destination and name it Log Data Dump. Image ## Step 5: Deploy Finally, you must deploy your pipeline in order to start exploring your log data. * Image ## Step 6: Analyze Log Patterns Once your data profiling pipeline has been built and deployed, the profiler report will build real time in a few seconds. Within minutes, you will see something similar to the following Image Immediately there are two insights of note: 1. There is an inordinate amount of **logs simply stating a Product has been found** coming from the `product-catalog` service. This is standard type of message that contains metadata on which product was found, but it can be quite costly to retain each and every one of these. Image 2. Unparsed events that appear to be custom Apache logs from the `frontend-proxy` service. While these are [defined in the demo code here](https://github.com/braxtonj/opentelemetry-demo/blob/main/src/frontend-proxy/envoy.tmpl.yaml#L80), we can ensure this data is structured and parsed properly to be fully searchable in any downstream Observability system. Image In the next section, we will build out a log data optimization pipeline to address both these concerns. # Log Data Optimization Pipeline Source: https://docs.mezmo.com/OptTutorial/5-optimize-logs Build a responsive log pipeline that enriches, parses, routes by state, aggregates, and samples logs before sending them to Mezmo Log Analysis. *** Why it matters Log data can be incredibly valuable, but this is often dependent on context and time. By utilizing the Mezmo Data Profiles along with [Responsive Pipelines](/telemetry-pipelines/configure-responsive-pipelines) and proper parsing, teams can ensure the right data shows up at the right time in the right place with the right format. ## Step 1: Create a new Pipeline to handle and route OpenTelemetry Logs We will begin by routing OpenTelemetry Logs to Mezmo Log Analysis. To do this while optimizing the data, create a new Mezmo Pipeline by clicking [New Pipeline](https://app.mezmo.com/pipelines/pipeline/new) in the platform. Give this a name like `Log Handler`. ## Step 2: Add OpenTelemetry Log Source Click `Add Source` and select your OpenTelemetry Log source from the `Shared Sources` list just like you did for the `Exploration` pipeline in the prior section. ## Step 3: Insert State Enrichment and Normalize In order to take advantage of [Responsive Pipelines](/telemetry-pipelines/configure-responsive-pipelines), we need to enrich each event with the current pipelines operational state. The easiest way to do this is by adding an enrichment script right off the bat. Click the `three dots` on your Otel Log Source and select `Add Node->Add Processor->Script Execution`. Image Paste in the following JavaScript and click `Save`. Note that the script does a bit more than add the `operational_state` state variable, we are also normalizing some of this Otel data to be easier to use in Mezmo Log Analysis. ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { metadata.resource.attributes["pipeline.path"] = "with_mezmo" const state = getPipelineStateVariable("operational_state") let line = message let app = metadata.resource.attributes["container.name"] let host = metadata.resource.attributes["container.hostname"] let level = metadata.level if( app == null || app == '' ){ app = metadata.resource["service.name"] } if( app == null || app == '' ){ app = metadata.resource["service_name"] } if( app == null || app == '' ){ app = metadata.scope.name } if( app == null || app == '' ){ app = 'na' } if( host == null || host == '' ){ host = metadata.headers["x-bus-partition-key"] } if( host == null || host == '' ){ host = metadata.attributes["log.file.path"] } if( host == null || host == '' ){ host = 'na' } if( level == null || level == '' ){ level = annotations.level } metadata.headers = null let new_msg = { "line":line, "app":app, "host":host, "level": level, "op_state":state, "meta":metadata, '_cnt': 1 } if( message == null ){ return null } return new_msg } ``` **Pro Tip** Note that one can assign many state variables to a single pipeline. This allows for different complex workflows to be automated like QA, curated deployment telemetry and more. ## Step 4: Parse Custom Apache Logs As noted in the prior section, the raw Apache data coming in from `frontend-proxy` is an unparsed, custom [format defined by the OpenTelemetry demo](https://github.com/braxtonj/opentelemetry-demo/blob/main/src/frontend-proxy/envoy.tmpl.yaml#L80). To ensure teams can take full advantage, we will parse this data with a Grok parser so that it is structured and easily searchable. Image To do this, we will add another `Route` Processor by App along with a Parse Sequentially processor on that slice of data. ### Step 4.a: App Router Connect a Route processor through the `three dots` or `Add Processor` at the bottom with the following configuration on application * Title: `App Router` * Route: * Title: `Frontend Proxy` * Criteria: `message.app` `equal` `frontend-proxy` Hit save. Image ### Step 4.b: Parse Sequentially Processor Now let's wire a `Parse Sequentially` processor to the `Frontend Proxy` output. We will use the following configuration * Field: `message.line` * Target field: `message.line_parsed` * Custom Parser: * Title: `Custom Apache` * Parser: `Grok Pattern` * Pattern: `%{SQUARE_BRACKET}%{TIMESTAMP_ISO8601:dt}%{SQUARE_BRACKET} %{DOUBLE_QUOTE}%{DATA:method} %{DATA:path} %{DATA:http_protocol}%{DOUBLE_QUOTE} %{DATA:rsp_code} %{DATA:rsp_flags} %{DATA:rsp_code_details} %{DATA:conn_term_details} %{DOUBLE_QUOTE}%{DATA:upstream_transport_failure_reason}%{DOUBLE_QUOTE} %{DATA:bytes_received} %{DATA:bytes_sent} %{DATA:duration} %{DATA:rsp_upstream_service_time} %{DOUBLE_QUOTE}%{DATA:req_forward_for}%{DOUBLE_QUOTE} %{DOUBLE_QUOTE}%{DATA:req_user_agent}%{DOUBLE_QUOTE} %{DOUBLE_QUOTE}%{DATA:req_id}%{DOUBLE_QUOTE} %{DOUBLE_QUOTE}%{DATA:req_authority}%{DOUBLE_QUOTE} %{DOUBLE_QUOTE}%{DATA:upstream_host}%{DOUBLE_QUOTE} %{DATA:upstream_cluster} %{DATA:upstream_local_addr} %{DATA:downstream_local_addr} %{DATA:downstream_remote_addr} %{DATA:requested_server_name} %{GREEDYDATA:route_name}` Image Finally, let's keep the original line with a quick Script, add a `Script Execution` processor connected to the `Parse Sequentially: Apache` output containing the following script: ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { let old_line = message.line message.line = message.line_parsed message.line.message = old_line message.line_parsed = null return message } ``` Your pipeline should look like this at this point. Image ## Step 5: Route Based on State After the initial Enrichment processor and Apache Parser, let's now route the data flow based on Pipelines state, i.e. that `operational_state`. Connect a Route processor to all prior outputs with the following configuration * Title: `State Router` * Route 1: * Title: `Normal` * Criteria: `message.op_state` `contains` `normal` * Route 2: * Title: `Incident` * Criteria: `message.op_state` `contains` `incident` * Route 3: * Title: `Deploy` * Criteria: `message.op_state` `contains` `deploy` Image You will end up with a pipeline that looks like the following Image ## Step 6: Aggregate Normal State Logs Now, lets optimize those noisy homepage flooding logs using a `Reduce` processor. Effectively this will let us count and aggregate mainly logs into one easy to watch signal. A common pattern is to turn this into a metric but today we will leave this as a log for the downstream tool. To accomplish this, create a new Route processor connected to `State Router`'s `Normal` and `Unmatched` outputs with the following configuration: * Title: `Template Router` * Route 1: * Title: `Product Found` * Criteria: `(message.app equal product-catalog) AND (message.line contains Product Found)` Image We will then add a `Reduce` processor to roll these logs up over 5 minutes. Connect the processor with the following configuration * Title: `5min Count` * Duration: `5 minutes` * Group By Field Path: `message.host` * Merge Strategy per Field: * Field Path: `message._cnt` `sum` Image Finally, we will convert the output into a summary message using the following configuration ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { message.line = { 'message': message._cnt.toString() + ' products found via product-catalog', 'count': message._cnt } return message } ``` At this point, your pipeline should look like the following Image ## Step 7: Sample Normal State Logs From `Template Router: Unmatached` let's add sample 1 out of every 10 while in `Normal` state Simply connect a `Sample` processor with the following configuration: * Rate: `1/10` ## Step 8: Wire to Mezmo Log Analysis Finally, we will send all of this data into Mezmo Log Analysis. Because of our earlier work normalizing data in Step 3, we can simply add a final Destination to all nodes (including the `Incident` and `Deploy` paths). Create a `Mezmo Log Analysis` destination with the following configuration * Ingestion key: generate or select * Query Parameters: * Hostname: `{{message.host}}` * Tags: `otel-demo` * Log Construction Scheme: * Scheme: `Message pass-through` Image ## Step 9: Deploy Next, we must deploy your pipeline in order to start optimizing that log data. To do this, select `Deploy` in the top right of the UX Image ## Step 10: Initiate State and Grab State ID Our final step is to initiate and grab the state ID for the pipeline for automation and control later on. To do this, first we will enact a state by toggling the Normal mode to Incident mode and back. Image Now that has been initiated, you will need to grab the `Log Handler` pipeline's ID (found in the URL at `app.mezmo.com/ACCOUNT_ID/pipelines/PIPELINE_ID`) along with a Pipeline API Key here. Then, modify the following script with both that `PIPELINE_ID` and Pipeline API Key ```bash theme={null} curl --request GET \ --url 'https://api.mezmo.com/v3/pipeline/state-variable?pipeline_id=PIPELINE_ID' \ --header 'Authorization: Token PIPELINE_API_KEY' ``` **Regions outside the US** Note that for different deployment regions, the url will change slightly. For instance, in the EU one would use `https://api.eu.mezmo.com/v3/pipeline/state-variable?pipeline_id=PIPELINE_ID` Take the response and save the `STATE_ID` for later. You will find it in the `operational_state`'s data packet, which should look something akin to ```bash theme={null} { "meta": { "pk": "id", "type": "pipeline-state-variable", "links": { "self": { "create": { "uri": "/v3/pipeline/{pipeline_id}/state-variable", "method": "post" }, "list": { "uri": "/v3/pipeline/{pipeline_id}/state-variable", "method": "get" }, "replace": { "uri": "/v3/pipeline/{pipeline_id}/state-variable/{id}", "method": "put" }, "update": { "uri": "/v3/pipeline/{pipeline_id}/state-variable", "method": "patch" }, "detail": null }, "related": { "pipeline": { "list": "/v3/pipeline", "detail": "/v3/pipeline/{pipeline_id}" } } }, "page": { "next": null, "previous": null } }, "data": [ { "id": "STATE_ID", "account_id": "ACCOUNT_ID", "pipeline_id": "PIPELINE_ID", "state": { "operational_state": "normal" }, "created_at": "UTC Timestamp", "updated_at": "UTC Timestamp" } ] } ``` ## Step 11: View In Mezmo Log Analysis Navigate to [Log Analysis](https://app.mezmo.com/logs/view) and view the incoming data. In particular, if you used the `tag` above you can simply search for `tag:otel-demo`. Let's look at the outcome of our prior Log Aggregation and Parsing. First, look for the aggregated data by searching for `tag:otel-demo "products found via product-catalog"`. Notice that instead of raw lines like we saw in the Profile, we now have an aggregated message to watch saving tens of thousands of log lines. Second, check out the newly parsed data by searching for `tag:otel-demo app:frontend-proxy`. While logs are displayed nicely in the Log Viewer, one can expand a line and see all the nested structure that is easily searchable. For instance, to see all 2xx responses enter the query `tag:otel-demo app:frontend-proxy resp_code:(>=200 AND <300)` Image **Learn About Log Analysis** If you want to learn more about Log Analysis and creating things like saved Views, Alerts and more check out our [docs here](/docs) or reaching out to [support@mezmo.com](mailto:support@mezmo.com) Now let's move on and see explore some simple but powerful optimizations for OpenTelemetry Metrics and Traces. # Metrics Data Optimization Pipeline Source: https://docs.mezmo.com/OptTutorial/6-optimize-metrics Build a Mezmo pipeline that enriches, state-routes, aggregates, and limits cardinality on OpenTelemetry metrics to reduce volume. *** ## Step 1: Create a new Pipeline to handle and route OpenTelemetry Metrics Create a new Mezmo Pipeline by clicking [New Pipeline](https://app.mezmo.com/pipelines/pipeline/new) in the platform. Give this a name like `Metric Handler`, select Create Blank Pipeline, then select Continue at the bottom. ## Step 2: Add OpenTelemetry Metric Source Click `Add Source` and select your OpenTelemetry Metric source from the `Shared Sources` list similar to before. ## Step 3: Insert State Enrichment As we did with the previous pipeline, we will prepare the data to take advantage of [Responsive Pipelines](/telemetry-pipelines/configure-responsive-pipelines) in the future by enriching each metric with the current pipelines operational state. Click the `three dots` on your Otel Metric Source and select `Add Node->Add Processor->Script Execution`. Image Paste in the following JavaScript and click `Save`. Note that the script does a bit more than add the `operational_state` state variable by tagging this data in-flight. ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { const state = getPipelineStateVariable("operational_state") message.op_state = state message.name = message.name.toString() message.tags.op_state = state metadata.resource.attributes["pipeline.path"] = "with_mezmo" if( message == null ){ return null } return message } ``` ## Step 4: Route Based on State After the initial Enrichment processor, let's route the data flow based on that `operational_state`. Connect a Route processor to the Enrichment Script with the following configuration: * Title: `State Router` * Route 1: * Title: `Normal` * Criteria: `message.op_state` `contains` `normal` * Route 2: * Title: `Incident` * Criteria: `message.op_state` `contains` `incident` * Route 3: * Title: `Deploy` * Criteria: `message.op_state` `contains` `deploy` You will end up with a pipeline that looks like the following Image ## Step 5: Normalize and Aggregate in Normal State A common approach to reduce Metric data volumes is to aggregate those metrics and reduce their cardinality. Mezmo makes this incredibly easy with intuitive processors. We will implement a 5 min aggregation on all incoming metrics and trim off the tags being used. Connected to your `Normal` and `Unmatched` outputs from the `State Router`, add a `Script Execution` processor with the following script: ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { let service_name = message.tags.service_name let host_id = message.tags.host_id if( service_name == null ){ service_name = metadata.resource.attributes['service.name'] } if( service_name == null ){ service_name = 'NA' } if( host_id == null ){ host_id = metadata.resource.attributes['host.id'] } if( host_id == null ){ host_id = 'NA' } message.tags = { 'service_name': service_name, 'host_id': host_id } return message } ``` Next, we will limit the cardinality of that `host_id` tag to 10 by connecting a `Tag Cardinality Limit` processor with the following configuration: * Tags: `message.tags.host_id` * Action: `drop_tag` * Value Limit: `10` * Mode: `Probablistic` Finally, let's aggregate on a 5 min window by adding an `Aggregate` processor. Leave the default configuration, only changing the following: * Interval (seconds): `300` Image ## Step 6: Sending Data Downstream Systems Now, connect all outputs to a Blackhole destination. This is simply a placeholder for any Observability system you'd like. Explore our destinations in-app or in our [docs](/telemetry-pipelines/supported-telemetry-data-destinations) to easily send telemetry data downstream into tools, data lakes and more. You should have a pipeline that looks like the following. Image ## Step 7: Deploy Finally, you must deploy your pipeline in order to start taking advantage of your metric optimizations. ## Step 8: Initiate State and Grab State ID Same as with the Logs, let's initiate the State and save the `State ID` of this pipeline for later. First, flip the State in the UX from Normal to Incident and back to Normal to initialize. Then, in your terminal window, run the following command with the metric `pipeline's ID` and grab that `State ID`. ```bash theme={null} curl --request GET \ --url 'https://api.mezmo.com/v3/pipeline/state-variable?pipeline_id=PIPELINE_ID' \ --header 'Authorization: Token PIPELINE_API_KEY' ``` # Trace Data Optimization Pipeline Source: https://docs.mezmo.com/OptTutorial/7-optimize-traces Build a Mezmo pipeline that enriches, state-routes, and samples OpenTelemetry traces to cut volume while preserving incident fidelity. *** ## Step 1: Create a new Pipeline to handle and route OpenTelemetry Traces Create a new Mezmo Pipeline by clicking [New Pipeline](https://app.mezmo.com/pipelines/pipeline/new) in the platform. Give this a name like `Trace Handler`. Select Create Blank Pipeline then Continue. ## Step 2: Add OpenTelemetry Trace Source Click `Add Source` and select your OpenTelemetry Trace source from the `Shared Sources` list similar to before. ## Step 3: Insert State Enrichment We will add the script to enrich each trace with the current pipelines operational state to be able to take advantage of [Responsive Pipelines](/telemetry-pipelines/configure-responsive-pipelines) in the future by. Click the `three dots` on your Otel Trace Source and select `Add Node->Add Processor->Script Execution`. Paste in the following JavaScript and click `Save`. Note that the script does a bit more than add the `operational_state` state variable by tagging this data in-flight. ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { const state = getPipelineStateVariable("operational_state") message.op_state = state message.name = message.name.toString() message.tags.op_state = state metadata.resource.attributes["pipeline.path"] = "with_mezmo" if( message == null ){ return null } return message } ``` ## Step 4: Route Based on State After the initial Enrichment processor, let's route the data flow based on that `operational_state`. Connect a Route processor to the Enrichment Script with the following configuration: * Title: `State Router` * Route 1: * Title: `Normal` * Criteria: `message.op_state` `contains` `normal` * Route 2: * Title: `Incident` * Criteria: `message.op_state` `contains` `incident` * Route 3: * Title: `Deploy` * Criteria: `message.op_state` `contains` `deploy` You will end up with a pipeline that looks like the following Image ## Step 5: Sample Traces in Normal State Add a 1/10 Trace Sample processor connected to the Normal and Unmatched routes with the following configuration: * Rate: `10` Image Note that Tail-based sampling is also available in Beta. ## Step 6: Sending Data Downstream Systems Now, connect all outputs to a Blackhole destination. This is simply a placeholder for any Observability system you'd like. Explore our destinations in-app or in our [docs](/telemetry-pipelines/supported-telemetry-data-destinations) to easily send telemetry data downstream into tools, data lakes and more. Image ## Step 7: Deploy Finally, you must deploy your pipeline in order to begin applying the trace reductions. ## Step 8: Initiate State and Grab State ID Same as with the Logs, let's initiate the State and save the `State ID` of this pipeline for later. First, flip the State in the UX from Normal to Incident and back to Normal to initialize. Then, in your terminal run the following command with the metric `pipeline's ID` and grab that `State ID`. ```bash theme={null} curl --request GET \ --url 'https://api.mezmo.com/v3/pipeline/state-variable?pipeline_id=PIPELINE_ID' \ --header 'Authorization: Token PIPELINE_API_KEY' ``` # Updating to Responsive Pipelines Source: https://docs.mezmo.com/OptTutorial/8-optimize-responsively Use a REST API shell script to switch your log, metric, and trace pipelines into incident mode for full-fidelity data on demand. *** Telemetry data is both fundamental and costly for any business. While this data is invaluable for troubleshooting, monitoring and various other concerns, it is not always valuable at the same time.Mezmo [Responsive Pipelines](/telemetry-pipelines/configure-responsive-pipelines) adapt in real time to the content and conditions of your telemetry data, ensuring it’s routed and processed intelligently. They enable dynamic filtering, sampling, enrichment, and destination routing without manual reconfiguration. This flexibility helps teams reduce noise, control costs, and deliver the right data to the right tools when it matters most. Responsive pipelines can be integrated with a number of tools, here we will explore the most flexible and generic. REST API calls and a simple shell script to `Incident` mode across multiple pipelines at once. ## Step 1: Modify and Run Responsive Test Script Take the below shell script and update with your Log, Metric, and Trace Pipeline State IDs that you grabbed at the end of each step. Save it as `switch_state.sh` ```bash theme={null} API_TOKEN="" LOG_PIPELINE_ID="" LOG_PIPELINE_STATE_ID="" METRIC_PIPELINE_ID="" METRIC_PIPELINE_STATE_ID="" TRACE_PIPELINE_ID="" TRACE_PIPELINE_STATE_ID="" curl --request PUT \ --url "https://api.mezmo.com/v3/pipeline/state-variable/${LOG_PIPELINE_STATE_ID}" \ --header "Authorization: Token ${API_TOKEN}" \ --header 'Content-Type: application/json' \ --data "{\"pipeline_id\": \"${LOG_PIPELINE_ID}\",\"state\": {\"operational_state\":\"$1\"}}" curl --request PUT \ --url "https://api.mezmo.com/v3/pipeline/state-variable/${METRIC_PIPELINE_STATE_ID}" \ --header "Authorization: Token ${API_TOKEN}" \ --header 'Content-Type: application/json' \ --data "{\"pipeline_id\": \"${METRIC_PIPELINE_ID}\",\"state\": {\"operational_state\":\"$1\"}}" curl --request PUT \ --url "https://api.mezmo.com/v3/pipeline/state-variable/${TRACE_PIPELINE_STATE_ID}" \ --header "Authorization: Token ${API_TOKEN}" \ --header 'Content-Type: application/json' \ --data "{\"pipeline_id\": \"${TRACE_PIPELINE_ID}\",\"state\": {\"operational_state\":\"$1\"}}" ``` Run the saved script with the following command to switch things to incident mode (or any other desired state) `sh switch_state.sh incident` ## Step 3: Evaluate Impact Notice that when in `Normal` mode, data in your pipelines are sampled and rolled up ensuring the needed signals are captured while remaining cost conscious. However, when in `Incident` or `Deployment` modes, data is grabbed at full fidelity. You will also see this represented in the UX ### Normal Mode Image ### Incident Mode Image Due to the flexibility of [Mezmo's API](/docs/api-referencesk9h), any pipeline can be integrated with just about any Incident Management or Deployment method, from PagerDuty to GitHub to Shell scripts. To learn more, reach out to Mezmo at [support@mezmo.com](mailto:support@mezmo.com). # What's Next? Source: https://docs.mezmo.com/OptTutorial/9-next-steps Recap the four OpenTelemetry optimization pipelines you built and explore further Mezmo capabilities like rehydration, alerting, and PII redaction. *** ## Recap We have successfully integrated OpenTelemetry Telemetry data (logs, metrics and traces) and optimized that data to support problem resolution as well as minimize overall observability costs. You should have Pipelines that looks like * Log Profiling Pipeline Image * Log Handler Pipeline Image * Metric Handler Pipeline Image * Trace Handler Pipeline Image ## Learn More This tutorial has given you practice into the very basic capabilities of the Mezmo Telemetry Pipeline. There are a host of other value functions like: * Data archiving and rehydration * Alerting * Confidential and PII data detection and redaction * Data parsing * Shared processor groups * Data Aggregation of which you can take advantage. Please visit the [Mezmo Docs Library ](https://docs.mezmo.com)for more on these and other capabilities - or reach out to Mezmo through Mezmo Chat for further assistance. # Get account details Source: https://docs.mezmo.com/api-reference/account/get-account-details /apis/combined-mezmo-api.yaml get /v3/pipeline/account Get list of settings applied to an account # Update account details Source: https://docs.mezmo.com/api-reference/account/update-account-details /apis/combined-mezmo-api.yaml patch /v3/pipeline/account Update account settings # Update pipeline processing status Source: https://docs.mezmo.com/api-reference/account/update-pipeline-processing-status /apis/combined-mezmo-api.yaml put /v3/pipeline/{pipeline_id}/processing-status Change a pipeline to enabled or disabled # Add usage alert Source: https://docs.mezmo.com/api-reference/alerting/add-usage-alert /apis/combined-mezmo-api.yaml post /v3/pipeline/{pipeline_id}/usage-threshold-alert Creates a new usage threshold alert configuration # Create component alert Source: https://docs.mezmo.com/api-reference/alerting/create-component-alert /apis/combined-mezmo-api.yaml post /v3/pipeline/{pipeline_id}/{component_kind}/{component_id}/alert Create an alert attached to a pipeline component # Delete component alert Source: https://docs.mezmo.com/api-reference/alerting/delete-component-alert /apis/combined-mezmo-api.yaml delete /v3/pipeline/{pipeline_id}/{component_kind}/{component_id}/alert/{alert_id} Removes an alert attached to a pipeline component # Fetch component alert Source: https://docs.mezmo.com/api-reference/alerting/fetch-component-alert /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id}/alert/{alert_id} Retrieves an alert attached to a pipeline component # List all threshold alerts Source: https://docs.mezmo.com/api-reference/alerting/list-all-threshold-alerts /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id}/usage-threshold-alert Returns configuration of existing threshold alerts # List component alerts Source: https://docs.mezmo.com/api-reference/alerting/list-component-alerts /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id}/alert List of alerts attached to a pipeline's components # Partially update an existing threshold alert Source: https://docs.mezmo.com/api-reference/alerting/partially-update-an-existing-threshold-alert /apis/combined-mezmo-api.yaml patch /v3/pipeline/{pipeline_id}/usage-threshold-alert/{usage_threshold_alert_id} Update a part of usage threshold alert # Remove usage threshold alert Source: https://docs.mezmo.com/api-reference/alerting/remove-usage-threshold-alert /apis/combined-mezmo-api.yaml delete /v3/pipeline/{pipeline_id}/usage-threshold-alert/{usage_threshold_alert_id} Removes an existing threshold alert # Retrieves an existing usage threshold alert Source: https://docs.mezmo.com/api-reference/alerting/retrieves-an-existing-usage-threshold-alert /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id}/usage-threshold-alert/{usage_threshold_alert_id} Returns usage threshold configuration # Update component alert Source: https://docs.mezmo.com/api-reference/alerting/update-component-alert /apis/combined-mezmo-api.yaml put /v3/pipeline/{pipeline_id}/{component_kind}/{component_id}/alert/{alert_id} Update an alert attached to a pipeline component # Update threshold alert Source: https://docs.mezmo.com/api-reference/alerting/update-threshold-alert /apis/combined-mezmo-api.yaml put /v3/pipeline/{pipeline_id}/usage-threshold-alert/{usage_threshold_alert_id} Replace an existing usage alert with this one # Create Archive Configuration Source: https://docs.mezmo.com/api-reference/archiving/create-archive-configuration /apis/combined-mezmo-api.yaml post /v1/config/archiving Use this method to configure archiving for an instance. Only one archiving configuration may exist at any time. # Delete Archive Configuration Source: https://docs.mezmo.com/api-reference/archiving/delete-archive-configuration /apis/combined-mezmo-api.yaml delete /v1/config/archiving Use this method to delete the archiving configuration. # Get Archive Configuration Source: https://docs.mezmo.com/api-reference/archiving/get-archive-configuration /apis/combined-mezmo-api.yaml get /v1/config/archiving Use this method to get an existing archiving configuration. # Update Archive Configuration Source: https://docs.mezmo.com/api-reference/archiving/update-archive-configuration /apis/combined-mezmo-api.yaml put /v1/config/archiving Use this method to update an existing archiving configuration. # Create Board Source: https://docs.mezmo.com/api-reference/board/create-board /apis/combined-mezmo-api.yaml post /v1/config/board Use this method to create a board. # Delete Board Source: https://docs.mezmo.com/api-reference/board/delete-board /apis/combined-mezmo-api.yaml delete /v1/config/board/{boardId} Use this method to delete a board. # Get Board Source: https://docs.mezmo.com/api-reference/board/get-board /apis/combined-mezmo-api.yaml get /v1/config/board/{boardId} Gets the configuration for a specific board. # List Boards Source: https://docs.mezmo.com/api-reference/board/list-boards /apis/combined-mezmo-api.yaml get /v1/config/board Get a list of boards. # Create New Category Source: https://docs.mezmo.com/api-reference/configuration/create-new-category /apis/combined-mezmo-api.yaml post /v1/config/categories/{type} Creates a new category of this type. # Create Preset Alert Source: https://docs.mezmo.com/api-reference/configuration/create-preset-alert /apis/combined-mezmo-api.yaml post /v1/config/presetalert Use this method to create a preset alert. # Create View Source: https://docs.mezmo.com/api-reference/configuration/create-view /apis/combined-mezmo-api.yaml post /v1/config/view Use this method to create a view and attach alerts to the view. # Delete Alert Source: https://docs.mezmo.com/api-reference/configuration/delete-alert /apis/combined-mezmo-api.yaml delete /v1/config/presetalert/{presetId} Use this method to delete a preset alert. # Delete Category Source: https://docs.mezmo.com/api-reference/configuration/delete-category /apis/combined-mezmo-api.yaml delete /v1/config/categories/{type}/{id} Use this method to delete a category. # Delete View Source: https://docs.mezmo.com/api-reference/configuration/delete-view /apis/combined-mezmo-api.yaml delete /v1/config/view/{viewId} Use this method to delete a view and any attached alerts. # Get Category Source: https://docs.mezmo.com/api-reference/configuration/get-category /apis/combined-mezmo-api.yaml get /v1/config/categories/{type}/{id} Get a specific category # Get Preset Alert Source: https://docs.mezmo.com/api-reference/configuration/get-preset-alert /apis/combined-mezmo-api.yaml get /v1/config/presetalert/{presetId} Get a specific preset alert # Get View Source: https://docs.mezmo.com/api-reference/configuration/get-view /apis/combined-mezmo-api.yaml get /v1/config/view/{viewId} Gets the configuration for a specific view. # List Categories Source: https://docs.mezmo.com/api-reference/configuration/list-categories /apis/combined-mezmo-api.yaml get /v1/config/categories/{type} Returns all Categories of a type. # List Preset Alerts Source: https://docs.mezmo.com/api-reference/configuration/list-preset-alerts /apis/combined-mezmo-api.yaml get /v1/config/presetalert Returns all Preset Alerts in the account. # List Views Source: https://docs.mezmo.com/api-reference/configuration/list-views /apis/combined-mezmo-api.yaml get /v1/config/view Get a list of views. # Update Alert Source: https://docs.mezmo.com/api-reference/configuration/update-alert /apis/combined-mezmo-api.yaml put /v1/config/presetalert/{presetId} Update a specific preset alert. # Update Category Source: https://docs.mezmo.com/api-reference/configuration/update-category /apis/combined-mezmo-api.yaml put /v1/config/categories/{type}/{id} Update a specific category. # Update View Source: https://docs.mezmo.com/api-reference/configuration/update-view /apis/combined-mezmo-api.yaml put /v1/config/view/{viewId} Use this method to update a view. You can change the view configuration details; add or remove view specific alerts; or attach and detach preset alerts. # Fetch classifier information Source: https://docs.mezmo.com/api-reference/data-classification-&-profiling/fetch-classifier-information /apis/combined-mezmo-api.yaml get /v3/pipeline/classifier/analysis Retrieve information about a classification of a shared source # Fetch usage metrics Source: https://docs.mezmo.com/api-reference/data-classification-&-profiling/fetch-usage-metrics /apis/combined-mezmo-api.yaml get /v3/pipeline/metric/usage/profile Query and fetch usage metrics from data profiling. # Get Field Summary for an analysis Source: https://docs.mezmo.com/api-reference/data-classification-&-profiling/get-field-summary-for-an-analysis /apis/combined-mezmo-api.yaml get /v3/pipeline/classifier/{analysis_id}/fields Get Field Summary for an analysis # Get Field Summary values for a field in an analysis Source: https://docs.mezmo.com/api-reference/data-classification-&-profiling/get-field-summary-values-for-a-field-in-an-analysis /apis/combined-mezmo-api.yaml get /v3/pipeline/classifier/{analysis_id}/fields/{field_name} Get Field Summary values for a field in an analysis # Get v3pipeline transform classifieranalysis Source: https://docs.mezmo.com/api-reference/data-classification-&-profiling/get-v3pipeline-transform-classifieranalysis /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id}/transform/{pipeline_transform_id}/classifier/analysis # Get v3pipelinetransformclassifieranalysis Source: https://docs.mezmo.com/api-reference/data-classification-&-profiling/get-v3pipelinetransformclassifieranalysis /apis/combined-mezmo-api.yaml get /v3/pipeline/transform/classifier/analysis # Update analysis options Source: https://docs.mezmo.com/api-reference/data-classification-&-profiling/update-analysis-options /apis/combined-mezmo-api.yaml patch /v3/pipeline/classifier/analysis/{analysis_id} Reset classification and profiling results # Add task result Source: https://docs.mezmo.com/api-reference/edge--local-deploy/add-task-result /apis/combined-mezmo-api.yaml post /v3/pipeline/account/local-deploy/tasks/{task_id}/results Add results for a given task # Fetch logs from Edge Source: https://docs.mezmo.com/api-reference/edge--local-deploy/fetch-logs-from-edge /apis/combined-mezmo-api.yaml post /v3/pipeline/account/local-deploy/log Receive logs from an Edge pipeline # Get client config Source: https://docs.mezmo.com/api-reference/edge--local-deploy/get-client-config /apis/combined-mezmo-api.yaml get /v3/pipeline/account/local-deploy/config Gets client configuration for Edge pipelines # Get usage metrics Source: https://docs.mezmo.com/api-reference/edge--local-deploy/get-usage-metrics /apis/combined-mezmo-api.yaml post /v3/pipeline/account/local-deploy/metric/usage Receive usage metrics from an Edge pipeline # List all heartbeats Source: https://docs.mezmo.com/api-reference/edge--local-deploy/list-all-heartbeats /apis/combined-mezmo-api.yaml get /v3/pipeline/account/local-deploy/heartbeat List all Edge check-ins for the account # Poll tasks Source: https://docs.mezmo.com/api-reference/edge--local-deploy/poll-tasks /apis/combined-mezmo-api.yaml get /v3/pipeline/account/local-deploy/tasks Poll and claim available local deployment tasks for a pipeline # Post client config Source: https://docs.mezmo.com/api-reference/edge--local-deploy/post-client-config /apis/combined-mezmo-api.yaml post /v3/pipeline/account/local-deploy/config Posts deployment configuration+liveness for Edge pipelines, returning pipeline config # Remove heartbeat Source: https://docs.mezmo.com/api-reference/edge--local-deploy/remove-heartbeat /apis/combined-mezmo-api.yaml delete /v3/pipeline/account/local-deploy/heartbeat/{edge_id} Removes existing Edge heartbeats # Attach a child organization to an enterprise Source: https://docs.mezmo.com/api-reference/enterprise-v1/attach-a-child-organization-to-an-enterprise /apis/combined-mezmo-api.yaml post /v1/enterprise/account > **Deprecated.** Superseded by the `/v3/enterprise` endpoints under the **Enterprise** tag. Attaches an existing organization to an enterprise. The `retention` and `owner` field are optional, and will be defaulted if not specified. To specify which child organization to attach, one of its service keys must be supplied through our typical service key authorization schema; either as basic authorization or through the `servicekey` header. # Delete an Enterprise ServiceKey Source: https://docs.mezmo.com/api-reference/enterprise-v1/delete-an-enterprise-servicekey /apis/combined-mezmo-api.yaml delete /v1/enterprise/keys/{id} > **Deprecated.** Superseded by the `/v3/enterprise` endpoints under the **Enterprise** tag. Delete an Enterprise ServiceKey from list of keys for the Enterprise # Detach a child org from Enterprise Source: https://docs.mezmo.com/api-reference/enterprise-v1/detach-a-child-org-from-enterprise /apis/combined-mezmo-api.yaml delete /v1/enterprise/account/{accountId} > **Deprecated.** Superseded by the `/v3/enterprise` endpoints under the **Enterprise** tag. Detaches a child organization from the enterprise. The child organization is not deleted and can continue to work afterwards. # Generate a new Key for the Enterprise Source: https://docs.mezmo.com/api-reference/enterprise-v1/generate-a-new-key-for-the-enterprise /apis/combined-mezmo-api.yaml post /v1/enterprise/keys > **Deprecated.** Superseded by the `/v3/enterprise` endpoints under the **Enterprise** tag. Generate a new ServiceKey for the enterprise # Get details of a child organization Source: https://docs.mezmo.com/api-reference/enterprise-v1/get-details-of-a-child-organization /apis/combined-mezmo-api.yaml get /v1/enterprise/account/{accountId} > **Deprecated.** Superseded by the `/v3/enterprise` endpoints under the **Enterprise** tag. Retrieves a specific child organization by its id. If the organization does not belong to the enterprise, a 404 error is returned. # List all accounts Source: https://docs.mezmo.com/api-reference/enterprise-v1/list-all-accounts /apis/combined-mezmo-api.yaml get /v1/enterprise/account > **Deprecated.** Superseded by the `/v3/enterprise` endpoints under the **Enterprise** tag. Retrieves a list of all child organizations that belong to the enterprise. # List all enterprise keys Source: https://docs.mezmo.com/api-reference/enterprise-v1/list-all-enterprise-keys /apis/combined-mezmo-api.yaml get /v1/enterprise/keys > **Deprecated.** Superseded by the `/v3/enterprise` endpoints under the **Enterprise** tag. Retrieves all keys of the enterprise # Retrieve a specific entperise key Source: https://docs.mezmo.com/api-reference/enterprise-v1/retrieve-a-specific-entperise-key /apis/combined-mezmo-api.yaml get /v1/enterprise/keys/{id} > **Deprecated.** Superseded by the `/v3/enterprise` endpoints under the **Enterprise** tag. Retrieves a specific enterprise key by its `id`. # Update Child Organization Contract Source: https://docs.mezmo.com/api-reference/enterprise-v1/update-child-organization-contract /apis/combined-mezmo-api.yaml put /v1/enterprise/account/{accountId} > **Deprecated.** Superseded by the `/v3/enterprise` endpoints under the **Enterprise** tag. Makes an update to the specified child organization. Used to modify the current retention with the `retention` field or the current owner with the `owner` field. # Create a new Enterprise Child Organization (Account) Source: https://docs.mezmo.com/api-reference/enterprise/create-a-new-enterprise-child-organization-account /apis/combined-mezmo-api.yaml post /v3/enterprise/account Create a new Enterprise Child Organization (Account) # Get an Enterprise and its child Accounts Source: https://docs.mezmo.com/api-reference/enterprise/get-an-enterprise-and-its-child-accounts /apis/combined-mezmo-api.yaml get /v3/enterprise Get an Enterprise and its child Accounts # Get pipeline health metrics for all child organizations of an enterprise Source: https://docs.mezmo.com/api-reference/enterprise/get-pipeline-health-metrics-for-all-child-organizations-of-an-enterprise /apis/combined-mezmo-api.yaml get /v3/enterprise/pipeline/metric/health Get pipeline health metrics for all child organizations of an enterprise # Get pipeline node types and schemas Source: https://docs.mezmo.com/api-reference/enterprise/get-pipeline-node-types-and-schemas /apis/combined-mezmo-api.yaml get /v3/enterprise/pipeline/node Get pipeline node types, operators and alert schemas # Get pipeline usage metrics for all child organizations of an enterprise Source: https://docs.mezmo.com/api-reference/enterprise/get-pipeline-usage-metrics-for-all-child-organizations-of-an-enterprise /apis/combined-mezmo-api.yaml get /v3/enterprise/pipeline/metric/usage Get pipeline usage metrics for all child organizations of an enterprise # Get pipelines for all child organizations of an enterprise Source: https://docs.mezmo.com/api-reference/enterprise/get-pipelines-for-all-child-organizations-of-an-enterprise /apis/combined-mezmo-api.yaml get /v3/enterprise/pipeline Get pipelines for all child organizations of an enterprise # Link Child Account Source: https://docs.mezmo.com/api-reference/enterprise/link-child-account /apis/combined-mezmo-api.yaml patch /v3/enterprise/account/{account_id}/attach Moves an existing individual customer account under the owner ship of an existing enterprise with the option to change its direct owner # Perform bulk operations on Enterprise Accounts Source: https://docs.mezmo.com/api-reference/enterprise/perform-bulk-operations-on-enterprise-accounts /apis/combined-mezmo-api.yaml patch /v3/enterprise/account Perform bulk operations (CREATE, UPDATE, DELETE) on Enterprise Accounts # Retrieve a specific child account under an enterprise organization Source: https://docs.mezmo.com/api-reference/enterprise/retrieve-a-specific-child-account-under-an-enterprise-organization /apis/combined-mezmo-api.yaml get /v3/enterprise/account/{account_id} Retrieve a specific child account under an enterprise organization # Unlink Child Account Source: https://docs.mezmo.com/api-reference/enterprise/unlink-child-account /apis/combined-mezmo-api.yaml patch /v3/enterprise/account/{account_id}/detach Removes a specific account from under an enterprise making it an independant account. The account will be downgraded to a trial account once the process is complete. # Create Exclusion Rule Source: https://docs.mezmo.com/api-reference/exclusions/create-exclusion-rule /apis/combined-mezmo-api.yaml post /v1/config/ingestion/exclusions Create a new ingestion exclusion rule to help reduce log volume # Delete Exclusion Rule Source: https://docs.mezmo.com/api-reference/exclusions/delete-exclusion-rule /apis/combined-mezmo-api.yaml delete /v1/config/ingestion/exclusions/{id} Use this method to delete an exclusion rule. # Get Exclusion Rule Source: https://docs.mezmo.com/api-reference/exclusions/get-exclusion-rule /apis/combined-mezmo-api.yaml get /v1/config/ingestion/exclusions/{id} Returns an exclusion rule with the id specified. # List Exclusion Rules Source: https://docs.mezmo.com/api-reference/exclusions/list-exclusion-rules /apis/combined-mezmo-api.yaml get /v1/config/ingestion/exclusions Returns a list of exclusion rules. Note: This does not return usage quota rules. # Update Exclusion Rule Source: https://docs.mezmo.com/api-reference/exclusions/update-exclusion-rule /apis/combined-mezmo-api.yaml patch /v1/config/ingestion/exclusions/{id} Updates an existing exclusion rule # Export log lines Source: https://docs.mezmo.com/api-reference/export/export-log-lines /apis/combined-mezmo-api.yaml get /v1/export Use this method to export logs in JSON format from a logging instance. # Export log lines Source: https://docs.mezmo.com/api-reference/export/export-log-lines-1 /apis/combined-mezmo-api.yaml get /v2/export Use this method to export logs in JSON format from a logging instance. # Create Group Source: https://docs.mezmo.com/api-reference/groups/create-group /apis/combined-mezmo-api.yaml post /v1/config/groups Use this method to create a log group where you can scope the data that is accessible by members in that group. # Delete Group Source: https://docs.mezmo.com/api-reference/groups/delete-group /apis/combined-mezmo-api.yaml delete /v1/config/groups/{groupId} Use this method to list the log groups. # Get Group Source: https://docs.mezmo.com/api-reference/groups/get-group /apis/combined-mezmo-api.yaml get /v1/config/groups/{groupId} Use this method to get information on a group. # List Groups Source: https://docs.mezmo.com/api-reference/groups/list-groups /apis/combined-mezmo-api.yaml get /v1/config/groups Use this method to list the log groups. # Update Group Source: https://docs.mezmo.com/api-reference/groups/update-group /apis/combined-mezmo-api.yaml patch /v1/config/groups/{groupId} Use this method to modify a log group. You can change the name of the group and the access scope that defines the data that is accessible by members in that group. # Create state variable Source: https://docs.mezmo.com/api-reference/incident-mode/create-state-variable /apis/combined-mezmo-api.yaml post /v3/pipeline/state-variable Add a pipeline state variable to the account # Fetch state variable Source: https://docs.mezmo.com/api-reference/incident-mode/fetch-state-variable /apis/combined-mezmo-api.yaml get /v3/pipeline/state-variable/{state_variable_id} Fetches the value of a state variable # List all state variables Source: https://docs.mezmo.com/api-reference/incident-mode/list-all-state-variables /apis/combined-mezmo-api.yaml get /v3/pipeline/state-variable List all state variables defined in the account # Partially update state variable Source: https://docs.mezmo.com/api-reference/incident-mode/partially-update-state-variable /apis/combined-mezmo-api.yaml patch /v3/pipeline/state-variable Update one or many state variables # Update state variable Source: https://docs.mezmo.com/api-reference/incident-mode/update-state-variable /apis/combined-mezmo-api.yaml put /v3/pipeline/state-variable/{state_variable_id} Edits an existing state variable # Get Index Rate Alert Source: https://docs.mezmo.com/api-reference/index-rate-alert/get-index-rate-alert /apis/combined-mezmo-api.yaml get /v1/config/index-rate Gets the configuration for an Index Rate Alert # Update Index Rate Alert Source: https://docs.mezmo.com/api-reference/index-rate-alert/update-index-rate-alert /apis/combined-mezmo-api.yaml put /v1/config/index-rate Use this method to update an index rate alert. You can change the alert's configuration details. # Send Log Lines Source: https://docs.mezmo.com/api-reference/ingestion/send-log-lines /apis/combined-mezmo-api.yaml post /logs/ingest Use this method to send logs to a logging instance. ### Authentication You can find instructions on authentication [here](/log-analysis-api/ref#authentication). Ingestion is similar to the other APIs, but instead of `servicekey` you will use `apikey` if you are using the header style authentication. ### Metadata Meta is a field reserved for custom information associated with a log line. To add metadata to an API call, specify the `meta` field under the lines object. Metadata can be viewed inside that line's context WARNING: If inconsistent value types are used, that line's metadata, will not be parsed. For example, if a line is passed with a meta object, such as `meta.myfield` of type String, any subsequent lines with `meta.myfield` must have a String as the value type for `meta.myfield`. > Please be aware of [service limits](https://docs.mezmo.com/docs/Mezmo-ingestion-service-limits) > on this endpoint # Retrieve a list of AI investigations for the account Source: https://docs.mezmo.com/api-reference/investigation/retrieve-a-list-of-ai-investigations-for-the-account /apis/combined-mezmo-api.yaml get /v3/ai-history/investigation Retrieve a list of AI investigations for the account. # Retrieve a specific AI investigation. Source: https://docs.mezmo.com/api-reference/investigation/retrieve-a-specific-ai-investigation /apis/combined-mezmo-api.yaml get /v3/ai-history/investigation/{investigation_id} Retrieve a specific AI investigation by its ID. # Create a new key Source: https://docs.mezmo.com/api-reference/keys/create-a-new-key /apis/combined-mezmo-api.yaml post /v1/config/keys Create a new key of the type specified by the `type` parameter. If a name is not supplied, one will be auto generated. # Delete a specific key Source: https://docs.mezmo.com/api-reference/keys/delete-a-specific-key /apis/combined-mezmo-api.yaml delete /v1/config/keys/{id} Delete a specific key by its `id`. # Retrieve a list of all the keys Source: https://docs.mezmo.com/api-reference/keys/retrieve-a-list-of-all-the-keys /apis/combined-mezmo-api.yaml get /v1/config/keys Retrieve a list of all the keys for an account. Supported key types include `ingestion` and `service`. Use the `type` parameter to filter which type of keys to retrieve. # Retrieve a specific key Source: https://docs.mezmo.com/api-reference/keys/retrieve-a-specific-key /apis/combined-mezmo-api.yaml get /v1/config/keys/{id} Retrieves a specific key by its `id`. # Update one or more fields of a key resource Source: https://docs.mezmo.com/api-reference/keys/update-one-or-more-fields-of-a-key-resource /apis/combined-mezmo-api.yaml put /v1/config/keys/{id} Updates all of the fields described in the request body of the specified key. Fields left out of the body will remain unaffected. # Log Volume Reduction Source: https://docs.mezmo.com/api-reference/log-volume-reduction/log-volume-reduction /apis/combined-mezmo-api.yaml post /v3/pipeline/{pipeline_id}/log-volume-reduction Post log volume reduction actions to a pipeline # Creates a new member Source: https://docs.mezmo.com/api-reference/members/creates-a-new-member /apis/combined-mezmo-api.yaml post /v1/config/members Create a new user and add to the team by providing 'email', 'role' and an optional list of RBAC groups. # Delete a member Source: https://docs.mezmo.com/api-reference/members/delete-a-member /apis/combined-mezmo-api.yaml delete /v1/config/members/{email} Removes a member from the organization. This does not delete the user. # Lists all members Source: https://docs.mezmo.com/api-reference/members/lists-all-members /apis/combined-mezmo-api.yaml get /v1/config/members Retrieves all of the current members in an organization and returns their email, role, and groups. # Retrieve a member Source: https://docs.mezmo.com/api-reference/members/retrieve-a-member /apis/combined-mezmo-api.yaml get /v1/config/members/{email} Returns the role and groups of the specified member. # Update a member Source: https://docs.mezmo.com/api-reference/members/update-a-member /apis/combined-mezmo-api.yaml put /v1/config/members/{email} Updates all the fields described in the request body of the specified member. Creates a member if the member doesn't exist already. Fields left out of the body will remain unaffected. # Fetch account usage metics Source: https://docs.mezmo.com/api-reference/metrics/fetch-account-usage-metics /apis/combined-mezmo-api.yaml get /v3/pipeline/metric/usage Query and fetch usage metrics for the current account. # Fetch log cluster samples Source: https://docs.mezmo.com/api-reference/metrics/fetch-log-cluster-samples /apis/combined-mezmo-api.yaml get /v3/pipeline/metric/usage/profile/log-clustering/samples Query and fetch log cluster samples from data profiling. # Fetch log usage metrics Source: https://docs.mezmo.com/api-reference/metrics/fetch-log-usage-metrics /apis/combined-mezmo-api.yaml get /v3/pipeline/metric/usage/profile/log-clustering Query and fetch log clustering usage metrics from data profiling. # Fetch pipeline health metrics Source: https://docs.mezmo.com/api-reference/metrics/fetch-pipeline-health-metrics /apis/combined-mezmo-api.yaml get /v3/pipeline/metric/health Query and fetch health information for the current account. # Get event metrics for a pipeline Source: https://docs.mezmo.com/api-reference/metrics/get-event-metrics-for-a-pipeline /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id}/event_metrics Query and fetch event metrics for a pipeline # Delete Parsing Template Source: https://docs.mezmo.com/api-reference/parsing-template/delete-parsing-template /apis/combined-mezmo-api.yaml delete /v1/config/parsing-template/{templateId} Use this method to delete a parsing template. # Get Parsing Template Source: https://docs.mezmo.com/api-reference/parsing-template/get-parsing-template /apis/combined-mezmo-api.yaml get /v1/config/parsing-template/{templateId} Gets the configuration for a specific parsing template. # List Parsing Templates Source: https://docs.mezmo.com/api-reference/parsing-template/list-parsing-templates /apis/combined-mezmo-api.yaml get /v1/config/parsing-template Get a list of custom parsing templates. # Export pipeline to manifest Source: https://docs.mezmo.com/api-reference/pipeline-export/export-pipeline-to-manifest /apis/combined-mezmo-api.yaml get /v3/pipeline/to-manifest/{pipeline_id} Export a SaaS pipeline to a portable YAML or JSON manifest. Requires pipeline:read. # Trigger pipeline export Source: https://docs.mezmo.com/api-reference/pipeline-export/trigger-pipeline-export /apis/combined-mezmo-api.yaml post /v3/pipeline/{pipeline_id}/export Prepare a pipeline for export into a terraform workspace. # Add prebuilt pipeline to account Source: https://docs.mezmo.com/api-reference/pipeline-management/add-prebuilt-pipeline-to-account /apis/combined-mezmo-api.yaml post /v3/pipeline/pre-built Creates a new pipeline in the account that is prebuilt # Create pipeline Source: https://docs.mezmo.com/api-reference/pipeline-management/create-pipeline /apis/combined-mezmo-api.yaml post /v3/pipeline Creates a new pipeline (non published) # Deletes a pipeline Source: https://docs.mezmo.com/api-reference/pipeline-management/deletes-a-pipeline /apis/combined-mezmo-api.yaml delete /v3/pipeline/{pipeline_id} Removes a pipeline from the account # Fetch audit events Source: https://docs.mezmo.com/api-reference/pipeline-management/fetch-audit-events /apis/combined-mezmo-api.yaml get /v3/pipeline/audit-event Retrieve a paginated list of pipeline audit events for an account # Fetch pipeline confg Source: https://docs.mezmo.com/api-reference/pipeline-management/fetch-pipeline-confg /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id} Retrieve a pipeline configuration # Fetch pipeline revision Source: https://docs.mezmo.com/api-reference/pipeline-management/fetch-pipeline-revision /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id}/revision/{pipeline_revision_lookup} Returns a pipeline configuration from a particular version # Import a manifest as a new pipeline Source: https://docs.mezmo.com/api-reference/pipeline-management/import-a-manifest-as-a-new-pipeline /apis/combined-mezmo-api.yaml post /v3/pipeline/from-manifest Import a manifest as a new unpublished draft pipeline. Requires pipeline:write. # Initialize a pipeline with provided components. All sources, transforms, and sinks are linked. Source: https://docs.mezmo.com/api-reference/pipeline-management/initialize-a-pipeline-with-provided-components-all-sources-transforms-and-sinks-are-linked /apis/combined-mezmo-api.yaml post /v3/pipeline/with-components Populates and publishes a new pipeline as specified. # Initialize a pipeline with provided components. The links for all components have to be determined by the caller. Source: https://docs.mezmo.com/api-reference/pipeline-management/initialize-a-pipeline-with-provided-components-the-links-for-all-components-have-to-be-determined-by-the-caller /apis/combined-mezmo-api.yaml post /v3/pipeline/with-linked-components Populates and publishes a new pipeline as specified. # Inspect a manifest's required secrets Source: https://docs.mezmo.com/api-reference/pipeline-management/inspect-a-manifests-required-secrets /apis/combined-mezmo-api.yaml post /v3/pipeline/from-manifest/prepare Inspect a manifest and report the secrets that must be supplied on import. Creates nothing. Requires pipeline:write. # List all pipeline revisions Source: https://docs.mezmo.com/api-reference/pipeline-management/list-all-pipeline-revisions /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id}/revision Returns a list of revisions that exist in the pipeline # List all pipelines Source: https://docs.mezmo.com/api-reference/pipeline-management/list-all-pipelines /apis/combined-mezmo-api.yaml get /v3/pipeline List all pipelines for the account # List available pipeline nodes Source: https://docs.mezmo.com/api-reference/pipeline-management/list-available-pipeline-nodes /apis/combined-mezmo-api.yaml get /v3/pipeline/node A list of all available nodes for use when creating pipelines # Publish pipeline Source: https://docs.mezmo.com/api-reference/pipeline-management/publish-pipeline /apis/combined-mezmo-api.yaml post /v3/pipeline/{pipeline_id}/publish Creates a new pipeline in the account # Revert pipeline to version Source: https://docs.mezmo.com/api-reference/pipeline-management/revert-pipeline-to-version /apis/combined-mezmo-api.yaml post /v3/pipeline/{pipeline_id}/revert Updates a pipeline configuration to a previous version # Summarize pipeline health for one or more pipelines Source: https://docs.mezmo.com/api-reference/pipeline-management/summarize-pipeline-health-for-one-or-more-pipelines /apis/combined-mezmo-api.yaml get /v3/pipeline/health Returns overall ingestion health as well as pipeline-specific health, such as lag in a pipeline's sources. # Update pipeline Source: https://docs.mezmo.com/api-reference/pipeline-management/update-pipeline /apis/combined-mezmo-api.yaml put /v3/pipeline/{pipeline_id} Update pipeline properties, like title # Add a sink to a pipeline Source: https://docs.mezmo.com/api-reference/pipeline-nodes/add-a-sink-to-a-pipeline /apis/combined-mezmo-api.yaml post /v3/pipeline/{pipeline_id}/sink # Add source Source: https://docs.mezmo.com/api-reference/pipeline-nodes/add-source /apis/combined-mezmo-api.yaml post /v3/pipeline/{pipeline_id}/source Adds a source to a pipeline # Add transform Source: https://docs.mezmo.com/api-reference/pipeline-nodes/add-transform /apis/combined-mezmo-api.yaml post /v3/pipeline/{pipeline_id}/transform Add a transform to a pipeline # Delete sink Source: https://docs.mezmo.com/api-reference/pipeline-nodes/delete-sink /apis/combined-mezmo-api.yaml delete /v3/pipeline/{pipeline_id}/sink/{pipeline_sink_id} Remove sink from a pipeline # Delete v3pipeline transform group Source: https://docs.mezmo.com/api-reference/pipeline-nodes/delete-v3pipeline-transform-group /apis/combined-mezmo-api.yaml delete /v3/pipeline/{pipeline_id}/transform-group/{pipeline_transform_group_id} # Fetch sink Source: https://docs.mezmo.com/api-reference/pipeline-nodes/fetch-sink /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id}/sink/{pipeline_sink_id} Fetch a sink configuration of a pipeline # Fetch transform Source: https://docs.mezmo.com/api-reference/pipeline-nodes/fetch-transform /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id}/transform/{pipeline_transform_id} Fetch transform configuration # Get v3pipeline transform group Source: https://docs.mezmo.com/api-reference/pipeline-nodes/get-v3pipeline-transform-group /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id}/transform-group/{pipeline_transform_group_id} # Post v3pipeline transform group Source: https://docs.mezmo.com/api-reference/pipeline-nodes/post-v3pipeline-transform-group /apis/combined-mezmo-api.yaml post /v3/pipeline/{pipeline_id}/transform-group # Put v3pipeline transform group Source: https://docs.mezmo.com/api-reference/pipeline-nodes/put-v3pipeline-transform-group /apis/combined-mezmo-api.yaml put /v3/pipeline/{pipeline_id}/transform-group/{pipeline_transform_group_id} # Remove source Source: https://docs.mezmo.com/api-reference/pipeline-nodes/remove-source /apis/combined-mezmo-api.yaml delete /v3/pipeline/{pipeline_id}/source/{pipeline_source_id} Removes a source from a pipeline # Remove transform Source: https://docs.mezmo.com/api-reference/pipeline-nodes/remove-transform /apis/combined-mezmo-api.yaml delete /v3/pipeline/{pipeline_id}/transform/{pipeline_transform_id} Removes transform from pipeline # Retrieve source Source: https://docs.mezmo.com/api-reference/pipeline-nodes/retrieve-source /apis/combined-mezmo-api.yaml get /v3/pipeline/{pipeline_id}/source/{pipeline_source_id} Fetches configuration of a source # Update sink Source: https://docs.mezmo.com/api-reference/pipeline-nodes/update-sink /apis/combined-mezmo-api.yaml put /v3/pipeline/{pipeline_id}/sink/{pipeline_sink_id} Updates a sink of a pipeline # Update source Source: https://docs.mezmo.com/api-reference/pipeline-nodes/update-source /apis/combined-mezmo-api.yaml put /v3/pipeline/{pipeline_id}/source/{pipeline_source_id} Updates a source in a pipeline # Update transform Source: https://docs.mezmo.com/api-reference/pipeline-nodes/update-transform /apis/combined-mezmo-api.yaml put /v3/pipeline/{pipeline_id}/transform/{pipeline_transform_id} Updates a transform of a pipeline # Delete v3transform group Source: https://docs.mezmo.com/api-reference/processor-groups/delete-v3transform-group /apis/combined-mezmo-api.yaml delete /v3/transform-group/{transform_group_id} Delete a Processor Group # Delete v3transform group transform Source: https://docs.mezmo.com/api-reference/processor-groups/delete-v3transform-group-transform /apis/combined-mezmo-api.yaml delete /v3/transform-group/{transform_group_id}/transform/{transform_group_transform_id} Delete a Processor from a Processor Group # Get v3transform group Source: https://docs.mezmo.com/api-reference/processor-groups/get-v3transform-group /apis/combined-mezmo-api.yaml get /v3/transform-group/{transform_group_id} Get a Processor Group # Get v3transform group 1 Source: https://docs.mezmo.com/api-reference/processor-groups/get-v3transform-group-1 /apis/combined-mezmo-api.yaml get /v3/transform-group List all Processor Groups for the account # Get v3transform group revision Source: https://docs.mezmo.com/api-reference/processor-groups/get-v3transform-group-revision /apis/combined-mezmo-api.yaml get /v3/transform-group/{transform_group_id}/revision/{transform_group_revision_lookup} # Get v3transform group revision 1 Source: https://docs.mezmo.com/api-reference/processor-groups/get-v3transform-group-revision-1 /apis/combined-mezmo-api.yaml get /v3/transform-group/{transform_group_id}/revision # Get v3transform group transform Source: https://docs.mezmo.com/api-reference/processor-groups/get-v3transform-group-transform /apis/combined-mezmo-api.yaml get /v3/transform-group/{transform_group_id}/transform/{transform_group_transform_id} Get a Processor from a Processor Group # Post v3transform group Source: https://docs.mezmo.com/api-reference/processor-groups/post-v3transform-group /apis/combined-mezmo-api.yaml post /v3/transform-group Create a new Processor Group # Post v3transform group publish Source: https://docs.mezmo.com/api-reference/processor-groups/post-v3transform-group-publish /apis/combined-mezmo-api.yaml post /v3/transform-group/{transform_group_id}/publish Publish changes to a Processor Group # Post v3transform group transform Source: https://docs.mezmo.com/api-reference/processor-groups/post-v3transform-group-transform /apis/combined-mezmo-api.yaml post /v3/transform-group/{transform_group_id}/transform Create a Processor in a Processor Group # Post v3transform groupfrom pipeline Source: https://docs.mezmo.com/api-reference/processor-groups/post-v3transform-groupfrom-pipeline /apis/combined-mezmo-api.yaml post /v3/transform-group/from-pipeline Create a new Processor Group from a set of Pipeline components # Put v3transform group Source: https://docs.mezmo.com/api-reference/processor-groups/put-v3transform-group /apis/combined-mezmo-api.yaml put /v3/transform-group/{transform_group_id} Update a Processor Group # Put v3transform group transform Source: https://docs.mezmo.com/api-reference/processor-groups/put-v3transform-group-transform /apis/combined-mezmo-api.yaml put /v3/transform-group/{transform_group_id}/transform/{transform_group_transform_id} Update a Processor in a Processor Group # Create data sample Source: https://docs.mezmo.com/api-reference/sampling-and-simulation/create-data-sample /apis/combined-mezmo-api.yaml post /v3/pipeline/{container_id}/source/{source_id}/sample Create sample data for use in simulation # Delete data sample Source: https://docs.mezmo.com/api-reference/sampling-and-simulation/delete-data-sample /apis/combined-mezmo-api.yaml delete /v3/pipeline/{container_id}/source/{source_id}/sample/{sample_id} Remove a data sample from the pipeline # Fetch data sample Source: https://docs.mezmo.com/api-reference/sampling-and-simulation/fetch-data-sample /apis/combined-mezmo-api.yaml get /v3/pipeline/{container_id}/source/{source_id}/sample/{sample_id} Returns a data sample # List data samples Source: https://docs.mezmo.com/api-reference/sampling-and-simulation/list-data-samples /apis/combined-mezmo-api.yaml get /v3/pipeline/{container_id}/source/{source_id}/sample Returns a list of data samples that exist in the pipeline # Post v3transform group simulate Source: https://docs.mezmo.com/api-reference/sampling-and-simulation/post-v3transform-group-simulate /apis/combined-mezmo-api.yaml post /v3/transform-group/{transform_group_id}/simulate Sends data to a vector instance with a simulated transform group # Retrieve grok pattern Source: https://docs.mezmo.com/api-reference/sampling-and-simulation/retrieve-grok-pattern /apis/combined-mezmo-api.yaml get /v3/pipeline/grok-pattern Retrieves a list of grok patterns available to use # Simulate grok pattern Source: https://docs.mezmo.com/api-reference/sampling-and-simulation/simulate-grok-pattern /apis/combined-mezmo-api.yaml post /v3/pipeline/grok-pattern/simulate Simulate data flowing through a grok pattern # Simulate regex pattern Source: https://docs.mezmo.com/api-reference/sampling-and-simulation/simulate-regex-pattern /apis/combined-mezmo-api.yaml post /v3/pipeline/regex-pattern/simulate View results of sending data using a regular expression # Simulate timestamp pattern Source: https://docs.mezmo.com/api-reference/sampling-and-simulation/simulate-timestamp-pattern /apis/combined-mezmo-api.yaml post /v3/pipeline/timestamp-format/simulate Simulate data with a timestamp format # Start pipeline tap Source: https://docs.mezmo.com/api-reference/sampling-and-simulation/start-pipeline-tap /apis/combined-mezmo-api.yaml post /v3/pipeline/{pipeline_id}/tap/{component_id} Start a tap operation to view data as it flows through the pipeline. # Start simulation of pipeline Source: https://docs.mezmo.com/api-reference/sampling-and-simulation/start-simulation-of-pipeline /apis/combined-mezmo-api.yaml post /v3/pipeline/{pipeline_id}/simulate/{pipeline_source_id} Sends data to a vector instance with a simulated pipeline # Update data sample Source: https://docs.mezmo.com/api-reference/sampling-and-simulation/update-data-sample /apis/combined-mezmo-api.yaml put /v3/pipeline/{container_id}/source/{source_id}/sample/{sample_id} Updates an existing data sample # Creates a new screen Source: https://docs.mezmo.com/api-reference/screens/creates-a-new-screen /apis/combined-mezmo-api.yaml post /v1/config/screen Creates a new screen # Delete a screen Source: https://docs.mezmo.com/api-reference/screens/delete-a-screen /apis/combined-mezmo-api.yaml delete /v1/config/screen/{id} Deletes a screen # List all screens Source: https://docs.mezmo.com/api-reference/screens/list-all-screens /apis/combined-mezmo-api.yaml get /v1/config/screen List all screens in a short form that only includes the screen id and title. # Retrieve a screen Source: https://docs.mezmo.com/api-reference/screens/retrieve-a-screen /apis/combined-mezmo-api.yaml get /v1/config/screen/{id} Retrieve a specific screen # Add signing key Source: https://docs.mezmo.com/api-reference/shared-sources/add-signing-key /apis/combined-mezmo-api.yaml put /v3/pipeline/gateway-route/{gateway_route_id}/signing-key Adds a signing key to a gateway route # Create access key Source: https://docs.mezmo.com/api-reference/shared-sources/create-access-key /apis/combined-mezmo-api.yaml post /v3/pipeline/gateway-route/{gateway_route_id}/access-key Create an access key associated with this gateway route # Create gateway route Source: https://docs.mezmo.com/api-reference/shared-sources/create-gateway-route /apis/combined-mezmo-api.yaml post /v3/pipeline/gateway-route Create a gateway route # Delete access key Source: https://docs.mezmo.com/api-reference/shared-sources/delete-access-key /apis/combined-mezmo-api.yaml delete /v3/pipeline/gateway-route/{gateway_route_id}/access-key/{id} Delete an access key # Delete gateway route Source: https://docs.mezmo.com/api-reference/shared-sources/delete-gateway-route /apis/combined-mezmo-api.yaml delete /v3/pipeline/gateway-route/{gateway_route_id} Delete an unused gateway route. If the route is used by any active pipeline, an error will be returned. # Delete signing key Source: https://docs.mezmo.com/api-reference/shared-sources/delete-signing-key /apis/combined-mezmo-api.yaml delete /v3/pipeline/gateway-route/{gateway_route_id}/signing-key/{id} Removes a signing key attached to a gateway route # List all gateway routes Source: https://docs.mezmo.com/api-reference/shared-sources/list-all-gateway-routes /apis/combined-mezmo-api.yaml get /v3/pipeline/gateway-route List all of the shareable gateway routes for an account # Retrieve gateway route Source: https://docs.mezmo.com/api-reference/shared-sources/retrieve-gateway-route /apis/combined-mezmo-api.yaml get /v3/pipeline/gateway-route/{gateway_route_id} Retreive a gateway route # Update gateway route Source: https://docs.mezmo.com/api-reference/shared-sources/update-gateway-route /apis/combined-mezmo-api.yaml put /v3/pipeline/gateway-route/{gateway_route_id} Updates a gateway route # Update gateway route property Source: https://docs.mezmo.com/api-reference/shared-sources/update-gateway-route-property /apis/combined-mezmo-api.yaml patch /v3/pipeline/gateway-route/{gateway_route_id} Patches a gateway route # Confirm Ingestion Suspension Source: https://docs.mezmo.com/api-reference/suspensions/confirm-ingestion-suspension /apis/combined-mezmo-api.yaml post /v1/config/ingestion/suspend/confirm After /suspend is called, a token is returned for use with this route as a confirmation. # Get Ingestion Status Source: https://docs.mezmo.com/api-reference/suspensions/get-ingestion-status /apis/combined-mezmo-api.yaml get /v1/config/ingestion/status Retrieve the status of ingestion # Initiate Ingestion Suspension Source: https://docs.mezmo.com/api-reference/suspensions/initiate-ingestion-suspension /apis/combined-mezmo-api.yaml post /v1/config/ingestion/suspend First step in suspending ingestion for the instance. Returns a token for use in /suspend/confirm to actually stop ingestion # Resume Ingestion Source: https://docs.mezmo.com/api-reference/suspensions/resume-ingestion /apis/combined-mezmo-api.yaml post /v1/config/ingestion/resume Resumes ingestion if it has been previously suspended # Get Usage By App Source: https://docs.mezmo.com/api-reference/usage-v1/get-usage-by-app /apis/combined-mezmo-api.yaml get /v1/usage/apps/{name} Gets the aggregated usage information for an app matching the name provided as a path parameter, during a time period. Returns null when not found. # Get Usage by Host Source: https://docs.mezmo.com/api-reference/usage-v1/get-usage-by-host /apis/combined-mezmo-api.yaml get /v1/usage/hosts/{name} Gets the aggregated usage information for a host matching the name provided as a path parameter, during a time period. Returns null when not found. # Get Usage by Tag Source: https://docs.mezmo.com/api-reference/usage-v1/get-usage-by-tag /apis/combined-mezmo-api.yaml get /v1/usage/tags/{name} Gets the aggregated usage information for a tag matching the name provided as a path parameter, during a time period. Returns null when not found. # List Usage By App Source: https://docs.mezmo.com/api-reference/usage-v1/list-usage-by-app /apis/combined-mezmo-api.yaml get /v1/usage/apps Lists aggregated usage information for all apps during a time period. # List Usage by Host Source: https://docs.mezmo.com/api-reference/usage-v1/list-usage-by-host /apis/combined-mezmo-api.yaml get /v1/usage/hosts Lists aggregated usage information for all hosts during a time period. # List Usage by Tag Source: https://docs.mezmo.com/api-reference/usage-v1/list-usage-by-tag /apis/combined-mezmo-api.yaml get /v1/usage/tags Lists aggregated usage information for all tags during a time period. # Get Usage By App Source: https://docs.mezmo.com/api-reference/usage-v2/get-usage-by-app /apis/combined-mezmo-api.yaml get /v2/usage/apps/{name} Gets the aggregated usage information for an app matching the name provided as a path parameter during a time period. The `results` will either have a singular item, or be an empty array. # Get Usage by Host Source: https://docs.mezmo.com/api-reference/usage-v2/get-usage-by-host /apis/combined-mezmo-api.yaml get /v2/usage/hosts/{name} Gets the aggregated usage information for a host matching the name provided as a path parameter during a time period. The `results` will either have a singular item, or be an empty array. # Get Usage by Tag Source: https://docs.mezmo.com/api-reference/usage-v2/get-usage-by-tag /apis/combined-mezmo-api.yaml get /v2/usage/tags/{name} Gets the aggregated usage information for a tag matching the name provided as a path parameter during a time period. The `results` will either have a singular item, or be an empty array. # List Usage By App Source: https://docs.mezmo.com/api-reference/usage-v2/list-usage-by-app /apis/combined-mezmo-api.yaml get /v2/usage/apps Lists aggregated usage information for all apps during a time period. # List Usage by Host Source: https://docs.mezmo.com/api-reference/usage-v2/list-usage-by-host /apis/combined-mezmo-api.yaml get /v2/usage/hosts Lists aggregated usage information for all hosts during a time period. # List Usage by Tag Source: https://docs.mezmo.com/api-reference/usage-v2/list-usage-by-tag /apis/combined-mezmo-api.yaml get /v2/usage/tags Lists aggregated usage information for all tags during a time period. # Retrieve account usage totals Source: https://docs.mezmo.com/api-reference/usage-v2/retrieve-account-usage-totals /apis/combined-mezmo-api.yaml get /v2/usage Get aggregated usage information for an account's data during a time period. # A2A Integration Source: https://docs.mezmo.com/aura/a2a-implementation A2A protocol endpoints, transport modes, the agent card URL, task lifecycle, and testing examples. This module wires the [A2A RC 1.0](https://github.com/a2a-protocol) protocol into the Aura web server via [`a2a-rs-server`](https://github.com/a2aproject/a2a-rs). Aura also serves a v0.3 JSON-RPC binding at the root for pre-1.0 clients (see [A2A v0.3 Root Binding](#a2a-v03-root-binding)). ## Enabling A2A A2A is **disabled by default**. Enable it with the `--enable-a2a` flag or `AURA_ENABLE_A2A` environment variable: | Flag | Env var | Default | | -------------- | ----------------- | ------- | | `--enable-a2a` | `AURA_ENABLE_A2A` | `false` | ```bash theme={null} AURA_ENABLE_A2A=true cargo run --bin aura -- webserver # or cargo run --bin aura -- webserver --enable-a2a ``` When disabled, the A2A endpoints (`/.well-known/agent-card.json`, `/a2a/v1/*`, and `POST /`) are not mounted and return 404. ## Endpoints | Method | Path | Transport | Description | | ---------- | ------------------------------ | ------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `GET` | `/.well-known/agent-card.json` | — | Agent card (capability discovery) | | `GET` | `/health` | — | Health check | | `POST` | `/a2a/v1/message:send` | REST (HTTP+JSON) | Send a message; returns task in `Working` state immediately | | `POST` | `/a2a/v1/message:stream` | REST (SSE) | Send a message and stream task updates | | `GET` | `/a2a/v1/tasks` | REST | List tasks | | `GET` | `/a2a/v1/tasks/{id}` | REST | Get a task by ID | | `POST` | `/a2a/v1/tasks/{id}:cancel` | REST | Cancel a task | | `GET` | `/a2a/v1/tasks/{id}:subscribe` | REST (SSE) | Subscribe to task updates | | `GET/POST` | `/a2a/v1/tasks/{id}/subscribe` | REST (SSE) | Subscribe to task updates (legacy path) | | `POST` | `/a2a/v1/tasks/{id}/cancel` | REST | Cancel a task (legacy path) | | `POST` | `/a2a/v1/rpc` | JSON-RPC 2.0 | All of the above via JSON-RPC envelope | | `POST` | `/` | JSON-RPC 2.0 (v0.3) | A2A v0.3 JSON-RPC binding for pre-1.0 clients (for example, kagent); see [A2A v0.3 Root Binding](#a2a-v03-root-binding) | ### `message:send` — immediate return `AuraRequestHandler` forces `return_immediately = true` on every `message:send` request. The HTTP response returns as soon as the task is queued in `Working` state, without waiting for the agent to finish. Poll `GET /a2a/v1/tasks/{id}` or subscribe via `message:stream` / `tasks/{id}:subscribe` to track completion. ## A2A v0.3 Root Binding `POST /` (the bare service root) serves an A2A v0.3 JSON-RPC 2.0 binding. It runs alongside the existing v1.0 JSON-RPC at `/a2a/v1/rpc` and the v1.0 REST mounts under `/a2a/v1/*`. A2A v0.x clients, such as kagent's bring-your-own (BYO) agent feature, read the pre-1.0 top-level `url` field of the agent card (see [v0.x Client Compatibility](#v0x-client-compatibility)). Finding none, they fall back to POSTing JSON-RPC at the bare service root. Before this binding, Aura advertised its endpoints only through `supportedInterfaces[]` and mounted JSON-RPC only at `/a2a/v1/rpc`. Those clients hit the root, got a 404, and appeared to hang silently because they never received a usable response. The root binding is active automatically whenever A2A is enabled with the existing `--enable-a2a` flag or `AURA_ENABLE_A2A` environment variable. There is no new flag or environment variable. See [Enabling A2A](#enabling-a2a). The root binding accepts the v0.3 slash-style method names: `message/send`, `message/stream`, `tasks/get`, `tasks/cancel`, and `tasks/resubscribe`. The root binding shares the same request handler, agent executor, and in-memory task store as the v1.0 bindings, so behavior matches. A root `message/send` returns immediately with a task in the `Working` state, the same as the v1.0 `message:send` (see [message:send — immediate return](#messagesend--immediate-return)). The resulting task is stored in the same task store, so you can poll it with a v0.3 `tasks/get` call to `POST /` or with the REST endpoint `GET /a2a/v1/tasks/{id}`. The `x-aura-model` header works the same way at the root as on the other endpoints: it selects the agent configuration when multiple configs are loaded, and is forwarded to the agent's MCP connections (see [Model selection](#model-selection-x-aura-model)). The root binding accepts only the v0.3 method names above. Any other method returns `-32601 Method not found`. This includes the v1.0 method names (`SendMessage`, `GetTask`, `CancelTask`), which belong to `/a2a/v1/rpc`, and the `push-notification` and extended-card methods, which are unsupported and match the card's `pushNotifications: false` capability. A malformed JSON body returns a JSON-RPC parse error `-32700` rather than a bare HTTP 422. All protocol failures are returned inside the JSON-RPC envelope at HTTP 200, as the A2A spec requires. These slash-style method names belong only to the root binding. They are distinct from the v1.0 method names (`SendMessage`, `GetTask`, `CancelTask`) used at `/a2a/v1/rpc`. The root binding is served on the same host and port as the rest of the server; it opens no new port. Enabling A2A does not add any flag or environment variable to the Aura process. However, because `POST /` was previously always a 404, a reverse proxy, load balancer, or Kubernetes Ingress in front of Aura may need a routing rule to forward bare `POST /` to the Aura backend. Watch for collisions with health-check paths or other services that share the same origin. Only `POST /` is mounted at the root; `GET /` continues to return 404, so health-check probes on `GET /` are unaffected. Disabling A2A unmounts this route along with the other A2A surfaces, and `POST /` then returns 404 again. The A2A endpoints, including this root binding, have no built-in authentication or authorization. This matches the rest of the A2A surface. Protect the endpoint with network policy, mutual TLS (mTLS), or proxy-level authentication as your deployment requires. ## Agent card URL (`AURA_SERVER_URL`) The agent card's `supportedInterfaces[].url` fields **must be absolute** (per the A2A spec). A2A clients read these URLs from the card and pass them straight to their HTTP layer, which rejects relative paths — so a client that fetches the card successfully will still fail on `message:send` if the advertised URLs are relative. Aura builds the interface URLs from a single canonical origin, configured via: | Flag | Env var | Default | | -------------- | ----------------- | -------------------------------- | | `--server-url` | `AURA_SERVER_URL` | derived from `--host` / `--port` | When `AURA_SERVER_URL` is unset, the origin is derived from the bind host/port, with a wildcard bind (`0.0.0.0` / `::`) mapped to `127.0.0.1`. That default is fine for local development but **wrong whenever the server is reached at a different address than it binds** — behind a reverse proxy, load balancer, Kubernetes Service/Ingress, or when the container port is remapped. In those cases set `AURA_SERVER_URL` to the externally-reachable origin clients actually use (scheme + host + optional port, **no path**): ```bash theme={null} AURA_SERVER_URL=https://aura.example.com cargo run --bin aura -- webserver ``` The card then advertises absolute endpoints under that origin: ```jsonc theme={null} { "url": "https://aura.example.com/", "preferredTransport": "JSONRPC", "supportedInterfaces": [ { "url": "https://aura.example.com/a2a/v1", "protocolBinding": "HTTP+JSON", "protocolVersion": "1.0" }, { "url": "https://aura.example.com/a2a/v1/rpc", "protocolBinding": "JSONRPC", "protocolVersion": "1.0" }, { "url": "https://aura.example.com/", "protocolBinding": "JSONRPC", "protocolVersion": "0.3" } ] } ``` A trailing slash on `AURA_SERVER_URL` is trimmed before the paths are appended. ### v0.x Client Compatibility The top-level `url` and `preferredTransport` fields let pre-1.0 A2A clients discover the endpoint. The third `supportedInterfaces` entry advertises the same root binding to v1.0-aware clients. Both use the same `AURA_SERVER_URL` origin already documented in the `AURA_SERVER_URL` table above, so there is no new configuration. See [A2A v0.3 Root Binding](#a2a-v03-root-binding) for how the root endpoint handles these clients. ## Model selection (`x-aura-model`) When the server is started with multiple agent configs (e.g. `--config agent-a.toml --config agent-b.toml`), A2A clients can target a specific agent by sending the `x-aura-model` request header. This mirrors the `model` field in the OpenAI-compatible `/v1/chat/completions` endpoint. | Scenario | Behavior | | ---------------------------- | ----------------------------------------------------------------------------------- | | Single config loaded | Header is ignored — the only config is always used | | Multi-config, header present | Matches against `agent.alias` (if set), otherwise `agent.name` | | Multi-config, header absent | Falls back to the server's default agent (`--default-agent` / `AURA_DEFAULT_AGENT`) | | No matching config found | Returns an `invalid_params` A2A error | The error message when no config matches: * Header provided: `"no agent configuration found for model ''"` (A2A `invalid_params`) * No header and no default: `"no agent configuration available"` (A2A `invalid_params`) `x-aura-model` is **not** part of the A2A spec — it is an Aura extension. Like all request headers, it is also forwarded to the agent's MCP connections via the `headers_from_request` mechanism. ## Multi-turn conversations Use `contextId` to group `message:send` requests into a single conversation. It is optional. To start a new conversation, omit `contextId`. The server generates one and returns it on the task object as `task.contextId`, which you read back from the first `message:send` response. The same value is present on the `GET /a2a/v1/tasks/{id}` response. The first `message:send` call is shown under [Testing with curl](#testing-with-curl) below. To continue the conversation, send another `message:send` with `message.contextId` set to that value, placed inside the `message` object alongside the other message fields. You group requests into one conversation only by `contextId`. When you send a follow-up on the same `contextId`, the agent includes its own prior answers as context, along with prior user prompts, so it can reference what it said earlier. ```bash theme={null} # is the task.contextId returned by the prior response. curl -s -X POST http://localhost:8080/a2a/v1/message:send \ -H "Content-Type: application/json" \ -H "A2A-Version: 1.0" \ -d '{ "message": { "messageId": "msg-002", "role": "ROLE_USER", "parts": [{ "text": "And what is that result times 3?" }], "contextId": "" } }' | jq . ``` Conversation continuity depends on the session store. The default in-memory store keeps a conversation only within a single server process for its lifetime. A Redis or Valkey session store shares conversation history by `contextId` across instances, subject to a configured TTL. To configure the durable or multi-pod backend, see [Session Store](/aura/configuration-reference#session-store-durable-and-multi-pod-deployments). There is no wire-contract change. `task.history` still contains only the user prompt. The agent's answer is still delivered as artifacts (the "Response" stream and "Final Info"). This section covers request-side `contextId` linkage and improved recall, not a new response field. ## Testing with curl Assumes the server is running on `localhost:8080`. ### Agent card ```bash theme={null} curl http://localhost:8080/.well-known/agent-card.json | jq . ``` ### Health check ```bash theme={null} curl http://localhost:8080/health ``` *** ### REST — send a message ```bash theme={null} # Single-config server — no x-aura-model needed curl -s -X POST http://localhost:8080/a2a/v1/message:send \ -H "Content-Type: application/json" \ -H "A2A-Version: 1.0" \ -d '{ "message": { "messageId": "msg-001", "role": "ROLE_USER", "parts": [{ "text": "What is 2 + 2?" }] } }' | jq . # Multi-config server — target a specific agent by alias or name curl -s -X POST http://localhost:8080/a2a/v1/message:send \ -H "Content-Type: application/json" \ -H "A2A-Version: 1.0" \ -H "x-aura-model: my-agent-alias" \ -d '{ "message": { "messageId": "msg-001", "role": "ROLE_USER", "parts": [{ "text": "What is 2 + 2?" }] } }' | jq . ``` The response is a task object in `Working` state. Grab the `id` field for follow-up calls. ### REST — get a task by ID ```bash theme={null} curl -s http://localhost:8080/a2a/v1/tasks/ | jq . ``` ### REST — list tasks ```bash theme={null} curl -s http://localhost:8080/a2a/v1/tasks | jq . ``` ### REST — cancel a task ```bash theme={null} curl -s -X POST http://localhost:8080/a2a/v1/tasks/:cancel | jq . ``` *** ### JSON-RPC — send a message ```bash theme={null} curl -s -X POST http://localhost:8080/a2a/v1/rpc \ -H "Content-Type: application/json" \ -H "A2A-Version: 1.0" \ -H "x-aura-model: my-agent-alias" \ -d '{ "jsonrpc": "2.0", "method": "SendMessage", "params": { "message": { "messageId": "msg-002", "role": "ROLE_USER", "parts": [{ "text": "Summarize the A2A protocol." }] } }, "id": 1 }' | jq . ``` ### JSON-RPC — get a task ```bash theme={null} curl -s -X POST http://localhost:8080/a2a/v1/rpc \ -H "Content-Type: application/json" \ -H "A2A-Version: 1.0" \ -d '{ "jsonrpc": "2.0", "method": "GetTask", "params": { "id": "" }, "id": 2 }' | jq . ``` ### JSON-RPC — cancel a task ```bash theme={null} curl -s -X POST http://localhost:8080/a2a/v1/rpc \ -H "Content-Type: application/json" \ -H "A2A-Version: 1.0" \ -d '{ "jsonrpc": "2.0", "method": "CancelTask", "params": { "id": "" }, "id": 3 }' | jq . ``` *** ### JSON-RPC (v0.3) — send a message This is the endpoint that pre-1.0 clients like kagent use automatically. The v0.3 binding uses the v0.3 wire format, which differs from the v1.0 examples above: `role` is lowercase (`user`), each part carries a `kind` discriminator (`{ "kind": "text", ... }`), and responses come back with v0.3 spellings (for example, kebab-case task states). ```bash theme={null} # No A2A-Version header; that header only applies to /a2a/v1/rpc curl -s -X POST http://localhost:8080/ \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "method": "message/send", "params": { "message": { "kind": "message", "messageId": "msg-003", "role": "user", "parts": [{ "kind": "text", "text": "Summarize the A2A protocol." }] } }, "id": 1 }' | jq . ``` *** ### JSON-RPC (v0.3) — get a task This polls a task created by the v0.3 `message/send` call above. The same task is also reachable through the REST endpoint `GET /a2a/v1/tasks/{id}`. ```bash theme={null} # No A2A-Version header; that header only applies to /a2a/v1/rpc curl -s -X POST http://localhost:8080/ \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "method": "tasks/get", "params": { "id": "" }, "id": 2 }' | jq . ``` *** ## Notes * **`A2A-Version` is optional** — when present on `/a2a/v1/rpc` requests, the version is validated. An unsupported value returns `-32009 Version not supported`. REST endpoints do not enforce the header. * **`messageId` and `role` are required** on the `Message` object — malformed bodies return `-32602 Invalid params`. * **Text-only parts** — the executor only accepts `text` parts; `file` and `data` parts return an error. * By default, tasks are stored in the `a2a-rs-server` in-memory `TaskStore` for the lifetime of the process. A Redis or Valkey [Session Store](/aura/configuration-reference#session-store-durable-and-multi-pod-deployments) persists tasks for durable or multi-pod deployments. Use `GET /a2a/v1/tasks/{id}` or `GetTask` to poll after `message:send` returns. * Request headers passed to `/a2a/v1/message:send` are forwarded to the agent's MCP connections (same `headers_from_request` mechanism as the OpenAI-compatible endpoint). This includes `x-aura-model`. * **`x-aura-model` is an Aura extension**, not part of the A2A spec. It selects the agent configuration when multiple configs are loaded — see [Model selection](#model-selection-x-aura-model). # Breaking Changes — 10 April 2026 Source: https://docs.mezmo.com/aura/breaking-changes-20260410-agent-llm-toml-configuration LLM behavior fields moved from [agent] to [llm]; Ollama params consolidated under [llm.additional_params]. # !!BREAKING CHANGES!! ## Summary Several fields that configure LLM behavior have been moved from the `[agent]` section to the `[llm]` section. This aligns these settings with the provider they configure and reduces duplication. The `[agent]` section now only contains agent behavior settings (system prompt, context, turn depth, tool filters, etc.). **The `[agent]` section will reject these fields with an unknown-field error.** **CONFIGS THAT HAVE NOT BEEN UPDATED WILL FAIL TO PARSE AND THE APP WILL FAIL TO START.** See [Startup Errors](#startup-errors). *** ## Migrated Fields The following fields must move from `[agent]` to `[llm]`: | Field | Old location | New location | | ------------------- | ------------ | ------------ | | `temperature` | `[agent]` | `[llm]` | | `reasoning_effort` | `[agent]` | `[llm]` | | `max_tokens` | `[agent]` | `[llm]` | | `context_window` | `[agent]` | `[llm]` | | `additional_params` | `[agent]` | `[llm]` | *** ## Before / After Examples ### `temperature`, `reasoning_effort`, `context_window`, `max_tokens` ```toml theme={null} # BEFORE [llm] provider = "openai" model = "gpt-5.1" [agent] name = "My Agent" system_prompt = "..." temperature = 0.3 reasoning_effort = "medium" context_window = 128000 max_tokens = 1000 # AFTER [llm] provider = "openai" model = "gpt-5.1" reasoning_effort = "medium" temperature = 0.3 context_window = 128000 max_tokens = 1000 [agent] name = "My Agent" system_prompt = "..." ``` ### Anthropic/Bedrock thinking parameters ```toml theme={null} # BEFORE [llm] provider = "anthropic" model = "claude-sonnet-4-5-20250929" [agent] name = "Thinking Agent" system_prompt = "..." temperature = 1.0 additional_params = { thinking = { type = "enabled", budget_tokens = 8000 } } # AFTER [llm] provider = "anthropic" model = "claude-sonnet-4-5-20250929" temperature = 1.0 additional_params = { thinking = { type = "enabled", budget_tokens = 8000 } } [agent] name = "Thinking Agent" system_prompt = "..." ``` For the nested table syntax variant: ```toml theme={null} # BEFORE [agent.additional_params.thinking] type = "enabled" budget_tokens = 8000 # AFTER [llm.additional_params.thinking] type = "enabled" budget_tokens = 8000 ``` *** ## Ollama-Specific Changes Two Ollama-specific fields (`num_ctx`, `num_predict`) have been removed as dedicated top-level `[llm]` fields. They must now be placed under `[llm.additional_params]`. ```toml theme={null} # BEFORE [llm] provider = "ollama" model = "llama3.2" num_ctx = 8192 num_predict = 2048 # AFTER [llm] provider = "ollama" model = "llama3.2" [llm.additional_params] num_ctx = 8192 num_predict = 2048 ``` *** ### Startup Errors Affected Configs: `deny_unknown_fields` on `AgentConfig` and `LlmConfig` Both `aura_config::AgentConfig` and `aura_config::LlmConfig` carry `#[serde(deny_unknown_fields)]`. Any errant or stale field in either section (including the removed fields listed above) will produce a hard parse error instead of being silently ignored, calling attention to the customer. # Breaking Changes — 21 April 2026 Source: https://docs.mezmo.com/aura/breaking-changes-20260421-llm-under-agent [llm] moved from top-level to [agent.llm], enabling per-worker LLM overrides in orchestration mode. # !!BREAKING CHANGES!! ## Summary The `[llm]` TOML section has been moved from the top level to `[agent.llm]`. LLM configuration is now a property of the agent, not a sibling of it. This unlocks **per-worker LLM overrides** in orchestration mode: each `[orchestration.worker.]` may declare its own `[orchestration.worker..llm]` to run a different model (or a different provider) than the coordinator, inheriting `[agent.llm]` when omitted. **CONFIGS THAT HAVE NOT BEEN UPDATED WILL FAIL TO PARSE AND THE APP WILL FAIL TO START.** The loader detects a top-level `[llm]` table and emits a migration error pointing to this document. See [Startup Errors](#startup-errors). *** ## What Moved | Field | Old location | New location | | ------------------------- | ----------------- | --------------------------------- | | The entire `[llm]` table | top-level `[llm]` | `[agent.llm]` | | `[llm.additional_params]` | top-level | `[agent.llm.additional_params]` | | `context_window` | `[llm]` | `[agent.llm]` (follows the table) | No fields were renamed — every provider-specific field inside `[llm]` keeps its name under `[agent.llm]`. *** ## Before / After Examples ### Minimal single-agent config ```toml theme={null} # BEFORE [llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-5.1" context_window = 128000 temperature = 0.3 [agent] name = "My Agent" system_prompt = "..." turn_depth = 5 # AFTER [agent] name = "My Agent" system_prompt = "..." turn_depth = 5 [agent.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-5.1" context_window = 128000 temperature = 0.3 ``` ### `additional_params` nested tables ```toml theme={null} # BEFORE [llm] provider = "anthropic" model = "claude-sonnet-4-5-20250929" temperature = 1.0 [llm.additional_params.thinking] type = "enabled" budget_tokens = 8000 # AFTER [agent] name = "Thinking Agent" system_prompt = "..." [agent.llm] provider = "anthropic" model = "claude-sonnet-4-5-20250929" temperature = 1.0 [agent.llm.additional_params.thinking] type = "enabled" budget_tokens = 8000 ``` ### Ollama `additional_params` ```toml theme={null} # BEFORE [llm] provider = "ollama" model = "qwen3:30b-a3b" fallback_tool_parsing = true [llm.additional_params] num_ctx = 32000 think = true # AFTER [agent] name = "Local Assistant" system_prompt = "..." [agent.llm] provider = "ollama" model = "qwen3:30b-a3b" fallback_tool_parsing = true [agent.llm.additional_params] num_ctx = 32000 think = true ``` *** ## New Capability: Per-Worker LLM Overrides Workers now accept an optional `[orchestration.worker..llm]` table. When omitted, the worker inherits `[agent.llm]` (including `context_window`). When present, the worker uses its own LLM configuration exclusively. ```toml theme={null} [agent] name = "Math Coordinator" system_prompt = "..." [agent.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-5.1" context_window = 128000 [orchestration] enabled = true # Inherits [agent.llm] — no override needed for the common case [orchestration.worker.arithmetic] description = "Basic arithmetic operations" preamble = "You are an arithmetic specialist." mcp_filter = ["add", "subtract", "multiply", "divide"] # Explicit override — this worker runs a cheaper model with a smaller context [orchestration.worker.formatting] description = "Formats numeric output for humans" preamble = "You format numbers as strings." mcp_filter = [] [orchestration.worker.formatting.llm] provider = "anthropic" api_key = "{{ env.ANTHROPIC_API_KEY }}" model = "claude-haiku-4-5-20251001" context_window = 200000 ``` The worker's resolved `context_window` is what the runtime reports in `aura.session_info` events for that worker and what downstream context-budget work (LOG-23439) will use to size per-worker scratchpads. *** ## Startup Errors ### Top-level `[llm]` is no longer accepted The loader performs a pre-parse check before deserialization. If it sees a top-level `[llm]` table it fails with: ``` Configuration uses the legacy top-level [llm] table. Move it under [agent.llm] (and any [llm.additional_params] under [agent.llm.additional_params]). Workers may optionally override the LLM via [orchestration.worker..llm]. ``` ### `deny_unknown_fields` remains `aura_config::AgentConfig` and `aura_config::LlmConfig` still carry `#[serde(deny_unknown_fields)]`. Any stray field in `[agent]` or `[agent.llm]` (including the fields that were moved out of `[agent]` in the 10 April 2026 migration) still produces a hard parse error. ### Worker LLM fields `[orchestration.worker..llm]` accepts the same fields as `[agent.llm]` (the full `LlmConfig` variant set per provider). The same `deny_unknown_fields` rules apply. # Breaking Changes — 23 July 2026 Source: https://docs.mezmo.com/aura/breaking-changes-20260723-empty-mcp-filter An empty `mcp_filter` (`mcp_filter = []`) now grants no MCP tools instead of every MCP tool. # 23 July 2026 # !!BREAKING CHANGES!! ## Summary An **empty** `mcp_filter` (`mcp_filter = []`) now grants **no MCP tools** instead of every MCP tool. This applies to orchestration workers (`[orchestration.worker.].mcp_filter`) and to the single-agent `[agent].mcp_filter`. An *omitted* `mcp_filter` still grants every MCP tool, so existing configs that don't write the field are unaffected. ## Why Previously `mcp_filter = []` and an omitted filter both meant "all tools", which made two things impossible or dangerous: * **"No MCP tools" was unexpressible.** Tool-free workers (like a writer that only synthesizes text) had to resort to a non-matching dummy pattern such as `["__none__"]`. * **Removing the last allowlist entry silently widened access.** Editing `mcp_filter = ["logs_*"]` down to `mcp_filter = []` expanded the worker from one tool family to *every* tool — the opposite of the intent. See issue [#378](https://github.com/mezmo/aura/issues/378). ## The new rule | Config | Tools granted | | -------------------------- | ------------------- | | `mcp_filter` omitted | every MCP tool | | `mcp_filter = []` | none | | `mcp_filter = ["mezmo_*"]` | matching tools only | ## Migration * If a config has `mcp_filter = []` **and relied on receiving all tools**: delete the line (omit the field). * If a config used a non-matching dummy pattern (e.g. `["__none__"]`) to keep a worker tool-free: replace it with `mcp_filter = []`. `client_tool_filter` is unchanged: an empty client-tool filter still means all client tools. # AURA CLI Reference Source: https://docs.mezmo.com/aura/cli-reference The aura command-line client — install, run, configure, and use every feature of the interactive terminal client. A fast, interactive terminal client for chat completions with tool execution. Built primarily as the command line interface for [AURA by Mezmo](https://mezmo.com/aura), but works with **any OpenAI-compatible API** — plug in your own models, agents, or LLM endpoints. *** ## Quick Start ### Build from the mono-repo ```bash theme={null} # Default build (standalone + HTTP — builds agents in-process from TOML config) cargo build -p aura-cli --release # HTTP-only mode (lightweight, no agent/MCP dependencies) cargo build -p aura-cli --release --no-default-features ``` The binary will be at `target/release/aura`. ### Run it #### HTTP mode (connect to a running `aura webserver`) ```bash theme={null} # Using environment variables export AURA_API_URL="https://api.example.com" export AURA_API_KEY="your-api-key" aura # Or pass flags directly aura --api-url "https://api.example.com" \ --api-key "your-api-key" ``` #### Standalone mode (no server needed) Standalone mode is enabled by default. The CLI loads agent configs directly and runs without an HTTP server. When `--api-url` is not set, standalone mode activates automatically: ```bash theme={null} # Uses config.toml in the current directory by default aura # Single TOML config file aura --config path/to/agent.toml # Directory of TOML configs (enables /model switching between agents) aura --config configs/ # One-shot query in standalone mode aura --config agent.toml --query "hello" # Select a specific agent from a config directory aura --config configs/ --model "Math Agent" ``` In standalone mode, the CLI builds agents in-process. MCP tools from the TOML config are available. CLI local tools (Shell, Read, Update, ...) become available when **both** sides opt in — pass `--enable-client-tools` and set `[agent].enable_client_tools = true` in the loaded TOML config (single-agent configs only; orchestrated configs drop client tools). See [Client-Side Tools](#client-side-tools) for details. The `/model` command works identically — it lists all loaded configs and lets you switch between them. *** ## Generating a starter config (`aura init`) New to AURA? `aura init` walks you through creating a ready-to-run `config.toml` — no hand-editing TOML required. ```bash theme={null} aura init # interactive; writes ./config.toml aura init -o my.toml # choose the output path ``` It will: * **Sense** your environment for a provider API key (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, …) and suggest that provider. * **Pick a provider** from a validated list (openai, anthropic, bedrock, gemini, ollama, openrouter). * **Handle the API key**: if the provider's conventional env var is already set it asks whether to use it; otherwise it prompts for the key (input is masked). The generated config references the key by its native env var (`api_key = "{{ env.OPENAI_API_KEY }}"`) — secrets are never written into `config.toml`. * **Verify & list models**: it queries the provider's live model list and shows a short, curated shortlist (newest per family — pick by number, accept the default, or type any id). For Ollama it lists whatever you have installed. * **Write** `config.toml`, plus a `.env` **only when you enter a new key** that isn't already in your environment. The `.env` holds a secret — add it to your `.gitignore`. `aura` loads `.env` automatically at startup, so a config generated here runs as-is. ### Flags | Flag | Description | | --------------------- | ------------------------------------------------------------------------------- | | `-o, --output ` | Output config path (default `config.toml`). | | `--provider

` | Provider: openai, anthropic, bedrock, gemini, ollama, openrouter. | | `--model ` | Model id (skips the model picker). | | `--api-key-env ` | Env var to read the key from (default per provider). | | `--region ` | AWS region (bedrock only). | | `--base-url ` | Base URL (ollama only; default `http://localhost:11434`). | | `--name ` | Agent name written to the config (default `assistant`). | | `--offline` | Skip live model-list verification. | | `--non-interactive` | Fail on missing values instead of prompting (automatic when stdin isn't a TTY). | | `--force` | Overwrite an existing config without asking. | For scripted/CI use, supply the required values as flags: ```bash theme={null} aura init --provider openai --model gpt-4o --non-interactive ``` *** ## Backends AURA CLI supports two backends, selected by the presence of `--api-url`: | Backend | When | Dependencies | Tools | | -------------------------------- | ------------------- | ---------------------------------------- | -------------------------------------------------------------------------------- | | **Direct** (standalone, default) | No `--api-url` flag | Full AURA stack (agents, MCP, providers) | MCP tools from TOML config; CLI local tools when `--enable-client-tools` is set | | **HTTP** | `--api-url ` | Lightweight — just HTTP client | Server-side MCP tools always; CLI local tools when both sides opt in (see below) | Both backends produce identical SSE event streams and share the same stream parser, so all CLI features (stream panel, tool display, orchestration events, etc.) work identically regardless of backend. ### Feature flag: `standalone-cli` The `standalone-cli` Cargo feature is **enabled by default**. It includes the full agent framework, MCP integration, and provider support. To build a lightweight HTTP-only client with no agent or MCP dependencies, disable default features: ```bash theme={null} # Default build — standalone + HTTP cargo build -p aura-cli # HTTP-only — small binary, fast compile, no agent dependencies cargo build -p aura-cli --no-default-features ``` *** ## Features * **Interactive REPL** with conversation history, streaming responses, and markdown rendering * **One-shot mode** for scripting and pipelines (`--query`) — stdout is the raw assistant response, no markers or markdown rendering * **Local tool execution** — the model can read files, search code, list directories, run shell commands, and edit files on your behalf * **Standalone mode** (default) — run agents directly from TOML config without a web server * **Conversation persistence** — pick up where you left off with `--resume` or `/resume` * **Tab completion** — cycle through matching slash commands, models, and conversations with `Tab` / `Shift+Tab`; your typed prefix is preserved as you narrow the matches * **Model selection** — browse and select models from the server (or loaded configs in standalone mode) * **Permission system** — control which local tools are allowed, denied, or prompted before execution * **SSE streaming** — real-time token-by-token output with a toggleable event panel * **Auto-compaction** — automatic context management when conversations grow large * **Mid-stream commands** — execute slash commands while the model is still streaming * **Works with any OpenAI-compatible API** — not locked to a single provider *** ## Environment Variables | Variable | Description | Default | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | | `AURA_API_URL` | Base API URL (paths like `/v1/chat/completions` are appended automatically) | `http://localhost:8080` | | `AURA_API_KEY` | Bearer token for authentication | *(none)* | | `AURA_MODEL` | Model name to use (in standalone mode, selects agent by name/alias) | *(none — omitted from request, server picks default)* | | `AURA_EXTRA_HEADERS` | Additional HTTP headers as comma-separated `key:value` pairs (e.g. `x-chat-session-id:foo,authorization:Bearer …`); overrides the auto-injected `x-chat-session-id` | *(none)* | | `AURA_ENABLE_FINAL_RESPONSE_SUMMARY` | Generate a one-line LLM-based title for each final response (adds an extra round-trip per turn). Set to `true` or `1` to enable. | `false` | | `AURA_ENABLE_CLIENT_TOOLS` | Advertise CLI local tools to the model and execute them locally (see [Client-Side Tools](#client-side-tools)) | `false` | | `AURA_LOG_FILE` | Path to a file for diagnostic tracing logs. Unset → no logging. Set → events are appended to the file in both REPL and one-shot mode (see [Logging](#logging)) | *(none — no logging)* | *** ## Command Line Arguments ``` aura [OPTIONS] ``` | Flag | Env Equivalent | Description | | ------------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | `--api-url ` | `AURA_API_URL` | Base API URL | | `--api-key ` | `AURA_API_KEY` | Bearer token for authentication | | `--model ` | `AURA_MODEL` | Model name (HTTP: starting model; standalone: selects agent by name/alias) | | `--system-prompt ` | — | System prompt (HTTP: see note below; standalone: append/replace agent prompt) | | `--query ` | — | Run a single query and exit (one-shot mode) | | `--resume ` | — | Resume a previous conversation by ID or prefix | | `--force` | — | Bypass warnings and non-critical errors (useful in one-shot/query mode) | | `--enable-client-tools[=]` | `AURA_ENABLE_CLIENT_TOOLS` | Advertise CLI local tools to the model (default: disabled — see [Client-Side Tools](#client-side-tools)) | | `--enable-final-response-summary[=]` | `AURA_ENABLE_FINAL_RESPONSE_SUMMARY` | Generate a one-line LLM title for each final response (adds an extra round-trip per turn; default: disabled) | | `--standalone` | — | Force standalone mode (default when `--api-url` is absent; mutually exclusive with `--api-url` flag, but overrides `AURA_API_URL` env var) | | `--config ` | — | Path to TOML agent config file or directory (standalone mode; defaults to `config.toml`) | | `--log-file ` | `AURA_LOG_FILE` | Append diagnostic tracing logs to this file. Omit for no logging (see [Logging](#logging)) | **Precedence:** CLI flags > environment variables > project `cli.toml` > global `cli.toml` > defaults. *** ## Configuration File The CLI looks for a TOML preferences file in two places, with the project file overriding the global one on a per-field basis: | File | Purpose | | -------------------------- | ------------------------------------------------------- | | `~/.aura/cli.toml` | Global defaults (across all projects) | | `/.aura/cli.toml` | Project-local override, found by walking up from `$PWD` | The project lookup walks up from the current working directory until it finds a `.aura/` directory (the same convention used by `.git`, `.editorconfig`, etc.). Run the CLI from anywhere inside your project tree and the closest `.aura/cli.toml` wins. `$HOME` is explicitly skipped so the global file is never picked up twice. **Renamed from `config.toml`.** Older versions read `~/.aura/config.toml`. The file is still read with a one-time deprecation warning — rename it to `cli.toml` at your convenience. The old name collided with AURA **agent** config TOMLs and will stop being read in a future release. ```toml theme={null} # ~/.aura/cli.toml or /.aura/cli.toml api_url = "https://api.example.com" api_key = "your-api-key" model = "gpt-4o" system_prompt = "You are a helpful assistant." enable_client_tools = true # opt in to local tool execution; default is false log_file = "/tmp/aura.log" # append-only; see Logging section below ``` **Note on system prompts:** In **HTTP mode**, `--system-prompt` is intended for OpenAI-compatible backends that support system messages. **AURA's server ignores system role messages** — the CLI will prompt you to confirm whether you're connecting to AURA or another service. In **standalone mode**, `--system-prompt` can append to or replace the agent's TOML-configured system prompt (you'll be asked which). In one-shot mode (`--query`), standalone silently appends; HTTP mode requires `--force`. **Don't commit secrets.** A project `cli.toml` checked into source control will share `api_key` with everyone who clones the repo. Keep secrets in `~/.aura/cli.toml`, in `AURA_API_KEY`, or pass them with `--api-key`. Any value set here is overridden by environment variables or CLI flags. *** ## One-Shot Mode `--query ` runs a single round of the conversation and exits. The output contract is strict: **stdout contains only the raw assistant response** — exactly what the model produced, with no bullet markers, no themed headers, no tool-execution summaries, no markdown rendering, and no trailing response-summary line. Anything that isn't the response goes to **stderr**: * Diagnostic logs from `--log-file` / `AURA_LOG_FILE` (file destination unchanged; the file is the same destination in REPL and one-shot mode) * Permission prompts for local tools (interactive, on TTY) * Errors and warnings (with `error:` / `warning:` prefixes, no markers) * The "no result for server tool X — set `AURA_CUSTOM_EVENTS=true`" hint This means typical pipe usage works without scrubbing: ```bash theme={null} aura --query "summarize the README" > summary.md aura --query "list three ideas as JSON" | jq . aura --query "what's the version?" 2>/dev/null | tee log.txt ``` Exit code follows the standard contract: `0` ⇒ stdout is the complete response; non-zero ⇒ stderr explains why and stdout is empty. The REPL retains its rich formatting (bullet markers, markdown rendering, tool-call summaries, animated headers). The strict-output rules above apply only to `--query` mode. *** ## Interactive Commands Once inside the REPL, the slash commands below are available. All slash commands can be executed while output is streaming. | Command | Description | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | | `/help` | Show available commands and keyboard shortcuts | | `/clear` | Start a new conversation (saves the current one first) | | `/expand` | Toggle expanded/compact tool call view | | `/stream` | Toggle SSE event stream panel | | `/conversations` | List saved conversations | | `/resume ` | Resume a saved conversation by ID prefix or name | | `/rename ` | Rename the current conversation | | `/model ` | Browse and select a model (see [Model Selection](#model-selection)) | | `/mcp` | List the active agent's configured MCP servers, or set one up with a guided wizard (see [Manage MCP Servers](#manage-mcp-servers)) | | `/style [name]` | Switch visual style: `normal`, `high-contrast`, `no-colors` | | `/quit` or `/exit` | Exit the REPL | ## Manage MCP Servers `/mcp` lists the active agent's configured MCP servers (name, transport, and target). It works in both standalone and HTTP modes. `/mcp add` launches an interactive setup wizard. It isn't available in HTTP-mode builds. The wizard walks you through the following steps: 1. Pick a server from the built-in catalog (Mezmo, PagerDuty, Datadog, or Kubernetes), or configure a custom server. 2. For a custom server, choose the transport (`http_streamable`, `sse`, or `stdio`) and enter the URL or command, plus an optional authentication header. 3. Name the server and provide credentials. You can reuse an existing environment variable or enter a value with masked input. 4. Confirm the setup. The wizard verifies the connection, then previews the exact `[mcp.servers.]` block before writing the config. The preview also includes a wildcard scratchpad entry that matches every tool, which the wizard adds automatically, without asking: `[mcp.servers..scratchpad]` with `"*" = { min_tokens = 5120 }`. This entry sends any tool output over `min_tokens` (default `5120`) to scratchpad storage instead of the context window. Because the scratchpad entry is part of the same preview, declining the write declines it too. The wizard appends the `[mcp.servers.]` block to your loaded single-file agent TOML config. Configuration-directory setups are not supported. Secrets go to a `.env` file next to your config, never into the TOML file, which uses `{{ env.VAR }}` placeholders. For orchestrated configs, the wizard can scope the new server's tools to specific workers through `mcp_filter`. You must restart AURA to activate a newly added server; there is no hot reload. The wizard supplements manual TOML editing rather than replacing it. See [`[mcp]`](/aura/configuration-reference#mcp) in the configuration reference for manual and advanced MCP setup, including the three transports. The scratchpad entry stays inert until agent-level scratchpad is enabled, and the wizard does not enable it for you. If your config lacks `[agent.scratchpad] enabled = true`, the wizard prints a note that interception is configured but inactive. To activate it, add `[agent.scratchpad] enabled = true`, a top-level `memory_dir`, and `[agent.llm].context_window` to your config, then see the [Scratchpad guide](/aura/scratchpad) for the full setup. *** ## Keyboard Shortcuts | Key | Action | | ----------------------- | ------------------------------------------------------------------------------ | | `Enter` | Submit input | | `Ctrl+C` | Cancel the current streaming request, or exit if idle | | `Ctrl+L` | Clear the current input line; if the line is empty, clear the screen | | `Tab` | Cycle forward through slash command and argument matches (`/model`, `/resume`) | | `Shift+Tab` | Cycle backward through matches | | `Esc` | Cancel tab-completion selection, or exit stream panel focus | | `Up` / `Down` | Navigate input history | | `Page Up` / `Page Down` | Jump 10 entries through input history | When the **stream panel** is visible, arrow keys and page keys scroll through SSE events instead. Press `Esc` to exit stream panel focus. *** ## Model Selection Use `/model` to browse and select which model to use for requests. ``` /model # list all available models /model gpt # filter models matching "gpt" /model gpt-4o # narrow to "gpt-4o"; press Enter to select the unique match ``` `/model` opens a numbered picker with one model per row. A `Name` and `Description` header appears when at least one listed model has a description and the terminal is wide enough for a description column. `❯` marks the row you have Tab-selected, and `✓` marks the model currently in use: ``` Name Description 1. Mezmo Anthropic SRE Agent Anthropic-backed config to solve all of your production problems ❯ 2. Mezmo OpenAI SRE Agent ✓ OpenAI-backed config for the same production problems ``` Descriptions are truncated to fit the terminal width. When the terminal is too narrow for a usable description column, the column is dropped rather than truncated to a stub. A listed model without a description shows a blank cell in the `Description` column; the column itself stays, and only a list where no model has a description drops it. Descriptions come from each agent config's `[agent].description` (see the [configuration reference](/aura/configuration-reference)). In **HTTP mode**, the model list, including ids and descriptions, comes from the server's `/v1/models` endpoint. In **standalone mode**, it comes from the loaded TOML configs, using each config's `alias` or `name` plus its description. Typing a filter that narrows to exactly one model still shows the unique-match auto-complete row, now including that model's description. The model list, descriptions included, is remembered per conversation, so it survives restarts. *** ## Client-Side Tools **USE AT YOUR OWN RISK.** Enabling client-side tools gives an LLM the ability to execute commands on your machine — shell commands, file reads/writes, filesystem search — with the same privileges as the user running `aura`. Treat `--enable-client-tools` as functionally equivalent to handing the model a shell prompt. See [Client-Side Tools](/aura/client-side-tools) for the full risk model, protocol mechanics, and server/client configuration. By default, AURA CLI is a **pure chat client** — no local tools are advertised to the model and the REPL never executes anything on the host. Pass `--enable-client-tools` (or set `AURA_ENABLE_CLIENT_TOOLS=true`) to opt in, at which point the model can call tools like `Shell`, `Read`, and `Update` and the REPL runs them locally with permission checks ([Permissions](#permissions)). ```bash theme={null} # Disabled (default) — chat only aura # Enabled — local tools available, gated by the permission system aura --enable-client-tools AURA_ENABLE_CLIENT_TOOLS=true aura # Explicitly disable (overrides config file) aura --enable-client-tools=false ``` **Single-agent configurations only.** Client-side tools are not supported when the selected config has `[orchestration].enabled = true` — tools advertised to an orchestrated config are dropped with a warning. **Both the CLI and the server agent must opt in** for local tools to fire — see [Client-Side Tools: Backend symmetry](/aura/client-side-tools#backend-symmetry) for the full mode/effective-behavior table. Precedence for resolving the CLI flag: `--enable-client-tools` argument or `AURA_ENABLE_CLIENT_TOOLS` env var > `/.aura/cli.toml` > `~/.aura/cli.toml` (`enable_client_tools = true|false`) > default (`false`). If local tools never fire when you expect them to, check that the agent's TOML has `enable_client_tools = true` (and the config is single-agent) as well as the CLI flag — in standalone mode the CLI prints a startup warning when the flag is set but no loaded config opts in. *** ## Permissions AURA CLI includes a permission system that controls which local tools the model is allowed to execute. Permission rules only matter when client-side tools are enabled (see [Client-Side Tools](#client-side-tools)). Configure permissions by creating a `.aura/permissions.json` file in your project directory: ```json theme={null} { "permissions": { "allow": ["ListFiles(*)", "Read(*)", "FindFiles(*)", "SearchFiles(*)"], "deny": ["Shell(*)"] } } ``` * **Allow rules** — matching tools execute immediately without prompting. * **Deny rules** — matching tools are blocked with a guidance message. * **No match** — tools with no matching rule prompt you for approval. The CLI walks up from the current working directory to find the closest `.aura/permissions.json`, the same way it finds `cli.toml`. Permissions are **project-scoped only** — there is no `~/.aura/permissions.json`. Running the CLI from outside any project directory means no permissions are loaded, and every local tool call prompts. **Renamed from `settings.json`.** Older versions read `.aura/settings.json`. The file is still read with a one-time deprecation warning; new "always allow" rules accepted at the prompt are written to `permissions.json`, migrating any existing rules forward. ### Available Local Tools | Tool | Description | | ---------------- | ------------------------------------------------------------------- | | `Read` | Read file contents (supports chunked reading with offset and limit) | | `ListFiles` | List directory contents | | `SearchFiles` | Search file contents with regex or literal patterns | | `FindFiles` | Find files recursively by glob pattern | | `FileInfo` | Get file or directory metadata | | `Shell` | Execute shell commands (last resort) | | `Update` | Signal intent to modify or create files | | `CompactContext` | Compact conversation history by discarding older messages | *** ## Conversations Conversations are automatically saved to `~/.aura/conversations/` and can be resumed: ```bash theme={null} aura --resume abc123 # from the CLI /conversations # list saved conversations /resume abc123 # resume by ID prefix /rename my chat # name the current conversation ``` *** ## Logging The CLI is silent by default — no tracing events are emitted unless you opt in by pointing the CLI at a log file. When set, fmt events are written to that path in **both REPL and one-shot mode** so stdout (and any pipe consuming it) stays untouched. Three places can supply the path, in precedence order: | Source | Form | | ----------- | ----------------------------- | | CLI flag | `--log-file /tmp/aura.log` | | Environment | `AURA_LOG_FILE=/tmp/aura.log` | | `cli.toml` | `log_file = "/tmp/aura.log"` | The file is opened in **append mode** and created if missing. The default filter is info-level for the aura crates and rig request handling; override it with `RUST_LOG` if you need different levels. **Log rotation, truncation, and pruning are your responsibility.** The CLI appends indefinitely — it never truncates, rotates, or compresses the file. Use `logrotate`, a cron job, `truncate -s 0`, or a shell wrapper to keep the file from growing unbounded. ### Standalone-mode OpenTelemetry When running in standalone mode (the default when `--api-url` is absent), the CLI runs the agent in-process. Set `OTEL_EXPORTER_OTLP_ENDPOINT` and the CLI will install an OpenTelemetry layer alongside (or instead of) the file fmt layer, emitting `agent.stream` → `agent.turn` → `mcp.tool_call`, with `orchestration.*` between them in orchestration mode. See [Tracing & Span Layout](/aura/tracing-spans) for the full span reference. The CLI omits the `chat_completions` / `streaming_completion` infrastructure spans because there is no HTTP layer in standalone mode — those live on a separate trace in the server. HTTP-mode CLIs skip OTel entirely: your traces come from the server process. OTel init is independent of `--log-file`; you can run with traces only (no log file), logs only (no OTel endpoint), or both. Other relevant OTel env vars, read by the `aura` crate: | Variable | Purpose | | ----------------------------- | ------------------------------------------------------------------------------------------------ | | `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector endpoint (gRPC). When unset, no OTel layer is installed even in standalone mode. | | `OTEL_SERVICE_NAME` | Resource attribute (defaults to `aura`). | | `OTEL_LOG_LEVEL` | Override the OTel layer's filter. Default captures `aura=trace`, `aura_cli=info`, and rig spans. | | `OTEL_RECORD_CONTENT` | When `true`, prompt/completion/tool args/results are recorded as span attributes. | | `OTEL_CONTENT_MAX_LENGTH` | Max bytes for content attributes (default 1000, rounded down to UTF-8 boundary). | *** ## SSE Stream Panel Toggle with `/stream` to see raw SSE events in real time. Supported event types: * `aura.tool_requested` / `aura.tool_start` / `aura.tool_complete` * `aura.usage` / `aura.tool_usage` * `aura.progress` / `aura.session_info` / `aura.reasoning` * `aura.orchestrator.*` — multi-agent orchestration events See the [Streaming API Guide](/aura/streaming-api-guide) for the full event reference. *** ## Compatibility AURA CLI speaks the standard [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat) and works with any compatible backend: * AURA by Mezmo * OpenAI API / Azure OpenAI * Local models via Ollama, LM Studio, vLLM, etc. * Any service implementing `/v1/chat/completions` ### Hybrid Tool Execution When client-side tools are enabled, both backends execute server-side tools (MCP, RAG) within the agent's stream and pause for client-side tools (Shell, Read, ...) to be executed in the REPL with permission checks. The CLI follows up with a `role: "tool"` result and the agent resumes. Server-side tool results arrive via `aura.tool_complete` SSE events; the CLI uses those rather than executing locally. When `--enable-client-tools` is off, the CLI only sees server-side tool execution. See [Client-Side Tools](#client-side-tools) for the full flow. # Client-Side Tools Source: https://docs.mezmo.com/aura/client-side-tools Risk model, protocol mechanics, and server/CLI configuration for client-side tool passthrough. **USE AT YOUR OWN RISK** **Enabling client-side tools grants the LLM the ability to call tools that execute on the *client's* machine** — with the same privileges as the user running the client. When a server agent opts in and a client (e.g. the [AURA CLI](/aura/cli-reference#client-side-tools)) advertises tools like `Shell`, `Read`, or `Update`, the LLM can invoke them and the client executes them locally. This is functionally equivalent to giving the model a shell prompt on every connecting client. **The risks are real:** * **Prompt injection.** Anything the model reads — a file, a tool output, an MCP response, a vector-store hit, a webpage retrieved by another tool — can contain instructions that hijack the model into running destructive commands (`rm -rf`, exfiltrating secrets, modifying source code, etc.). The server cannot tell a legitimate request from an injected one. * **Hallucination.** The model can confidently call the wrong tool with the wrong arguments. There is no undo for a `Shell("rm -rf ...")` invocation. * **No sandbox.** The server only forwards tool calls; execution happens client-side with full host privileges. There is no container, no `chroot`, no `syscall` filter — if you can run it from your shell, the model can run it through the client. * **Permission filters reduce blast radius but are not a security boundary.** Server-side `client_tool_filter` and the CLI's allow/deny globs control which tools the model *can ask for* or execute without prompting — not what a tool does once invoked. Globs are easy to over-grant (`Shell(*)` allows anything); treat allow-rules conservatively and prefer prompt-on-execute for anything sensitive. **Only enable on agents (and clients) where:** * You trust the model, the provider, and every data source the model can read (configs, MCP servers, vector stores, web fetches). * You trust every client that will connect with local tools enabled and the user account it runs under. * Worst-case loss (deleted files, leaked credentials, modified source) is acceptable or recoverable (version control, backups). Disabled by default on both sides. Opting in is your decision and your responsibility — and your users'. ## How it works The server honors a `tools` array on incoming chat completion requests. Whether those tools are actually attached to the LLM is a **per-agent opt-in** in TOML — there is no server-wide flag. Tools that get attached are registered as **passthrough** tools: the LLM sees them alongside any server-side MCP tools and can call them, but instead of executing server-side, the stream terminates with `finish_reason: "tool_calls"` so the client can run the tool locally and submit the result back as a `role: "tool"` follow-up. ```bash theme={null} # 1) Initial request advertising a client-side tool curl -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "stream": true, "messages": [{"role": "user", "content": "What time is it?"}], "tools": [{ "type": "function", "function": { "name": "get_current_time", "description": "Get the current time", "parameters": {"type": "object", "properties": {}} } }] }' # 2) Stream ends with finish_reason: "tool_calls". The client executes the tool # locally and submits the result back in a follow-up request: curl -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "stream": true, "tools": [ ... same tools array ... ], "messages": [ {"role": "user", "content": "What time is it?"}, {"role": "assistant", "content": null, "tool_calls": [ {"id": "call_abc", "type": "function", "function": {"name": "get_current_time", "arguments": "{}"}} ]}, {"role": "tool", "tool_call_id": "call_abc", "content": "2026-04-30T14:30:00Z"} ] }' ``` When the loaded agent doesn't opt in (the default), any `tools` field on the request is silently dropped; the server runs MCP tools as usual but never asks the client to execute anything. Per-agent opt-in is the design — accepting client-supplied tool definitions means trusting the client to execute them, so it should be a deliberate config decision. ## Single-agent configurations only Client-side tools are not supported in orchestrated (multi-agent) configurations — when `[orchestration].enabled = true`, any `tools` array on the request is dropped with a warning. The reason: the passthrough mechanism requires terminating the user-facing SSE stream with `finish_reason: "tool_calls"`, which doesn't compose with the coordinator/worker pipeline. If you need local tools, use a single-agent config. ## Server-side configuration `enable_client_tools` and `client_tool_filter` are per-agent `[agent]` fields — see [`[agent]`](/aura/configuration-reference#agent) in the configuration reference for the full field table. ```toml theme={null} [agent] name = "Assistant" system_prompt = "..." enable_client_tools = true client_tool_filter = ["Read", "ListFiles", "Find*"] # optional; omitted/empty = all ``` `client_tool_filter` is a list of glob patterns matched against the request's `tools[].function.name`. An empty or omitted filter means all client tools are available. A request that supplies tools never reaches an agent that did not opt in. ## Client-side configuration (AURA CLI) The [AURA CLI](/aura/cli-reference) is the reference client implementation. `--enable-client-tools` (or `AURA_ENABLE_CLIENT_TOOLS=true`) turns on advertisement of its local tools (`Shell`, `Read`, `Update`, `ListFiles`, `FindFiles`, `SearchFiles`, `FileInfo`); the CLI's permission system (`.aura/permissions.json`) then governs which of those actually execute without a prompt. See the [CLI Reference](/aura/cli-reference) for the flag/env var syntax, precedence rules, and the permission system. ## Backend symmetry Both halves of the system gate this independently — **and both must opt in for local tools to fire**, in HTTP mode and standalone mode alike. Standalone runs the same handler path in-process, so the agent's `[agent].enable_client_tools = true` is required there too. | Mode | TOML side (`[agent].enable_client_tools`) | Client side (`--enable-client-tools`) | Effective | | ------------------------------------ | ----------------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------ | | Standalone, single-agent, both on | `true` | required | Local tools fully enabled | | Standalone, single-agent, TOML off | absent / `false` | (any) | Tools advertised but the in-process server **silently drops** them; CLI prints a startup warning | | HTTP, single-agent, both on | `true` | required | Local tools fully enabled | | HTTP, single-agent, agent off | absent / `false` | (any) | Tools advertised but server **silently drops** them — no local tools | | Standalone or HTTP, **orchestrated** | any orchestrated config | (any) | Tools advertised but server **drops with warning** — orchestration unsupported | | Both off | absent / `false` | client without flag | Pure chat | If local tools never fire when you expect them to, check that **both** sides are opted in: the agent's TOML has `enable_client_tools = true` (and the config is single-agent), and the client has local tools enabled. # Configuration Reference Source: https://docs.mezmo.com/aura/configuration-reference Complete TOML field reference for AURA — agent identity, LLM providers, MCP, vector stores, scratchpad, skills, orchestration, HITL, and session storage. AURA is configured with a single TOML file. The file path defaults to `config.toml` in the working directory; override it with the `CONFIG_PATH` environment variable. ```bash theme={null} CONFIG_PATH=configs/my-agent.toml cargo run --bin aura -- webserver ``` To validate a config file, start the web server or CLI against it — both validate immediately and exit with a clear error if parsing fails, before binding to any port or entering the REPL: ```bash theme={null} cargo run --bin aura -- webserver --config your-config.toml # exits on parse error before binding cargo run --bin aura -- --config your-config.toml # exits on parse error before REPL ``` ## Environment Variable Interpolation Any string value in the config can reference an environment variable using `{{ env.VAR_NAME }}`. An optional `| default: 'value'` fallback prevents a hard error when the variable is unset. ```toml theme={null} api_key = "{{ env.OPENAI_API_KEY }}" Authorization = "Bearer {{ env.GITHUB_PERSONAL_ACCESS_TOKEN | default: '' }}" ``` *** ## Root-Level Fields | Field | Type | Default | Description | | ------------ | ------ | ------- | ------------------------------------------------------------------------------------------------------------------ | | `memory_dir` | string | — | Base directory for scratchpad storage and orchestration artifact persistence. Required when scratchpad is enabled. | ```toml theme={null} memory_dir = "/tmp/aura-sessions" ``` *** ## `[agent]` Defines the agent identity, system prompt, and behavioral settings. | Field | Type | Default | Description | | ----------------------- | --------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | *required* | Display name. Doubles as the model identifier clients send in requests. | | `alias` | string | — | Stable identifier for model selection. Clients send this as the `model` field. Useful when the name contains spaces. | | `description` | string | — | Optional one-line, human-readable summary of the agent. Use it to distinguish multiple loaded configs that would otherwise differ only by `name` or `alias`. Display-only. It appears in `/v1/models`, `/aura/info`, and the CLI `/model` picker, but is never part of runtime settings and never sent to the model. Omitted from `/v1/models` and `/aura/info` responses when unset. Not to be confused with the required per-worker `description` under `[orchestration]`, which guides planning and is not shown to people. It is free-form text with no enforced length limit; the CLI `/model` picker collapses whitespace and truncates it to fit the terminal. | | `system_prompt` | string | *required* | The agent's system prompt. Multi-line strings use TOML `"""..."""` syntax. | | `turn_depth` | integer | `5` | Max tool-call rounds per user turn. Acts as a failsafe to prevent models from spinning out in unbounded tool-call loops. | | `nudge_last_turn` | bool | `false` | Append a wrap-up warning to tool output on the final turn before `turn_depth` terminates the run (orchestration workers are told to call `submit_result`), rather than silently losing all gathered work. | | `nudge_turns_remaining` | integer | — | Start wrap-up warnings when N or fewer tool-calling turns remain. Independent of `nudge_last_turn`; both default to off and can be enabled separately. | | `mcp_filter` | list of strings | — | Glob patterns selecting which MCP tools to expose. When omitted, all tools are included. | | `enable_client_tools` | bool | `false` | Allow the agent to invoke client-side tools advertised in the request. See [Client-Side Tools](/aura/client-side-tools) for security implications. | | `client_tool_filter` | list of strings | — | Glob patterns narrowing which client tools are allowed (requires `enable_client_tools = true`). | | `model_owner` | string | *(LLM provider name)* | Overrides the `owned_by` field in `/v1/models` responses. | | `created_at` | integer | *(current time)* | Creation timestamp in milliseconds since epoch. Shown in `/v1/models` responses. | | `hidden` | bool | `false` | Hides this agent from the `/v1/models` list. Useful for agents meant only for internal callers. | ```toml theme={null} [agent] name = "DevOps Assistant" alias = "devops" description = "General-purpose assistant with tool access" system_prompt = """ You are a DevOps assistant with access to GitHub. Help with code review, PR management, and repo exploration. """ turn_depth = 10 mcp_filter = ["get_*", "list_*", "search_*"] model_owner = "acme" hidden = false ``` *** ## Multiple Agents `CONFIG_PATH` can point to a single TOML file or a directory of `.toml` files. When pointed at a directory, AURA loads every `.toml` file and serves each as a selectable agent: ``` configs/ ├── research-assistant.toml ├── devops-agent.toml └── code-reviewer.toml ``` ```bash theme={null} CONFIG_PATH=configs/ cargo run --bin aura -- webserver ``` Each agent is identified by its `alias` (if set) or `name`. Clients discover available agents via `GET /v1/models` and select one by passing its identifier as the `model` field in chat completion requests — the same field tools like LibreChat, OpenWebUI, and CLI clients use to present a model picker. Agent selection follows this order: 1. If only one config is loaded, it is always used (the `model` field is ignored). 2. Otherwise, `model` is matched first, then `DEFAULT_AGENT` if `model` is absent. 3. Returns a 400 error if multiple configs are loaded and neither `model` nor `DEFAULT_AGENT` is supplied at all. 4. Returns a 404 error if a `model` or `DEFAULT_AGENT` value is supplied but matches no loaded config. ```toml theme={null} [agent] name = "DevOps Assistant" alias = "devops" # clients send "model": "devops" description = "DevOps assistant with GitHub access" system_prompt = "You are a DevOps expert." model_owner = "mezmo" # override owned_by in /v1/models (defaults to LLM provider) ``` Aliases must be unique across all loaded configs. If two configs share the same `name` and neither has an alias, loading fails with a validation error. **Hidden agents** are excluded from `GET /v1/models` and the CLI's `/model` list but remain fully accessible when a caller targets them by exact name or alias. Set `hidden = true` to hide agents that are in development, restricted to known callers, or should not appear in model pickers (LibreChat, OpenWebUI, etc.): ```toml theme={null} [agent] name = "Internal Triage Agent" hidden = true # excluded from /v1/models and CLI model list; still callable by name system_prompt = "..." ``` *** ## `[agent.llm]` Configures the LLM provider. The `provider` field is a discriminant that selects the variant and its required fields. ### Common Fields These fields are available on all providers except where noted. | Field | Type | Default | Description | | ------------------- | ------- | ------- | --------------------------------------------------------------------------------------------------------------------------- | | `provider` | string | — | **Required.** One of: `openai`, `anthropic`, `bedrock`, `gemini`, `ollama`, `openrouter`. | | `model` | string | — | **Required.** Model identifier (e.g. `gpt-4o`, `claude-sonnet-4-20250514`). | | `max_tokens` | integer | — | Maximum tokens in the response. | | `context_window` | integer | — | Context window size in tokens. Required when scratchpad is enabled. Used for usage reporting in `aura.session_info` events. | | `temperature` | float | — | Sampling temperature (0.0–2.0). Higher values increase randomness. | | `additional_params` | table | — | Provider-specific parameters merged into the API request body. Useful for features like extended thinking. | ### `provider = "openai"` ```toml theme={null} [agent.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-4o" max_tokens = 8192 context_window = 128000 temperature = 0.7 # base_url = "https://api.openai.com/v1" # custom endpoint # reasoning_effort = "medium" # none, minimal, low, medium, high, xhigh (provider/model support varies) ``` Extra fields: | Field | Type | Default | Description | | ------------------ | ------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `api_key` | string | — | **Required.** OpenAI API key. | | `base_url` | string | — | Override the API base URL (useful for compatible proxies). | | `reasoning_effort` | string | — | `none`, `minimal`, `low`, `medium`, `high`, or `xhigh`. Controls reasoning depth. Support varies by model and version. Aura passes the value straight to the OpenAI API, which returns a 400 error for combinations of `model` and `reasoning_effort` that it does not support. | ### `provider = "anthropic"` ```toml theme={null} [agent.llm] provider = "anthropic" api_key = "{{ env.ANTHROPIC_API_KEY }}" model = "claude-sonnet-4-20250514" context_window = 200000 # Enable extended thinking: [agent.llm.additional_params] thinking = { type = "adaptive", budget_tokens = 8000 } ``` Extra fields: | Field | Type | Default | Description | | ---------- | ------ | ------- | -------------------------------- | | `api_key` | string | — | **Required.** Anthropic API key. | | `base_url` | string | — | Override the API base URL. | ### `provider = "bedrock"` Uses AWS credentials from the environment (AWS profile, IAM role, or environment variables). No API key field. ```toml theme={null} [agent.llm] provider = "bedrock" model = "us.anthropic.claude-3-5-sonnet-20241022-v2:0" region = "{{ env.AWS_REGION }}" profile = "default" # optional context_window = 200000 ``` Extra fields: | Field | Type | Default | Description | | --------- | ------ | ------- | --------------------------------------------------------------- | | `region` | string | — | **Required.** AWS region (e.g. `us-east-1`). | | `profile` | string | — | AWS profile name. Uses the default credential chain if omitted. | ### `provider = "gemini"` ```toml theme={null} [agent.llm] provider = "gemini" api_key = "{{ env.GOOGLE_API_KEY }}" model = "gemini-2.0-flash" context_window = 1000000 ``` Extra fields: | Field | Type | Default | Description | | ---------- | ------ | ------- | ----------------------------- | | `api_key` | string | — | **Required.** Google API key. | | `base_url` | string | — | Override the API base URL. | ### `provider = "ollama"` No API key required. Defaults to `http://localhost:11434`. ```toml theme={null} [agent.llm] provider = "ollama" model = "qwen3:30b-a3b" base_url = "http://localhost:11434" context_window = 32768 fallback_tool_parsing = true # Pass Ollama-specific parameters: [agent.llm.additional_params] num_ctx = 32768 top_k = 40 ``` Extra fields: | Field | Type | Default | Description | | ----------------------- | ------ | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `base_url` | string | `"http://localhost:11434"` | Ollama server URL. | | `fallback_tool_parsing` | bool | `false` | Parse tool calls from streamed text output. Enable for models that emit tool calls as text (e.g. some qwen3 variants) rather than native tool-call structures. | ### `provider = "openrouter"` Access 300+ models through a single API key. Uses OpenRouter's reasoning wire format (`reasoning` + `reasoning_details`). For other OpenAI-compatible APIs (Fireworks, Together), use `provider = "openai"` with `base_url` instead. ```toml theme={null} [agent.llm] provider = "openrouter" api_key = "{{ env.OPENROUTER_API_KEY }}" model = "anthropic/claude-sonnet-4" # base_url = "https://custom-endpoint/v1" # OpenRouter-compatible proxies only ``` Extra fields: | Field | Type | Default | Description | | ---------- | ------ | ------- | --------------------------------- | | `api_key` | string | — | **Required.** OpenRouter API key. | | `base_url` | string | — | Override the base URL. | *** ## `[mcp]` Configures Model Context Protocol (MCP) tool servers. | Field | Type | Default | Description | | ------------------ | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sanitize_schemas` | bool | `true` | Sanitize tool schemas for OpenAI function-calling compatibility. Fixes `anyOf` unions, missing types, and optional parameters that strict mode rejects. | ```toml theme={null} [mcp] sanitize_schemas = true ``` ### `[mcp.servers.]` Each server is a named entry under `[mcp.servers]`. The `transport` field selects the connection type. Tool names are not namespaced by server. If two servers register a tool with the same name, registration overwrites the earlier entry — since registration order is not the same as config declaration order, which tool ends up callable is effectively arbitrary, not deterministically "first" or "last" in TOML order ([#186](https://github.com/mezmo/aura/issues/186)). #### `transport = "http_streamable"` (recommended) Connects to an MCP server over HTTP using the current MCP streamable transport (post-2025-11-05). ```toml theme={null} [mcp.servers.my_tools] transport = "http_streamable" url = "http://localhost:8081/mcp" description = "My tool server" [mcp.servers.my_tools.headers] Authorization = "Bearer {{ env.MCP_TOKEN }}" ``` #### `transport = "sse"` Connects using the legacy SSE-based MCP protocol. Supports the same `headers`, `headers_from_request`, and `scratchpad` options as `http_streamable`. ```toml theme={null} [mcp.servers.legacy_server] transport = "sse" url = "http://localhost:8082/sse" ``` #### `transport = "stdio"` Spawns a local child process. Each agent request creates its own process instance. `cmd` is a list where `cmd[0]` is the executable and `cmd[1..]` are fixed arguments that are part of the command (e.g. a script path). `args` are additional arguments appended after. ```toml theme={null} [mcp.servers.everything] transport = "stdio" cmd = ["npx"] args = ["-y", "@modelcontextprotocol/server-everything"] [mcp.servers.everything.env] MY_VAR = "value" # Script-based example: # cmd = ["python3", "/opt/mcp-servers/weather.py"] # args = ["--verbose"] ``` #### Static Headers Add static headers to every request to an HTTP or SSE server: ```toml theme={null} [mcp.servers.my_tools.headers] Authorization = "Bearer {{ env.MCP_TOKEN }}" X-Tenant-ID = "acme" ``` #### Header Forwarding (`headers_from_request`) Forward headers from the incoming API request to the MCP server. The table maps outgoing header name → incoming request header name. Useful for per-user auth delegation. ```toml theme={null} [mcp.servers.my_tools.headers_from_request] # Outgoing header = Incoming request header Authorization = "x-user-token" X-User-ID = "x-user-id" ``` When `headers_from_request` is set, the forwarded header takes precedence over any matching static header from `headers`. #### Per-Tool Scratchpad Thresholds Override when a tool's output gets intercepted by the scratchpad system. Keys are glob patterns matched against tool names; the most specific (longest) pattern wins. ```toml theme={null} [mcp.servers.my_tools.scratchpad] "get_large_*" = { min_tokens = 100 } # intercept all large-prefixed tools "get_small_*" = { min_tokens = 99999 } # effectively disable interception "get_report" = { min_tokens = 200 } # custom threshold for one tool ``` *** ## `[agent.scratchpad]` Controls context window management. When enabled, large MCP tool outputs are saved to disk and replaced with a file pointer. The agent then uses eight exploration tools (`head`, `slice`, `grep`, `schema`, `item_schema`, `get_in`, `iterate_over`, `read`) to selectively read the data it needs. See [Scratchpad](/aura/scratchpad) for the full feature guide. Requires `memory_dir` to be set at the root level and `context_window` to be set on `[agent.llm]`. Orchestration also accepts the legacy `[orchestration.artifacts].memory_dir` as a fallback when the top-level field is absent. | Field | Type | Default | Description | | ----------------------- | ------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | bool | `false` | Activate scratchpad interception. | | `context_safety_margin` | float | `0.20` | Fraction of the context window (0.0–1.0) reserved for reasoning and output. Scratchpad intercepts output before this budget is exhausted. | | `max_extraction_tokens` | integer | `10000` | Maximum tokens a single exploration tool call may return, preventing a single read from flooding the context. | | `turn_depth_bonus` | integer | `6` | Extra tool-call turns added when scratchpad is active, to leave room for exploration calls after interception. | ```toml theme={null} memory_dir = "/tmp/aura-sessions" [agent] turn_depth = 10 [agent.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-4o" context_window = 128000 [agent.scratchpad] enabled = true context_safety_margin = 0.20 max_extraction_tokens = 10000 turn_depth_bonus = 6 ``` *** ## `[agent.skills]` Points the agent at directories of on-demand skills. Skills follow the [Agent Skills specification](https://agentskills.io/specification): each skill is a subdirectory containing a `SKILL.md` file with YAML frontmatter (`name`, `description`). The `name` must match the directory name and consist of lowercase alphanumerics and hyphens only (1–64 characters, no leading/trailing/consecutive hyphens). See [Skills](/aura/skills) for the full feature guide. ```toml theme={null} [[agent.skills.local]] source = "skills/" # relative to process CWD, or absolute [[agent.skills.local]] source = "/opt/shared-skills" ``` A `load_skill` tool is exposed to the agent that loads and executes skills on demand, along with a `read_skill_file` tool for fetching individual resource files from a skill's directory. *** ## `[[vector_stores]]` Configures RAG (Retrieval-Augmented Generation) stores. Each entry creates a `vector_search_` tool the agent can call. Multiple stores create multiple search tools. | Field | Type | Description | | ---------------- | ------ | ------------------------------------------------------------------ | | `name` | string | Unique identifier. Used in worker `vector_stores` lists. | | `context_prefix` | string | Optional hint shown to the LLM describing what the store contains. | | `type` | string | Store type: `in_memory`, `qdrant`, or `bedrock_kb`. | ### `type = "qdrant"` ```toml theme={null} [[vector_stores]] name = "docs" type = "qdrant" url = "http://localhost:6334" collection_name = "documents" context_prefix = "Technical documentation and API references" [vector_stores.embedding_model] provider = "openai" model = "text-embedding-3-small" api_key = "{{ env.OPENAI_API_KEY }}" ``` ### `type = "in_memory"` ```toml theme={null} [[vector_stores]] name = "knowledge" type = "in_memory" [vector_stores.embedding_model] provider = "openai" model = "text-embedding-3-small" api_key = "{{ env.OPENAI_API_KEY }}" ``` ### `type = "bedrock_kb"` Managed RAG uses AWS credentials, so no embedding model is needed. ```toml theme={null} [[vector_stores]] name = "company_docs" type = "bedrock_kb" knowledge_base_id = "{{ env.BEDROCK_KB_ID }}" region = "{{ env.AWS_REGION }}" # profile = "default" # optional # managed = true # optional context_prefix = "Company documentation" ``` Extra fields: | Field | Type | Default | Description | | ------------------- | ------ | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `knowledge_base_id` | string | — | **Required.** Bedrock Knowledge Base ID. | | `region` | string | — | **Required.** AWS region (e.g. `us-east-1`). | | `profile` | string | — | AWS profile name. Pins the Knowledge Base client to this profile; uses the default credential chain if omitted. | | `managed` | bool | `false` | Set to `true` when the store points at a Bedrock managed knowledge base; Aura then sends `managedSearchConfiguration` instead of `vectorSearchConfiguration` on retrieve requests. Leave it `false` or omit it for a classic vector-store-backed knowledge base. | When you set `profile`, the Knowledge Base client uses only that AWS profile's credentials, even if static AWS environment credentials (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`) are also present. When you omit `profile`, it falls back to the default AWS credential chain. This makes profile-based cross-account access reliable alongside static environment credentials used elsewhere in the deployment — for example, an IRSA (IAM Roles for Service Accounts) profile that assumes a role via web identity. Both managed and classic Bedrock knowledge bases handle embeddings for you. The `managed` flag only selects which kind of knowledge base the store points at. If you leave `managed` unset or `false` for a managed knowledge base, retrieve requests fail with a `ValidationException`. The full error reads: `Incompatible configuration: vectorSearchConfiguration is not supported for managed knowledge bases. Use managedSearchConfiguration instead.` Set `managed = true` to resolve it. ### Embedding Models Used by `in_memory` and `qdrant` types. ```toml theme={null} # OpenAI: [vector_stores.embedding_model] provider = "openai" model = "text-embedding-3-small" api_key = "{{ env.OPENAI_API_KEY }}" # AWS Bedrock: [vector_stores.embedding_model] provider = "bedrock" model = "amazon.titan-embed-text-v2:0" region = "{{ env.AWS_REGION }}" profile = "default" # optional ``` *** ## `[tools]` Enables built-in server-side tools. | Field | Type | Default | Description | | -------------- | --------------- | ------- | --------------------------------------------------------------------------------- | | `filesystem` | bool | `false` | Expose basic read-only filesystem tools (read file, list directory) to the agent. | | `custom_tools` | list of strings | `[]` | Reserved for future use. | ```toml theme={null} [tools] filesystem = true ``` *** ## `[hitl]` Human-in-the-loop (HITL) approval gates let an agent ask for permission before running selected MCP tools. `[hitl]` is the enable bit — there is no separate `enabled` field; presence of the table turns it on, and `route` is required when it's present. See [HITL](/aura/hitl) for the full route contracts, SSE lifecycle events, and current scope (single-agent vs. orchestration worker gating). | Field | Type | Default | Description | | ------------------ | --------------------- | ---------- | ---------------------------------------------------------- | | `require_approval` | list of glob patterns | `[]` | Tool-name globs that gate a matching call behind approval. | | `route` | table | *required* | The decision route — see `[hitl.route]` below. | ### `[hitl.route]` Tagged by `mode`: `"conversational"` or `"webhook"`. | Field | Type | Default | Description | | -------------- | ------- | --------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | `mode` | string | — | **Required.** `"conversational"` (attended, over an open SSE stream) or `"webhook"` (unattended, posts to a URL). | | `timeout_secs` | integer | `60` (conversational) / `300` (webhook) | Seconds to wait for a decision before failing closed. | | `url` | string | — | **Required for `webhook` mode only.** Must start with `http://` or `https://`. | ```toml theme={null} [hitl] require_approval = ["kubectl_*", "restart_*", "dangerous_*"] [hitl.route] mode = "webhook" url = "https://approvals.example.com/aura" timeout_secs = 300 ``` *** ## Session Store (Durable and Multi-Pod Deployments) Unlike every other section on this page, the session store is **not configured in TOML** — it's deployment infrastructure (one instance per server, not per-agent), so it's configured only via environment variables. By default, cross-request session state (A2A tasks, parked HITL approvals) lives in process memory — correct for a single pod, the CLI, and local dev. Behind a load balancer with multiple replicas, configure a shared Redis/Valkey backend and every cross-request flow works no matter which pod serves each request: A2A `message:send` → poll → `list` → history-by-context, A2A `subscribe`/`cancel` against a task executing on another pod, and conversational HITL approvals resolved by a `POST /v1/approvals/{id}` that lands away from the pod that parked them. ```bash theme={null} export AURA_SESSION_STORE=redis # memory (default) | redis export AURA_SESSION_STORE_URL=redis://valkey:6379 # redis:// or rediss:// (Valkey compatible) export AURA_SESSION_STORE_PREFIX=aura:prod # optional namespace (default "aura") export AURA_SESSION_STORE_CONNECT_TIMEOUT_SECS=5 # optional export AURA_SESSION_STORE_TASK_TTL_SECS=86400 # optional; 0 = no expiry ``` The server pings the backend at startup and fails fast if it is unreachable; `/health` reports the backend and its ping latency. The Redis backend requires building with the `session-store-redis` cargo feature (`cargo build --release --features aura-cli/session-store-redis`). The in-memory backend is always available. See [the session storage design doc](https://github.com/mezmo/aura/blob/main/docs/design/session-storage.md) for the design and Helm packaging roadmap. *** ## `[orchestration]` Enables multi-agent orchestration mode. A coordinator agent decomposes user queries into tasks and delegates them to specialized worker agents for parallel execution. The coordinator's system prompt comes from `[agent].system_prompt`. Workers are defined in `[orchestration.worker.]` sections. Execution loop: * `Plan`: coordinator decomposes the request into a task DAG. * `Execute`: dependency-ready tasks run in parallel waves on worker agents. * `Continue`: coordinator consolidates worker outputs and routes to a final response, replan, or clarification. Workers run with isolated task context windows and filtered MCP/vector-store access based on each worker block. | Field | Type | Default | Description | | -------------------------------- | --------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | bool | `false` | Activate orchestration mode. | | `max_planning_cycles` | integer | `3` | Maximum plan→execute→continue cycles per request. | | `max_plan_parse_retries` | integer | `3` | Retries on plan parse failure before falling back to single-task execution. | | `allow_direct_answers` | bool | `true` | Allow the coordinator to answer simple queries directly without delegating. | | `allow_clarification` | bool | `true` | Allow the coordinator to ask for clarification on ambiguous queries. | | `tools_in_planning` | string | `"summary"` | How much tool info is shown during planning: `"none"`, `"summary"` (tool names per worker), or `"full"` (names + descriptions). | | `max_tools_per_worker` | integer | `10` | Truncates the tool list per worker in the planning prompt with a `(+N more)` suffix. | | `coordinator_vector_stores` | list of strings | `[]` | Names of vector stores the coordinator can access. | | `worker_system_prompt` | string | — | Custom system prompt injected into generic (non-specialized) workers. | | `duplicate_call_nudge_threshold` | integer | `3` | Consecutive identical tool calls before appending a guidance annotation. | | `duplicate_call_block_threshold` | integer | `5` | Consecutive identical tool calls before appending an abort annotation. | ```toml theme={null} [orchestration] enabled = true max_planning_cycles = 3 allow_direct_answers = true tools_in_planning = "full" coordinator_vector_stores = ["runbooks"] memory_dir = "/tmp/aura-orchestration" # legacy flat field; prefer top-level memory_dir ``` ### `[orchestration.timeouts]` | Field | Type | Default | Description | | -------------------------------- | ------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `per_call_timeout_secs` | integer | `0` (disabled) | Per-call timeout for coordinator and worker LLM calls. Set to a positive value to enable. | | `stream_inactivity_timeout_secs` | integer | `0` (disabled) | Max seconds of silence between stream items on coordinator and worker orchestration streams before Aura fails the affected task with a no-stream-progress error. Suspends during tool execution and re-arms on each stream item. Must be below `per_call_timeout_secs` (when set) to take effect. Set to a positive value to enable. | See [request lifecycle](/aura/request-lifecycle) for the operational guidance and caveats on this timeout. ```toml theme={null} [orchestration.timeouts] per_call_timeout_secs = 120 stream_inactivity_timeout_secs = 60 ``` ### `[orchestration.artifacts]` Controls persistence and artifact promotion. | Field | Type | Default | Description | | ------------------------------------- | ------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- | | `memory_dir` | string | — | Base directory for run artifacts. Alias: `memory_path`. (Prefer top-level `memory_dir`.) | | `result_artifact_threshold` | integer | `4000` | Character threshold for writing worker results to artifact files instead of inlining them. | | `result_summary_length` | integer | `2000` | Max inline summary length when a result is promoted to an artifact. | | `session_history_turns` | integer | `3` | Max prior run manifests injected as session context in the coordinator preamble. Set to `0` to disable. | | `persistence_drain_timeout_ms` | integer | `2000` | Timeout (ms) for draining in-flight persistence writes between execution phases. | | `tool_output_artifact_threshold` | integer | `500` | Character threshold for promoting tool outputs to artifact files. | | `tool_output_duration_threshold_ms` | integer | `5000` | Duration threshold (ms) for promoting tool outputs regardless of size. | | `show_tool_reasoning_in_continuation` | bool | `false` | Include condensed tool reasoning traces in continuation prompts so the coordinator can see why workers called specific tools. | | `max_session_runs` | integer | `20` | Max run directories retained per session before oldest runs are pruned. Set to `0` to disable pruning. | ```toml theme={null} [orchestration.artifacts] memory_dir = "/tmp/aura-orchestration" result_artifact_threshold = 4000 result_summary_length = 2000 session_history_turns = 3 max_session_runs = 20 ``` ### `[orchestration.worker.]` Defines a specialized worker. Worker names must be unique case-insensitively (to avoid filesystem collisions) and non-empty. | Field | Type | Default | Description | | --------------- | --------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `description` | string | — | **Required.** One-line description shown to the coordinator during planning. | | `preamble` | string | — | **Required.** Complete system prompt for this worker (replaces the generic worker template). | | `mcp_filter` | list of strings | — | Glob patterns selecting which MCP tools this worker can access. When omitted, all tools are included; an empty list `[]` grants no tools. | | `vector_stores` | list of strings | `[]` (none) | Names of vector stores this worker can access. Workers have no RAG access by default. | | `turn_depth` | integer | *(inherits `[agent].turn_depth`)* | Max tool-call rounds for this worker. | ```toml theme={null} [orchestration.worker.operations] description = "Logs, pipelines, metrics, and system analysis" preamble = """ You are an Operations Specialist with access to observability tools. Use tools to investigate incidents — do not fabricate data. """ mcp_filter = ["mezmo_*"] vector_stores = ["runbooks"] turn_depth = 8 ``` #### Per-Worker LLM Override Workers inherit `[agent.llm]` by default. Provide `[orchestration.worker..llm]` to use a different model for a specific worker (e.g. a cheaper model for simple tasks). ```toml theme={null} [orchestration.worker.summarizer.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-4o-mini" context_window = 128000 ``` #### Per-Worker Scratchpad Override ```toml theme={null} [orchestration.worker.analyst.scratchpad] enabled = true max_extraction_tokens = 5000 ``` #### Per-Worker Skills Override `None` (field absent) inherits `[agent.skills]`. An explicit empty list disables skills. A non-empty list replaces the agent's skills entirely (no merging). ```toml theme={null} [[orchestration.worker.researcher.skills.local]] source = "skills/research" ``` *** ## Complete Examples ### Minimal: OpenAI ```toml theme={null} [agent] name = "Assistant" system_prompt = "You are a helpful assistant." [agent.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-4o" ``` ### Single Agent with MCP Tools ```toml theme={null} [mcp] sanitize_schemas = true [mcp.servers.github] transport = "http_streamable" url = "https://api.githubcopilot.com/mcp/" description = "GitHub repository operations" [mcp.servers.github.headers] Authorization = "Bearer {{ env.GITHUB_PERSONAL_ACCESS_TOKEN }}" [agent] name = "DevOps Assistant" system_prompt = """ You are a DevOps assistant with access to GitHub. Help with code review, PR management, and repo exploration. """ turn_depth = 10 [agent.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-4o" ``` ### Single Agent with Scratchpad Large tool outputs are intercepted and saved to disk; the agent uses exploration tools to read what it needs. ```toml theme={null} memory_dir = "/tmp/aura-sessions" [agent] name = "Data Analyst" system_prompt = """ You are a data analysis assistant. When you see a scratchpad pointer ([scratchpad: file_id=...]), use the exploration tools (head, grep, schema, get_in, etc.) to selectively read the data you need. Do not re-call the original tool. """ turn_depth = 10 [agent.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-4o" context_window = 128000 [agent.scratchpad] enabled = true context_safety_margin = 0.20 max_extraction_tokens = 10000 [mcp.servers.data_api] transport = "http_streamable" url = "http://localhost:9000/mcp" [mcp.servers.data_api.scratchpad] "get_dataset_*" = { min_tokens = 100 } ``` ### Multi-Agent Orchestration with Per-Worker Models ```toml theme={null} memory_dir = "/tmp/aura-orchestration" [agent] name = "SRE Coordinator" system_prompt = """ You are an SRE coordinator. Decompose observability tasks into sub-tasks and delegate to the appropriate specialist worker. - Kubernetes cluster state and workloads: delegate to k8s-specialist - Prometheus metrics and alerts: delegate to metrics-analyst """ [agent.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-4o" context_window = 128000 [mcp.servers.kubernetes] transport = "http_streamable" url = "http://localhost:8081/mcp" [mcp.servers.prometheus] transport = "http_streamable" url = "http://localhost:8082/mcp" [orchestration] enabled = true max_planning_cycles = 3 tools_in_planning = "full" allow_direct_answers = true [orchestration.timeouts] per_call_timeout_secs = 120 [orchestration.worker.k8s-specialist] description = "Kubernetes cluster inspection: namespaces, workloads, pods, events" turn_depth = 8 mcp_filter = ["namespaces_list", "pods_*", "resources_*", "events_list"] preamble = """ You are a Kubernetes Specialist. Use tools to inspect the cluster. Never guess cluster state — always call the relevant tool first. """ [orchestration.worker.k8s-specialist.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-4o" context_window = 128000 [orchestration.worker.metrics-analyst] description = "Prometheus metrics queries, target health, and alert status" turn_depth = 8 mcp_filter = ["execute_query", "execute_range_query", "list_metrics", "get_targets"] preamble = """ You are a Prometheus Analyst. Query metrics directly — do not fabricate values. Report metric names, labels, and exact values from tool results. """ [orchestration.worker.metrics-analyst.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-4o-mini" context_window = 128000 ``` ### RAG with Qdrant ```toml theme={null} [[vector_stores]] name = "docs" type = "qdrant" url = "http://localhost:6334" collection_name = "documents" context_prefix = "Technical documentation and API references" [vector_stores.embedding_model] provider = "openai" model = "text-embedding-3-small" api_key = "{{ env.OPENAI_API_KEY }}" [[vector_stores]] name = "runbooks" type = "qdrant" url = "http://localhost:6334" collection_name = "runbooks" context_prefix = "Operational runbooks and incident response procedures" [vector_stores.embedding_model] provider = "openai" model = "text-embedding-3-small" api_key = "{{ env.OPENAI_API_KEY }}" [agent] name = "Knowledge Assistant" system_prompt = """ You are a knowledge assistant. Use the vector_search_docs and vector_search_runbooks tools to ground your answers in documentation. """ [agent.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-4o" ``` ### Local Ollama (No API Key) ```toml theme={null} [agent] name = "Local Assistant" system_prompt = "You are a helpful assistant running locally." [agent.llm] provider = "ollama" model = "qwen3:30b-a3b" base_url = "http://localhost:11434" context_window = 32768 fallback_tool_parsing = true [agent.llm.additional_params] num_ctx = 32768 ``` ### AWS Bedrock with Knowledge Base ```toml theme={null} [[vector_stores]] name = "company_kb" type = "bedrock_kb" knowledge_base_id = "{{ env.BEDROCK_KB_ID }}" region = "{{ env.AWS_REGION }}" context_prefix = "Company internal documentation" [agent] name = "Enterprise Assistant" system_prompt = """ You are an enterprise assistant. Use the company knowledge base to answer questions grounded in internal documentation. """ [agent.llm] provider = "bedrock" model = "us.anthropic.claude-3-5-sonnet-20241022-v2:0" region = "{{ env.AWS_REGION }}" context_window = 200000 ``` # Example Configs Source: https://docs.mezmo.com/aura/example-configs Example TOML configurations and advanced quickstarts for AURA agents. Example TOML configurations and advanced quickstarts for AURA agents. **New to AURA?** Start with the [Quickstart](/aura/quickstart). It installs AURA, generates a config, and connects it to your tools. Come back here when you're ready to customize. ## Reference Configuration [`reference.toml`](https://github.com/mezmo/aura/blob/main/examples/reference.toml) is the fully annotated config with every option documented. Use it as a starting point: ```bash theme={null} cp examples/reference.toml config.toml # Edit config.toml with your API key and settings cargo run --bin aura -- webserver ``` If encountering issues and more verbose debugging output is necessary, add the --verbose flag for the running binary ```bash theme={null} export OPENAI_API_KEY="sk-..." CONFIG_PATH=examples/minimal/openai.toml cargo run --bin aura -- webserver --verbose ``` ## Minimal Configs Bare-minimum configs to get running with a single LLM provider. No MCP tools — just add an API key. ```bash theme={null} export OPENAI_API_KEY="sk-..." CONFIG_PATH=examples/minimal/openai.toml cargo run --bin aura -- webserver ``` | File | Provider | API key needed? | | ------------------------------------------------------------------------------------------- | -------------- | --------------- | | [`openai.toml`](https://github.com/mezmo/aura/blob/main/examples/minimal/openai.toml) | OpenAI | Yes | | [`anthropic.toml`](https://github.com/mezmo/aura/blob/main/examples/minimal/anthropic.toml) | Anthropic | Yes | | [`bedrock.toml`](https://github.com/mezmo/aura/blob/main/examples/minimal/bedrock.toml) | AWS Bedrock | AWS credentials | | [`gemini.toml`](https://github.com/mezmo/aura/blob/main/examples/minimal/gemini.toml) | Google Gemini | Yes | | [`ollama.toml`](https://github.com/mezmo/aura/blob/main/examples/minimal/ollama.toml) | Ollama (local) | No | ## Complete Agent Configs Full agent compositions that combine an LLM provider with real MCP tool servers and tailored system prompts. Each file documents its prerequisites and required environment variables. | File | Description | | ---------------------------------------------------------------------------------------------------------------------------- | -------------------- | | [`devops-assistant.toml`](https://github.com/mezmo/aura/blob/main/examples/complete/devops-assistant.toml) | GitHub MCP | | [`incident-response-mezmo.toml`](https://github.com/mezmo/aura/blob/main/examples/complete/incident-response-mezmo.toml) | PagerDuty + Mezmo | | [`incident-response-datadog.toml`](https://github.com/mezmo/aura/blob/main/examples/complete/incident-response-datadog.toml) | PagerDuty + Datadog | | [`kubernetes-sre.toml`](https://github.com/mezmo/aura/blob/main/examples/complete/kubernetes-sre.toml) | K8s MCP + Prometheus | ### Serving Multiple Agents Point `CONFIG_PATH` at a directory to serve every `.toml` file as a selectable agent: ```bash theme={null} CONFIG_PATH=examples/complete/ cargo run --bin aura -- webserver ``` Clients discover agents via `GET /v1/models` and select one with the `model` field in chat requests. Each agent is identified by its `alias` (if set) or `name`. ## Advanced Quickstarts Self-contained Docker Compose setups for specific use cases. Each has its own guide with step-by-step instructions. | Quickstart | Description | | ------------------------------------------ | ----------------------------------------------------------------------- | | [Kubernetes SRE](/aura/quickstart-k8s-sre) | AI-powered SRE agent on KIND with Kubernetes and Prometheus MCP servers | # Human-in-the-Loop Approval Gates Source: https://docs.mezmo.com/aura/hitl Configure webhook and conversational approval gates for orchestration worker tool calls. Human-in-the-loop (HITL) approval gates let an agent ask for permission before running selected MCP tools. They compose in both single-agent and orchestration mode. Use them for operations that need a human decision before execution, such as production changes or destructive actions. Current behavior: * Gates compose in both single-agent mode and for orchestration workers. A single-agent run reports `scope.kind = "single"`; an orchestration worker reports `scope.kind = "worker"`. * Webhook routing works for unattended approvals. * Conversational routing works for attended approvals over an open SSE stream. The AURA CLI in HTTP mode is the first attended client. * Matching tool calls are blocked until the configured route approves them. * Human denials are returned to the model as normal tool feedback, so the agent can explain the denial without treating it as a transport failure. * Timeouts, cancellation, and webhook channel failures still fail closed as tool errors. * Conversational HITL requires `stream=true`; non-streaming requests are rejected because approval prompts are delivered over SSE. * Approval lifecycle events emit on streaming responses. Webhook emits `aura.approval_requested` and `aura.approval_completed`; conversational also emits `aura.approval_pending` while the tool call is parked. ## Configure a webhook gate Add a top-level `[hitl]` table and a required `[hitl.route]` table: ```toml theme={null} [hitl] require_approval = ["kubectl_*", "restart_*", "dangerous_*"] [hitl.route] mode = "webhook" url = "https://approvals.example.com/aura" timeout_secs = 300 ``` `require_approval` is a list of glob patterns matched against MCP tool names. When an agent calls a matching tool, Aura requests approval through the configured route before the MCP tool runs. A tool is gated if it matches **any** pattern in the list, so pattern order does not affect whether a tool is gated. To leave a tool ungated, do not list a pattern that matches it. When more than one pattern matches, the first in config order is reported as `origin.matched_pattern` in the webhook payload and SSE events; that is the only effect of ordering. The `request_approval` tool is never matched by these globs. It is excluded from the gate so the agent can ask for approval without triggering the gate itself. `timeout_secs` defaults to `300` for webhooks. If the webhook does not return a decision before the timeout, the tool does not run. ## Orchestration example ```toml theme={null} [agent] name = "SRE Orchestrator" system_prompt = "Route operational work to the right worker." turn_depth = 8 [agent.llm] provider = "openai" api_key = "{{ env.OPENAI_API_KEY }}" model = "gpt-5.2" context_window = 200_000 [mcp.servers.k8s] transport = "http_streamable" url = "http://k8s-mcp:8080/mcp" [hitl] require_approval = ["k8s_apply_*", "restart_*", "delete_*"] [hitl.route] mode = "webhook" url = "https://approvals.example.com/aura" timeout_secs = 300 [orchestration] enabled = true max_planning_cycles = 2 [orchestration.worker.operations] description = "Operational changes that may affect running services" preamble = "Use Kubernetes tools carefully. Do not retry denied actions." mcp_filter = ["k8s_*", "restart_*"] ``` The gate is added before the worker's MCP tools execute. A denied call returns a successful blocked tool result to the worker: ```text theme={null} Tool call blocked by human approval denial: maintenance window not open. Do not execute this action. ``` The worker sees that message and can explain the denial to the user. The MCP tool itself is not called. ## Webhook request Aura sends a JSON request to the configured webhook URL. The request uses a flat wire shape with `kind` tags for `scope` and `origin`: ```json theme={null} { "version": 1, "decision_id": "019edc27-e4d2-7950-abbf-e37a9060887d", "request_id": "req_d6df99fd5c8b4eb6af6e0de049e9c0d6", "scope": { "kind": "worker", "run_id": "019edc27-d7e1-73d2-ac33-1a1e21b3fffd", "task_id": 0, "worker": "operations", "session_id": "cs_264c5a09257c4089886cb00ae2ef03c4" }, "origin": { "kind": "config_gate", "matched_pattern": "restart_*" }, "items": [ { "tool_name": "restart_deployment", "arguments": { "namespace": "prod", "deployment": "api" } } ] } ``` Fields: | Field | Meaning | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `version` | Approval webhook protocol version. | | `decision_id` | Unique id for this approval decision. | | `request_id` | Aura request id for the chat completion. | | `scope` | Which agent surface is asking, independent of why. `kind = "single"` for a single-agent run or `kind = "worker"` for an orchestration worker. Both HITL origins carry a scope. | | `origin` | Why approval was requested. `kind = "config_gate"` (a configured glob matched the tool call, carries the matched glob) or `kind = "agent_requested"` (the agent called `request_approval`, carries the reason). | | `items` | Tool call payloads awaiting approval. One item per request. | ## Webhook response Approve the tool call: ```json theme={null} { "approved": true } ``` Deny the tool call, optionally with a reason: ```json theme={null} { "approved": false, "reason": "maintenance window not open" } ``` Response behavior: | Outcome | Tool execution | Worker-visible result | | --------------------- | ------------------ | -------------------------------------------------------------------- | | `approved: true` | Runs the tool. | The worker receives the MCP tool result. | | `approved: false` | Tool does not run. | The worker receives a blocked-action message with the denial reason. | | Timeout | Tool does not run. | The worker receives a tool error: approval timed out. | | Non-2xx response | Tool does not run. | The worker receives a tool error: approval channel error. | | Invalid JSON response | Tool does not run. | The worker receives a tool error: approval channel error. | ## SSE lifecycle events Approval routes emit lifecycle events on streaming responses. These events are emitted even when `AURA_CUSTOM_EVENTS=false` because clients may need to react to approval state. ```text theme={null} event: aura.approval_requested data: { ... } event: aura.approval_pending data: { ... } event: aura.approval_completed data: { ... } ``` `aura.approval_requested` includes `decision_id`, `tool_name`, `origin`, and `scope`. `aura.approval_pending` is emitted only by the conversational route and contains the attended prompt payload that an Aura-aware client renders before posting a decision. `aura.approval_completed` includes `decision_id`, terminal `outcome`, `duration_ms`, and `scope`. Outcome kinds are `approved`, `denied`, `timed_out`, `cancelled`, and `errored`; `errored` means the approval channel failed before a human decision was obtained. Aura also stamps this `decision_id` on the `execute_tool` span for every gated call, regardless of outcome. This lets you correlate an approval with the authorized action directly in the trace. See [Tracing & Span Layout](/aura/tracing-spans#span-attributes) for the full list of span attributes, including `decision_id`. ## Conversational route Use conversational routing when the approver is present on the chat stream. The server parks the worker tool call, sends `aura.approval_pending` over SSE, and waits for a decision on the approval ingress endpoint: ```toml theme={null} [hitl] require_approval = ["multiply", "divide", "dangerous_*"] [hitl.route] mode = "conversational" timeout_secs = 120 ``` The chat request must set `stream=true`. Aura rejects non-streaming requests for conversational HITL because there is no channel for the pending approval prompt. An attended client resolves a pending approval by POSTing the same decision shape as a webhook response: ```http theme={null} POST /v1/approvals/{decision_id} ``` ```json theme={null} { "approved": false, "reason": "maintenance window not open" } ``` The AURA CLI supports this flow in HTTP mode. It renders `aura.approval_pending`, prompts for approve/deny, and POSTs the decision back to the server. One-shot CLI mode fails loud instead of prompting because it has no interactive approval surface. ## Webhook manual smoke test Start a webhook service that accepts the request shape above and returns an approval response. Then run Aura with an orchestration config that uses: ```toml theme={null} [hitl] require_approval = ["mock_tool"] [hitl.route] mode = "webhook" url = "http://localhost:9988" timeout_secs = 300 ``` Use `mock_tool` so the glob matches a tool the worker actually calls. Put the webhook on a different port than the mock MCP server (9999) to avoid a collision. Ask an orchestration worker to use the gated tool. An approval should let the tool run. A denial with a custom reason should produce a successful blocked tool result containing that reason. ## Conversational manual smoke test Run Aura with an orchestration config that uses `mode = "conversational"`, a route timeout shorter than `[orchestration.timeouts].per_call_timeout_secs`, and at least one gated worker tool. Connect with the AURA CLI in HTTP mode and send a query that forces the worker to call the gated tool. Expected behavior: * The CLI renders an approval prompt from `aura.approval_pending`. * Approving the prompt POSTs to `/v1/approvals/{decision_id}` and lets the tool run. * Denying the prompt POSTs the denial and returns blocked-action feedback to the worker. ## Current limitations * The webhook route is synchronous. Aura waits for the webhook response during the tool call. * With the default in-memory session store, conversational approvals are single-instance, so only the server process that emitted `aura.approval_pending` can resolve them. * You can resume a parked approval on a different pod by configuring the optional Redis or Valkey session store. A `POST /v1/approvals/{id}` request that lands on any instance then resolves an approval parked on another. See [Session Store](/aura/configuration-reference#session-store-durable-and-multi-pod-deployments). * Webhook egress has no built-in authentication layer. Put auth, signing, or network controls in front of the webhook service. # AURA: open source agent harness for production SRE work Source: https://docs.mezmo.com/aura/index AURA is an open source SRE agent for investigating production incidents, with scoped tool access, human approval gates, and end-to-end OpenTelemetry tracing. Written in Rust and licensed Apache 2.0, AURA takes an LLM plus the tools you already run and turns them into an agent that can work a live incident: reading cluster state, querying logs and metrics, checking your runbooks, and reporting what it found.

You declare the agent in TOML. Workers are separate roles, each with its own prompt and tool list, so the worker that inspects a cluster is not the worker that changes it. Tools connect over MCP, so anything with an MCP server is reachable without writing an adapter. ## What it does Model output alone does not make an agent you can point at production. The harness decides which tools each worker can reach, what pauses before a write, what happens when a tool call fails, what state survives across turns, and what record you have when the run is over. You own the config and deployment, and the telemetry goes where you point it. Nothing has to route through a vendor for the agent to work. ## Use cases Every worker gets an explicit tool list, so read access and write access live in different roles across the agent. Approval gates name the mutating tools that need a human decision. Reads run straight through; writes wait for a yes. MCP servers over HTTP, SSE, and stdio: Kubernetes, Prometheus, Grafana, GitHub, Notion, AWS, Mezmo, and anything else. Every model call and tool call emits an OpenTelemetry span in OpenInference conventions, readable in any backend. RAG over in-memory, Qdrant, or Bedrock knowledge bases, plus skills that load task instructions only when needed. OpenAI, Anthropic, Gemini, Bedrock, OpenRouter, or a local Ollama instance. Same config, different provider block. ## Start here Install the binary, run `aura init`, connect an MCP server, and start asking questions. Bring up AURA and a demo environment with Compose, no local installs required. Run AURA as a long-lived in-cluster service with the Helm chart and a read-only Kubernetes MCP server. ## Examples Each of these is a recorded run against a live environment. Where a video has a written walkthrough, the link goes straight to it. ### Install and connect Install on a desktop, run `aura init` against an Anthropic model, wire in a Grafana MCP server, and investigate. Watch the walkthrough → Helm install AURA and the Kubernetes MCP server into a cluster, read-only through both RBAC and a startup flag. Watch the walkthrough → Add Mezmo, GitHub over HTTP streamable, and Grafana over stdio, then narrow which worker holds which tools. Point AURA at OpenAI, Anthropic, Gemini, Bedrock, OpenRouter, or a local Ollama with one provider block. ### Investigate an incident A one-sentence health check turns up a pod in CrashLoopBackOff with 788 restarts that nobody asked about. A coworker's message with no service name or time range. The log worker builds a window and traces checkout to payment. AURA connects through the AWS MCP proxy with existing CLI credentials, finds the log group, and verifies the timeout. Two config sections point AURA at Dash0's hosted MCP server, read-only, with credentials in environment variables. The agent doesn't know the stack runs on Compose. It opens with generic advice, then works down to the mount and owner. The tracing worker follows a slow request from the gateway through three services and back, no manual span-hunting. ### Give it your runbooks A read-only Notion key in `.env` and a runbook worker, so on-call guidance comes from your pages instead of the model. A fine-grained token scoped to one repo, plus sources of truth and routing rules for a runbook engineer worker. ### Keep it contained Deny a `pods_delete` call and confirm nothing changed, then approve the same call and watch it run. Bedrock as the provider, a managed knowledge base doing retrieval in-account, and every call landing in CloudTrail. ### Let it run on its own A cron job starts a one-shot run with inspect access and exactly one mutation tool, bounded by prompt and by RBAC. A sync-fail hook sends a symptom over A2A. AURA finds the version mismatch and edits the one config map it may touch. A GitHub worker with a reduced tool list and an approval gate on issue creation, carrying a root cause into a PR. A nightly run compares live cluster state against declared config and opens an issue for each drift it finds. ## Join the project Ask questions, share what you are running, and talk through an approach before you open a PR. Apache 2.0, written in Rust. Clone it, read the config examples, and open an issue when something does not fit. Stars are how other SREs find the project. If AURA saved you an afternoon, a star is the cheapest way to say so. Tell us the tool, the failure you keep hitting, and what the agent should do. We build these into walkthroughs. # Installation Source: https://docs.mezmo.com/aura/installation Install AURA with the install script, a direct binary download, Homebrew, or the APT and RPM package repositories. AURA supports Linux and macOS on `amd64` and `arm64`. ## Install AURA **Run the install script** ```bash theme={null} curl -fsSL https://raw.githubusercontent.com/mezmo/aura/main/scripts/install.sh | bash ``` By default the [install script](https://raw.githubusercontent.com/mezmo/aura/main/scripts/install.sh) attempts to install from the AURA package repository through a native package manager, then falls back to Homebrew and finally a direct binary download. The script is configured entirely through the environment variables below. After installing, confirm the version: ```bash theme={null} aura --version ``` **Install script environment variables** | Variable | Description | Default | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | | `AURA_VERSION` | Version to install. A leading `v` is optional and is stripped. For `direct`, `latest` follows the GitHub releases/latest redirect to the newest release. For `deb` and `rpm`, a value pins the package version and `latest` lets the package manager choose. Homebrew cannot pin a version: under `auto`, a pinned version skips Homebrew; with `AURA_INSTALL_METHOD=homebrew`, it is a fatal error. | `latest` | | `AURA_INSTALL_METHOD` | How to install: `auto`, `homebrew`, `direct`, `deb`, or `rpm`, lowercase (for example `deb`, not `DEB`). Any other value is a fatal error. | `auto` | | `AURA_INSTALL_PATH` | Install directory for the `direct` method, created if it does not exist. Under `auto`, setting it skips Homebrew, Debian, and RPM in favor of a direct install. Combining it with an explicit `homebrew`, `deb`, or `rpm` method is a fatal error. | `~/.local/bin` | | `AURA_COMPONENT` | Component to install: `all`, `server`, or `cli`. Any other value is a fatal error. | `all` | | `AURA_REQUIRE_CHECKSUM` | `direct` method only. When `1` (the default), a missing checksums file or a missing entry for an asset is a fatal error; set it to `0` to downgrade those to a warning. A checksum mismatch, or the absence of any checksum tool (`sha256sum`, `shasum`, or `openssl`) when there is a checksum to verify, is always fatal regardless of this setting. Any value other than `0` or `1` is a fatal error. | `1` | | `AURA_CHECKSUMS` | `direct` method only. Path to a local `checksums.txt` file. When set, the script verifies against this file instead of downloading `checksums.txt` from the release. | (unset) | **Upgrade** Re-run the install script. It overwrites the existing install in place. Install the `mezmo/tap/aura` formula: ```bash theme={null} brew install mezmo/tap/aura ``` **Add the AURA repository** Write the repository file. On openSUSE and SLES, use `/etc/zypp/repos.d/mezmo-aura.repo` instead: ```bash theme={null} sudo tee /etc/yum.repos.d/mezmo-aura.repo > /dev/null <<'EOF' [mezmo-aura] name=mezmo-aura baseurl=https://dl.cloudsmith.io/public/mezmo/aura/rpm/any-distro/any-version/$basearch repo_gpgcheck=1 gpgcheck=1 enabled=1 autorefresh=1 gpgkey=https://dl.cloudsmith.io/public/mezmo/aura/gpg.05C8AD333177EB1F.key sslverify=1 metadata_expire=300 type=rpm-md EOF ``` Then install the `aura` package: ```bash theme={null} sudo dnf install aura ``` Substitute `microdnf`, `yum`, or `zypper` if that is what the system uses. To pin a version, append it to the package name (`dnf install aura-0.1.18`). `zypper` uses `aura=0.1.18` instead, and needs `sudo zypper --gpg-auto-import-keys refresh mezmo-aura` once to trust the signing key. To list the versions the repository carries, run `dnf --showduplicates list aura` (`zypper search -s aura`), or browse the [Cloudsmith package list](https://cloudsmith.io/~mezmo/repos/aura/packages/). After installing, confirm the version: ```bash theme={null} aura --version # or: rpm -q aura ``` Individual `.rpm` files are also attached to each [GitHub release](https://github.com/mezmo/aura/releases), for mirroring into an internal repository. **Add the AURA repository** ```bash theme={null} sudo apt-get install -y ca-certificates curl gnupg curl -fsSL https://dl.cloudsmith.io/public/mezmo/aura/gpg.05C8AD333177EB1F.key \ | sudo gpg --dearmor -o /usr/share/keyrings/mezmo-aura-archive-keyring.gpg # This one source line is correct on every Debian and Ubuntu release echo 'deb [signed-by=/usr/share/keyrings/mezmo-aura-archive-keyring.gpg] https://dl.cloudsmith.io/public/mezmo/aura/deb/debian bookworm main' \ | sudo tee /etc/apt/sources.list.d/mezmo-aura.list sudo apt-get update sudo apt-get install -y aura ``` To pin a version, qualify the package name: `sudo apt-get install aura=0.1.18`. To list the versions the repository carries, run `apt-cache madison aura`, or browse the [Cloudsmith package list](https://cloudsmith.io/~mezmo/repos/aura/packages/). After installing, confirm the version: ```bash theme={null} aura --version # or: dpkg -s aura ``` Individual `.deb` files are also attached to each [GitHub release](https://github.com/mezmo/aura/releases), for mirroring into an internal repository. The links below always resolve to the newest release. Each asset is a bare executable with no file extension. | Platform | `aura` | | --------------------- | --------------------------------------------------------------------------------------------- | | macOS (Apple Silicon) | [aura-darwin-arm64](https://github.com/mezmo/aura/releases/latest/download/aura-darwin-arm64) | | macOS (Intel) | [aura-darwin-amd64](https://github.com/mezmo/aura/releases/latest/download/aura-darwin-amd64) | | Linux (x86-64) | [aura-linux-amd64](https://github.com/mezmo/aura/releases/latest/download/aura-linux-amd64) | | Linux (ARM64) | [aura-linux-arm64](https://github.com/mezmo/aura/releases/latest/download/aura-linux-arm64) | For older versions and `checksums.txt`, browse the [GitHub Releases](https://github.com/mezmo/aura/releases) page. Each download arrives non-executable and keeps its platform suffix, so make it executable, rename it to `aura`, and move it somewhere on your `PATH`. Mezmo code-signs and Apple notarizes the macOS release binaries, for Intel (`amd64`) and Apple Silicon (`arm64`). They run without the Gatekeeper "unidentified developer" prompt, even when downloaded through a web browser from the [GitHub Releases](https://github.com/mezmo/aura/releases) page. ## Next Steps * Configure and use the CLI at the [CLI Reference](/aura/cli-reference). * Run the server at the [Web Server Reference](/aura/web-server-reference). * Try the containerized stack at the [Quickstart](/aura/quickstart). # Ollama Guide Source: https://docs.mezmo.com/aura/ollama-guide Ollama configuration, fallback tool-call parsing, and practical guidance for local models. AURA supports running local models through [Ollama](https://ollama.ai), including fallback parsing for tool-call formats that are emitted as plain text instead of native tool-call structures. ## Basic Configuration ```toml theme={null} [agent] name = "Local Assistant" system_prompt = "You are a helpful assistant." [agent.llm] provider = "ollama" model = "qwen3:30b-a3b" # base_url = "http://localhost:11434" # optional; this default is used automatically fallback_tool_parsing = true [agent.llm.additional_params] num_ctx = 32000 think = true ``` `base_url` defaults to `http://localhost:11434` when omitted. Use `http://host.docker.internal:11434` when AURA runs inside a container and Ollama runs on the host. All Ollama-specific parameters (`num_ctx`, `num_predict`, `think`, `seed`, `top_k`, `top_p`, etc.) go under `[agent.llm.additional_params]`. See [Ollama model parameters](https://github.com/ollama/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values) for the full list. ## Fallback Tool Parsing When `fallback_tool_parsing = true`, AURA tries to detect and execute tool calls from text output patterns commonly produced by local model families. Known handled styles include: * Pythonic-like calls (for example Llama-style patterns) * XML-ish function wrappers (common in some Qwen outputs) * JSON objects containing name/parameters payloads This improves tool reliability with local models that do not consistently emit structured function-calling payloads. **Known issue (orchestration).** `fallback_tool_parsing` is currently honored only in single-agent mode. In orchestration mode the coordinator and workers are built with fallback parsing **disabled** — a bug, not intended behavior, tracked in [#193](https://github.com/mezmo/aura/issues/193). Until it's fixed, a local model that emits tool calls as text instead of native `tool_calls` can stall in orchestration (the coordinator never registers a routing decision). Workaround: run such models in single-agent mode (`[orchestration].enabled = false`), or use a model with reliable native tool-calling. ## Practical Guidance * Prefer instruction-tuned variants (`*-instruct`) when you need reliable tool execution. * Keep prompts explicit about expected tool-call output format. * Validate behavior with your exact model build and quantization. ## "Thinking model" Caveat Thinking model variants have known malformed XML tool-call issues in some builds. AURA's fallback parser handles many of these cases, but reliability still depends on model artifact quality and prompt format constraints. # AURA Quickstart Source: https://docs.mezmo.com/aura/quickstart Install the aura binary locally, generate a config, connect an MCP server, and put your agent to work. **Prerequisites:** macOS or Linux, and an LLM API key (OpenAI, Anthropic, Gemini, Bedrock, or OpenRouter), or a local [Ollama](https://ollama.com) instance. ## 1. Install AURA This pipes a remote script to `bash`; to review the script or its checksum behavior first, see the [Installation](/aura/installation#install-aura) page. ```bash theme={null} curl -fsSL https://raw.githubusercontent.com/mezmo/aura/main/scripts/install.sh | bash ``` This gets you the `aura` CLI, which is all this quickstart needs. The script auto-selects the best install method for your platform: a native package, **Homebrew**, or a direct binary download. If `aura` isn't found after a direct download, add the install directory to your `PATH`: ```bash theme={null} export PATH="$HOME/.local/bin:$PATH" ``` The script takes no arguments and is configured entirely by environment variables. See [Installation](/aura/installation) for all install methods and environment variables. ## 2. Generate a config ```bash theme={null} aura init ``` The wizard asks a few questions, then writes a ready-to-run `config.toml` in the current directory: * **Provider.** It first checks your environment for a conventional API key (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) and suggests the matching provider as the default. * **API key.** If that variable is already set, it asks whether to use it. Otherwise it prompts for the key with masked input. * **Model.** It queries the provider's live model list and offers a short, curated shortlist. Pick by number, accept the default, or type any model id. * **Output.** It writes `config.toml`, plus a `.env` only when you entered a key that wasn't already in your environment. Your key never lands in `config.toml`. The generated file references it as `api_key = "{{ env.OPENAI_API_KEY }}"` instead. If a `.env` was written, add it to your `.gitignore`. ## 3. Start chatting ```bash theme={null} aura ``` AURA launches as a local chat assistant, loading `./config.toml` and running the agents on your machine. The generated config is an **SRE orchestrator**: a coordinator that routes each request to one of three specialist workers (`incident-responder`, `metrics-analyst`, and `log-analyst`), running them in parallel when the tasks are independent. You can add or remove workers, rewrite their prompts, and change any other option in `config.toml` later, using the [Configuration Reference](/aura/configuration-reference). ## 4. Connect your tools Right now AURA's workers can reason about problems and analyze data you paste in, but they can't query anything yet. At the AURA chat prompt, type: ``` /mcp add ``` This launches a guided wizard that adds an [MCP](https://modelcontextprotocol.io) server to your config, giving the workers tools they can call. Pick one from the built-in catalog: | Server | What your agents get | You'll need | | ---------- | ----------------------------------------------------- | ----------------------------------------------------------------- | | Mezmo | Log analysis, exports, and pipeline monitoring | A Mezmo service API key (**Settings > API Keys**) | | PagerDuty | Incidents, on-call schedules, and escalation policies | A PagerDuty API token (**User Settings > API Access**) | | Datadog | Metrics, monitors, dashboards, and APM traces | A Datadog API key and application key (**Organization Settings**) | | Kubernetes | Read-only cluster inspection through your kubeconfig | Node.js (for `npx`) and a working kubeconfig context | Or choose **custom** to configure any other MCP server over `http_streamable`, `sse`, or `stdio`. The wizard then: 1. Collects credentials with masked input, or reuses an environment variable you already have set. 2. Connects to the server and lists the tools it discovered, all in memory, before anything touches disk. A credential that doesn't work never reaches your config. 3. Previews the exact `[mcp.servers.]` block and asks whether to write it. Pressing Enter accepts. If the connection check failed, the wizard says so and pressing Enter declines instead. 4. Writes the config, then asks which workers should see the new tools and records an `mcp_filter` for each. You might give the Kubernetes tools to `incident-responder` alone and leave `log-analyst` without them. Secrets go to a `.env` beside your config; the TOML gets `{{ env.VAR }}` placeholders. AURA resolves them only to authenticate to the MCP server, so your credentials never enter a prompt or a tool definition and the model never sees them. Nothing is written until you confirm. ## 5. Restart and put it to work MCP servers are loaded at startup, so restart AURA to activate the one you just added. Run `/quit`, then: ```bash theme={null} aura ``` Try the starter prompt the wizard printed for your server. For PagerDuty: > Who is on call right now, and are there any open incidents? Run `/mcp` at any time to list the servers the active agent has configured. ## What's next * **[Web Server Reference](/aura/web-server-reference)**: run AURA as a daemon that serves these same agents over an HTTP API, so alerts and other automated events can trigger agent runs. * **[Example Configs](/aura/example-configs)**: minimal per-provider configs and complete agent compositions to start from. * **[CLI Reference](/aura/cli-reference)**: every slash command, flag, and the full `/mcp add` walkthrough. * **[Configuration Reference](/aura/configuration-reference)**: every option available in `config.toml`. * **[Full-Stack Docker Quickstart](/aura/quickstart-docker)**: run AURA with a browser chat UI and a Phoenix trace viewer via `docker compose`. * **[Kubernetes SRE Quickstart](/aura/quickstart-k8s-sre)**: an AI-powered SRE agent on a KIND cluster with real Kubernetes and Prometheus MCP servers. ## Troubleshooting ### `aura: command not found` The install directory isn't on your `PATH`. Run `export PATH="$HOME/.local/bin:$PATH"` and add it to your shell profile. ### `aura init` can't list models `aura init` fetches the provider's model list to confirm your key works and to build the shortlist it offers you. If that request fails, because you're offline or the key was rejected, it warns and continues, allowing you to type the model id yourself. Pass `--offline` to skip the request entirely. AWS Bedrock has no model-list endpoint, so it's always skipped. ### Workers say they have no tools No MCP servers are configured yet. Run `/mcp add`, then restart `aura`. ### `/mcp add` isn't available The wizard edits the `config.toml` on your machine, so it's only available when AURA runs agents from that file. Connecting to an [AURA web server](/aura/web-server-reference) with `--api-url` disables the `/mcp add` command, since the config lives on the server instead. ### Newly added MCP server isn't showing up There's no hot reload. Restart `aura` after `/mcp add` finishes. # Full-Stack Docker Quickstart Source: https://docs.mezmo.com/aura/quickstart-docker Run the whole AURA stack with docker compose — orchestrator mode, a chat UI, and a trace viewer. Bring up the whole stack with one command — AURA in **orchestrator mode**, a chat UI, and a trace viewer — all from a clone of the repo. Just want to talk to an agent? The [AURA Quickstart](/aura/quickstart) installs a binary and has you chatting in three commands, with no Docker and no clone. Use this page when you want the browser chat UI and the Phoenix trace viewer running alongside AURA. **Prerequisites:** [Docker](https://docs.docker.com/get-docker/), a clone of the [AURA repository](https://github.com/mezmo/aura), and an LLM API key (OpenAI, Anthropic, or a local [Ollama](https://ollama.com) instance). ## 1. Configure your LLM provider ```bash theme={null} cp .env.example .env ``` Edit `.env` and set your provider, model, and API key: ```bash theme={null} LLM_PROVIDER=openai # or: anthropic, ollama LLM_MODEL=gpt-5.2 # or: claude-sonnet-4-20250514, llama3.1 LLM_API_KEY=sk-... # your API key (use "unused" for Ollama/llama-server) ``` ## 2. Start everything ```bash theme={null} docker compose up -d ``` AURA boots in orchestrator mode. A coordinator routes each request, answering simple ones directly and decomposing operational tasks across the `incident-responder`, `metrics-analyst`, and `log-analyst` workers defined in `quickstart.toml`. ## 3. Chat with your agent The [AURA CLI](/aura/cli-reference) ships in the same Docker image and connects to the in-container server automatically. Exec into the running container: ```bash theme={null} docker exec -it aura ./aura --api-url http://localhost:8080 ``` It renders the coordinator's plan and worker activity as the response streams. Check startup progress with `docker compose logs -f aura`. ### Or use a browser | Service | URL | Description | | --------- | ---------------------------------------------- | --------------------- | | LibreChat | [http://localhost:3080](http://localhost:3080) | Chat with your agent | | Phoenix | [http://localhost:6006](http://localhost:6006) | Inspect LLM traces | | AURA API | [http://localhost:8080](http://localhost:8080) | OpenAI-compatible API | **LibreChat first-time setup:** Create your user account on the signup page. The agent model is pre-configured as "Aura Orchestrator". ### Build the CLI from source Prefer to build locally instead of using the bundled binary? Connect to the quickstart server: ```bash theme={null} cargo build -p aura-cli --release ./target/release/aura ``` The CLI defaults to **standalone mode** — it runs agents in-process from a TOML config, no server needed: ```bash theme={null} cargo build -p aura-cli --release ./target/release/aura --config quickstart.toml ``` See the [CLI Reference](/aura/cli-reference) for the full feature set. ## Customize Your Agent Edit `quickstart.toml` to change coordinator routing and worker behavior, add tools, or enable vector search. Edit `.env` to switch LLM providers. Then apply changes: ```bash theme={null} docker compose up -d # picks up .env changes and recreates if needed ``` `docker compose restart aura` is fine for `quickstart.toml`-only changes, but `.env` changes require `docker compose up -d` to take effect. ### Switch LLM provider Update `LLM_PROVIDER`, `LLM_MODEL`, and `LLM_API_KEY` in `.env`, then `docker compose up -d`. **Anthropic:** ```bash theme={null} LLM_PROVIDER=anthropic LLM_MODEL=claude-sonnet-4-20250514 LLM_API_KEY=sk-ant-... ``` **Ollama** (local, no API key): ```bash theme={null} LLM_PROVIDER=ollama LLM_MODEL=llama3.1 LLM_API_KEY=unused LLM_BASE_URL=http://host.docker.internal:11434 ``` Also uncomment the `base_url` line in `quickstart.toml`. **Ollama + orchestration (known issue):** the quickstart defaults to orchestration mode, where `fallback_tool_parsing` is currently *not* applied to the coordinator or workers — a bug tracked in [#193](https://github.com/mezmo/aura/issues/193). Until it's fixed, a local model that relies on fallback parsing (tool calls emitted as text rather than native tool calls) can stall in orchestration. Workaround: run the quickstart in single-agent mode — set `[orchestration].enabled = false` and uncomment `fallback_tool_parsing = true` in `quickstart.toml`. Models with reliable native tool-calling work as-is. See the [Ollama guide](/aura/ollama-guide) for details. **[llama-server](https://github.com/ggml-org/llama.cpp/tree/master/tools/server)** (llama.cpp, local, no API key): llama-server exposes an OpenAI-compatible API, so use the `openai` provider with a `base_url` override: ```bash theme={null} LLM_PROVIDER=openai LLM_MODEL=local-model LLM_API_KEY=unused LLM_BASE_URL=http://host.docker.internal:8080/v1 ``` Also uncomment the `base_url` line in `quickstart.toml`. The `LLM_MODEL` value can be anything — llama-server ignores it and uses whatever model it was started with. ### Add MCP tool servers `quickstart.toml` already ships an active `[mcp]` block with an empty `[mcp.servers]` table. It also includes commented example servers for alerting, metrics, and logs. Uncomment one, or add your own under `[mcp.servers]`. The metrics example looks like this: ```toml theme={null} [mcp.servers.my-metrics] transport = "http_streamable" url = "http://host.docker.internal:9090/mcp" description = "Prometheus metrics server" ``` Use `host.docker.internal` to reach services running on your host machine. Then scope the tools per worker: set `mcp_filter` on each worker that should use them (see [Customize orchestration](#customize-orchestration)). A worker with no `mcp_filter` receives **all** MCP tools, so set it explicitly on every tool-using worker and give tool-free workers (like the default `log-analyst`) an explicit empty filter (`mcp_filter = []`). ### Add vector search Add a `[[vector_stores]]` section to `quickstart.toml`. Options: * **Qdrant** (self-hosted): add a Qdrant instance to the compose file or point at an external one. Embeddings can be generated via OpenAI or AWS Bedrock. * **AWS Bedrock Knowledge Base** (managed): set `type = "bedrock_kb"` with a `knowledge_base_id` and `region`. No embedding model needed — the KB manages embeddings internally. Registering a store under `[[vector_stores]]` only defines it — no agent can query it until you attach it. Add the store's `name` to a worker's `vector_stores` list (e.g. `vector_stores = ["docs"]`), or to `[orchestration].coordinator_vector_stores` to give the coordinator access. See [`examples/reference.toml`](https://github.com/mezmo/aura/blob/main/examples/reference.toml) for both. ### Serve multiple agents Create a directory with one TOML file per agent: ``` configs/ ├── research-assistant.toml ├── devops-agent.toml └── code-reviewer.toml ``` Then update `docker-compose.yml` to mount and serve the directory: ```yaml theme={null} environment: CONFIG_PATH: "/app/config/configs" volumes: - ./configs:/app/config/configs:ro ``` Restart with `docker compose up -d`. Clients that support model selection (LibreChat, OpenWebUI, etc.) will show each agent in their model picker via `GET /v1/models`. The example configs in `examples/` reference provider-specific env vars (e.g. `OPENAI_API_KEY`) rather than the quickstart's `LLM_API_KEY`. Add the appropriate keys to your `.env` — they're automatically loaded into the container via `env_file`. See `.env.example` for the full list. #### Hide an agent from discovery Set `hidden = true` in an agent's `[agent]` block to keep it out of discovery listings. AURA omits a hidden agent from the `GET /v1/models` response and from the CLI's `/model` list, so it won't appear in client model pickers. The agent stays fully usable. Any caller that already knows its name or alias can still select it by sending that value as the `model` field. This helps when an agent isn't ready yet, or when you want only known callers to invoke it during development/testing/production. ```toml theme={null} [agent] name = "hidden agent" hidden = true system_prompt = "You are a hidden agent that does not show up in listings, but still invokable by known callers." ``` The `hidden` field defaults to `false`. It accepts either a TOML boolean (`true` or `false`) or the quoted strings `"true"` and `"false"`. The quoted form is convenient when a templating tool such as Helm renders the value as a string. If you load only a single hidden agent, both `GET /v1/models` and the CLI `/model` list come back empty even though the agent is still the active, invocable default. ### Customize orchestration `quickstart.toml` ships with orchestration enabled: a coordinator and three specialized workers (`incident-responder`, `metrics-analyst`, and `log-analyst`) that reason with the LLM alone until you connect an MCP server. The coordinator's routing is controlled by the `[orchestration]` block: ```toml theme={null} [orchestration] enabled = true max_planning_cycles = 2 allow_direct_answers = true # simple queries answered without workers allow_clarification = true # vague requests prompt follow-up questions tools_in_planning = "summary" # coordinator sees tool names during planning ``` Each `[orchestration.worker.]` block defines a worker. Give a worker tools by configuring an MCP server (see [Add MCP tool servers](#add-mcp-tool-servers) above) and listing matching tool globs in its `mcp_filter`, or point it at a vector store via `vector_stores`: ```toml theme={null} [orchestration.worker.operations] description = "Operational analysis and diagnostics" preamble = """ You are an operations specialist completing one assigned task. Use your tools for every operation — do not guess results. """ mcp_filter = ["ops_*"] # glob patterns selecting which MCP tools this worker can use turn_depth = 5 [orchestration.worker.knowledge] description = "Documentation and knowledge retrieval" preamble = """ You are a knowledge specialist completing one assigned task. Search available documentation to answer the question. """ mcp_filter = [] # no MCP tools: vector search only vector_stores = ["docs"] turn_depth = 5 ``` A worker whose `mcp_filter` is *omitted* is granted **every** MCP tool. Set an explicit `mcp_filter` on each worker once an MCP server is configured; `mcp_filter = []` keeps a worker tool-free. Each worker inherits the agent's LLM by default. To run a worker on a different model, add a complete `[orchestration.worker..llm]` block — see the [orchestration config reference](/aura/configuration-reference#orchestration) for all fields. Restart with `docker compose restart aura` and try asking a multi-step question. Watch the coordinator plan and dispatch in the CLI's event panel, or in Phoenix at [http://localhost:6006](http://localhost:6006). To run a single agent instead of orchestration, set `[orchestration].enabled = false` and configure a single `[agent]` — see [`examples/reference.toml`](https://github.com/mezmo/aura/blob/main/examples/reference.toml). ### Full configuration reference See [`examples/reference.toml`](https://github.com/mezmo/aura/blob/main/examples/reference.toml) for all available options. ## What's Next Once the stack is running, try these more advanced setups: * **[Kubernetes SRE Quickstart](/aura/quickstart-k8s-sre)** — Deploy an AI-powered SRE agent on a KIND cluster with real Kubernetes and Prometheus MCP servers. * **[Example Configs](/aura/example-configs)** — Minimal per-provider configs and complete agent compositions to use as starting points. ## Architecture ```mermaid theme={null} graph TD Browser -->|":3080"| LibreChat Terminal -->|"docker exec"| CLI["aura"] LibreChat <-->|"/v1/chat/completions"| AURA CLI <-->|"/v1/chat/completions"| AURA AURA -->|"OTel gRPC :4317"| Phoenix AURA <-->|"API calls"| LLM["LLM Provider
(OpenAI, etc.)"] AURA <-->|"MCP"| MCP["MCP Tool Servers"] LibreChat --- MongoDB["MongoDB
(LibreChat storage)"] Browser -->|":6006"| Phoenix subgraph compose["docker compose"] LibreChat[":3080 LibreChat"] AURA[":8080 AURA (orchestrator)"] Phoenix[":6006 Phoenix"] CLI MongoDB end ``` * **aura** runs inside the AURA container (`docker exec`) and talks to the same OpenAI-compatible `/v1/chat/completions` endpoint, rendering coordinator and worker events as they stream. * **LibreChat** sends chat requests to AURA's OpenAI-compatible `/v1/chat/completions` endpoint. MongoDB is used by LibreChat internally for user accounts and conversation history — AURA does not use it. * **AURA** runs the coordinator that routes each request, dispatches workers, executes MCP tools, calls the configured LLM provider, and streams responses back. * **Phoenix** receives OpenTelemetry traces from AURA so you can inspect every coordinator and worker step. ## Troubleshooting **LibreChat shows "no models available"** AURA may still be starting. Wait for the health check to pass (`docker compose logs aura --tail 5`) and refresh. **"connection refused" in AURA logs** If referencing services on your host, use `host.docker.internal` instead of `localhost` in `quickstart.toml`. **Reset everything** ```bash theme={null} docker compose down -v ``` # Kubernetes SRE Orchestration Quickstart Source: https://docs.mezmo.com/aura/quickstart-k8s-sre Deploy an AI-powered Kubernetes SRE agent using orchestration mode to coordinate cluster inspection and metrics analysis specialists. Deploy an AI-powered Kubernetes SRE agent that uses **orchestration mode** to coordinate specialized workers -- one for cluster inspection, one for metrics analysis -- each with access to only the tools they need. ## What You'll Build ``` User Query | v +------------------------------------------+ | AURA (Coordinator) | | Routes requests to the right specialist | | | | +-------------------+ +----------------+ | | | cluster_inspector | | metrics_analyst| | | | K8s tools only | | Prom tools only| | | +--------+----------+ +-------+--------+ | +-----------|--------------------|----------+ | | v v +---------------+ +-----------------+ | K8s MCP | | Prometheus MCP | | Server | | Server | +-------+-------+ +--------+--------+ | | v v +---------------+ +-----------------+ | Kubernetes | | Prometheus | | API | | (OTel Demo) | +---------------+ +-----------------+ ``` The **coordinator** receives user queries and dispatches them to: * `cluster_inspector` -- filtered to Kubernetes MCP tools (pods, deployments, logs, events) * `metrics_analyst` -- filtered to Prometheus MCP tools (PromQL queries, alerts, targets) ## Prerequisites * [Docker](https://docs.docker.com/get-docker/) * [kubectl](https://kubernetes.io/docs/tasks/tools/) * [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) * [Helm](https://helm.sh/docs/intro/install/) 3.12+ * An OpenAI API key (or another [supported LLM provider](https://github.com/mezmo/aura/blob/main/examples/reference.toml)) ## Setup All commands assume you're in the repo root. ### 1. Create a KIND cluster ```bash theme={null} kind create cluster --name aura-sre ``` ### 2. Deploy the OpenTelemetry Demo The [OpenTelemetry Demo](https://opentelemetry.io/docs/demo/) deploys a microservices application with Prometheus, Grafana, and Jaeger -- giving your SRE agent real workloads and metrics to inspect. ```bash theme={null} helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm install otel-demo open-telemetry/opentelemetry-demo ``` Wait for pods to come up (this takes a few minutes on first pull): ```bash theme={null} kubectl get pods -w ``` Not every pod needs to be `Running` before proceeding. As long as the Prometheus pod is ready, you can continue. Verify Prometheus is running and note the service name: ```bash theme={null} kubectl get svc | grep prometheus ``` You should see a service like `prometheus` on port `9090`. Note the name -- if it differs, use it in the `--set` flag in step 3 below. ### 3. Deploy the MCP servers Both MCP servers have community Helm charts. Install them with: ```bash theme={null} # Kubernetes MCP Server — read-only cluster access # Binds the built-in "view" ClusterRole for read access to cluster resources. helm install kubernetes-mcp-server \ oci://ghcr.io/containers/charts/kubernetes-mcp-server \ --set ingress.enabled=false \ --set config.read_only=true \ --set 'rbac.extraClusterRoleBindings[0].name=view' \ --set 'rbac.extraClusterRoleBindings[0].roleRef.name=view' \ --set 'rbac.extraClusterRoleBindings[0].roleRef.external=true' # Prometheus MCP Server — connected to the OTel Demo's Prometheus # Override probes to use TCP (the MCP server has no GET health endpoint). helm install prometheus-mcp-server \ oci://ghcr.io/pab1it0/charts/prometheus-mcp-server \ --set prometheus.url="http://prometheus:9090" \ --set livenessProbe.httpGet=null \ --set 'livenessProbe.tcpSocket.port=http' \ --set readinessProbe.httpGet=null \ --set 'readinessProbe.tcpSocket.port=http' ``` **Different Prometheus service name?** Run `kubectl get svc | grep prometheus` and replace the URL above with the correct service name from step 2. Verify both MCP servers are healthy before proceeding: ```bash theme={null} # Check the Kubernetes MCP server can reach the API kubectl logs -l app.kubernetes.io/name=kubernetes-mcp-server --tail=5 # Check the Prometheus MCP server connected to Prometheus kubectl logs -l app.kubernetes.io/name=prometheus-mcp-server --tail=5 ``` Wait for them to start: ```bash theme={null} kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=kubernetes-mcp-server --timeout=120s kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=prometheus-mcp-server --timeout=120s ``` ### 4. Deploy AURA ```bash theme={null} export OPENAI_API_KEY="sk-..." helm install aura ./deployment/helm/aura \ -f examples/quickstart-k8s-sre/aura-values.yaml \ --set secrets.openaiApiKey="$OPENAI_API_KEY" ``` **Using a different LLM provider?** Edit `aura-values.yaml` and update the `[agent.llm]` section. See [`examples/reference.toml`](https://github.com/mezmo/aura/blob/main/examples/reference.toml) for all provider options. Wait for AURA: ```bash theme={null} kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=aura --timeout=120s ``` ### 5. Try it out The [AURA CLI](/aura/cli-reference) ships **inside the same image** as the server, so there's no separate CLI container to deploy. Exec into the running AURA pod and launch the bundled CLI against the in-pod server: ```bash theme={null} kubectl exec -it deploy/aura -- ./aura --api-url http://localhost:8080 --model kubernetes-sre ``` This drops you into an interactive REPL. Try these queries: ``` Check the health of my cluster. Are all pods running? Are there any high CPU or memory usage concerns? ``` ``` What services are running in the default namespace and what are their error rates? ``` ``` Are there any pods in CrashLoopBackOff? If so, show me their logs and related metrics. ``` ``` Show me the top 5 pods by memory usage and check if any are close to their limits. ``` The coordinator dispatches to both workers: `cluster_inspector` checks pod status and events, while `metrics_analyst` queries Prometheus for resource usage. You can toggle the SSE event panel with `/stream` to watch the orchestration in real time. Type `/quit` to exit. **Prefer a browser or a local CLI?** Port-forward the service with `kubectl port-forward svc/aura 8080:80`, then point any OpenAI-compatible client — or a locally built `aura --api-url http://localhost:8080` — at it. ## How the orchestration config works Open `aura-values.yaml` and look at the `config.content` section. The key pieces: **`[orchestration]`** -- enables orchestration mode. The coordinator agent receives every query and decides whether to answer directly, ask for clarification, or dispatch to workers. **`[orchestration.worker.cluster_inspector]`** -- a worker with `mcp_filter` set to only Kubernetes tool names. Even though both MCP servers are connected, this worker can only see and use K8s tools. **`[orchestration.worker.metrics_analyst]`** -- a worker with `mcp_filter` set to only Prometheus tool names. It can only query metrics, not touch the cluster. This separation means each worker operates with least-privilege access to tools, and the coordinator handles synthesis across domains. ### Managing Large Tool Outputs with Scratchpad This example enables scratchpad. Large Kubernetes workload listings, log exports, and Prometheus metric dumps can overflow the model's context window, so scratchpad stores them on disk instead. Both MCP servers routinely return large payloads. The `cluster_inspector` and `metrics_analyst` workers call those tools, so scratchpad keeps each worker's context focused on its own task. The coordinator has no scratchpad. * **`[agent.scratchpad]`** enables scratchpad. Both workers, `cluster_inspector` and `metrics_analyst`, inherit it because neither overrides it. * **`[mcp.servers.kubernetes.scratchpad]`** and **`[mcp.servers.prometheus.scratchpad]`** set per-server thresholds that decide when a tool's output is diverted to disk. * Because scratchpad is enabled, **`[agent.llm]`** declares **`context_window`** rather than only **`max_tokens`**. Scratchpad sizes its context budget against the declared context window. For the full list of scratchpad options, see the [Scratchpad](/aura/scratchpad) reference. ## Customizing tool filters The `mcp_filter` arrays in `aura-values.yaml` list the exact tool names exposed by each MCP server. The names in this quickstart were sourced from the upstream repos: * [kubernetes-mcp-server tools](https://github.com/containers/kubernetes-mcp-server) -- core toolset, `read_only = true` * [prometheus-mcp-server tools](https://github.com/pab1it0/prometheus-mcp-server) -- all tools To verify the tools AURA actually discovered at runtime: ```bash theme={null} kubectl logs -l app.kubernetes.io/name=aura | grep -i "tool" ``` If a tool name in `mcp_filter` doesn't match any real tool, it's silently ignored. If *none* match, the worker has zero tools and will fail. ## Cleanup ```bash theme={null} helm uninstall aura helm uninstall prometheus-mcp-server helm uninstall kubernetes-mcp-server helm uninstall otel-demo kind delete cluster --name aura-sre ``` # Request Lifecycle Source: https://docs.mezmo.com/aura/request-lifecycle Request flow, timeout configuration, cancellation, and graceful shutdown behavior. ## Overview AURA manages per-request state (cancellation tokens, subscriptions) across streaming SSE connections. This document covers the lifecycle, timeout configuration, and known limitations. *** ## Request Flow ``` Client POST /v1/chat/completions │ ▼ ┌─────────────────────────────────────┐ │ Shutdown middleware check │ │ (503 if shutdown_token cancelled) │ └─────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────┐ │ Generate request_id (UUID) │ └─────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────┐ │ Spawn producer task │ │ - Register cancellation token │ │ - Subscribe to progress events │ │ - Subscribe to tool events │ └─────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────┐ │ Stream chat with TimeoutHook │ │ - Tool calls set thread context │ │ - Tool results clear context │ └─────────────────────────────────────┘ │ ├──────────────────┬──────────────────┐ ▼ ▼ ▼ ┌────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ Normal │ │ Timeout/ │ │ Shutdown │ │ completion │ │ Disconnect │ │ (grace expired) │ └────────────────┘ └──────────────────┘ └──────────────────┘ │ │ │ │ ▼ ▼ │ ┌──────────────────┐ ┌──────────────────┐ │ │ Cancel token │ │ Cancel hook + │ │ │ Send MCP cancel │ │ registry │ │ │ Evict from pool │ │ Send [DONE] │ │ └──────────────────┘ │ Then MCP cleanup │ │ │ │ (no pool evict) │ │ │ └──────────────────┘ └──────────┬───────┴──────────────────┘ ▼ ┌─────────────────────────────────────┐ │ Cleanup (RAII guard) │ │ - Unregister cancellation │ │ - Unsubscribe progress │ │ - Unsubscribe tool events │ └─────────────────────────────────────┘ ``` *** ## Timeout Configuration ### Production Defaults | Setting | Default | Env Variable | Purpose | | ------------------------- | -------------- | -------------------------------- | ------------------------------------------------------ | | First chunk timeout | 90 sec | `FIRST_CHUNK_TIMEOUT_SECS` | Max wait for first provider chunk | | Stream timeout | 15 min | `STREAMING_TIMEOUT_SECS` | Max request duration | | Stream inactivity timeout | Disabled (`0`) | `STREAM_INACTIVITY_TIMEOUT_SECS` | Max silence between stream items after the first chunk | | Shutdown grace period | 30 sec | `SHUTDOWN_TIMEOUT_SECS` | Time for in-flight requests to finish on shutdown | | Heartbeat | 15 sec | — | Disconnect detection | ### Rationale * **First chunk timeout (90 sec)**: Catches provider connection failures early. If the LLM hasn't sent any data within this window, the request is aborted rather than hanging for the full stream timeout. The window is sized for reasoning models, which can legitimately take over a minute before the first chunk. * **Stream inactivity timeout (disabled by default)**: Closes the gap between the first-chunk timeout, which guards only the opening chunk, and the stream timeout, which is the whole-request budget. Once streaming starts, a provider that goes silent between chunks runs all the way to the stream timeout by default. Enable this to fail a stalled stream sooner. * **Stream timeout (15 min)**: Supports long-running MCP tools. Set to 0 to disable (not recommended). * **Heartbeat (15 sec)**: Standard SSE keepalive. Detects disconnect during silent tool execution. ### Tuning Stream Inactivity Timeouts There are two knobs at two layers, both disabled by default (`0`). The TOML `stream_inactivity_timeout_secs` (in `[orchestration.timeouts]`, see [configuration reference](/aura/configuration-reference)) governs coordinator and worker orchestration streams. The server `STREAM_INACTIVITY_TIMEOUT_SECS` (see [web server reference](/aura/web-server-reference)) governs single-agent streaming requests. The TOML knob exempts tool execution from its timer. The TOML deadline suspends during tool execution and re-arms on each stream item, so slow MCP tool calls and paused HITL approvals do not trip it. This makes it the reliable knob for orchestrated deployments. When the timer fires, the behavior depends on the layer. For orchestration, the affected task fails with a no-stream-progress error (an agent-timeout failure) and the run completes with partial results instead of dying mid-wave. For the server single-agent path, the request terminates as a timeout. Orchestrated worker activity reaches the server layer as tool and progress events. Those events re-arm the server deadline but cannot suspend it, so a worker tool call that goes quiet past the window can still trip the server-layer timeout. For orchestrated deployments, rely on the TOML `stream_inactivity_timeout_secs`. If you also set the server `STREAM_INACTIVITY_TIMEOUT_SECS`, size it above the TOML window. Aura pins OpenAI to Chat Completions, which streams nothing while the model is thinking, so a long think reads as inactivity and can trip the deadline. Anthropic, Gemini, and OpenAI-compatible `reasoning_content` streams re-arm while thinking, so this affects first-party OpenAI only. Size the window above worst-case think time for those models, or leave the knob off. *** ## Tool Event Correlation Rig spawns tool execution in separate tokio tasks. The hook context (where the LLM decides to call a tool) and the execution context (where MCP actually runs) are decoupled, requiring a mechanism to correlate `tool_call_id` across these boundaries — AURA does this with a per-request FIFO queue. This relies on Rig's streaming mode executing tools sequentially within a request; see [Rig Fork Changes](https://github.com/mezmo/aura/blob/main/docs/rig-fork-changes.md) for the validation methodology if you're tracking Rig upstream compatibility. *** ## Cleanup Mechanism Request resources (cancellation registration, progress/tool-event subscriptions) are cleaned up via an RAII guard that ensures cleanup runs even on panic. If the async runtime is already gone (process exit), cleanup is skipped — resources are reclaimed with the process. *** ## MCP Cancellation On client disconnect or timeout: 1. The cancellation token signals all waiting code 2. Aura sends `notifications/cancelled` to MCP servers 3. The agent is evicted from its connection pool to prevent stale connections MCP servers receive the cancellation notification and can abort in-progress operations. *** ## Graceful Shutdown The server uses a two-phase shutdown with separate cancellation tokens: | Token | Cancelled | Purpose | | ----------------------- | ------------------------------------------ | ---------------------------------------- | | `shutdown_token` | Immediately on signal | Middleware rejects new requests with 503 | | `stream_shutdown_token` | After `SHUTDOWN_TIMEOUT_SECS` grace period | Terminates remaining in-flight streams | ### Shutdown Sequence 1. **SIGTERM/SIGINT** received 2. **Phase 1 (immediate)**: `shutdown_token` cancelled — middleware returns 503 for all new requests 3. **Grace period**: In-flight streams continue running for up to `SHUTDOWN_TIMEOUT_SECS` (default 30s). Streams that complete naturally during this window are unaffected. 4. **Phase 2 (drain)**: `stream_shutdown_token` cancelled — remaining streams: * Cancel hook + request registry (stops in-flight MCP tool execution) * Send `[DONE]` to client (before MCP cleanup, so client gets clean termination) * Run MCP cleanup (send `notifications/cancelled`, close connections) * No pool eviction (pool is dying with the server) 5. **Server stops**: Workers have 10s to complete Phase 2 cleanup ### Shutdown vs Disconnect/Timeout | Behavior | Disconnect/Timeout | Shutdown | | --------------- | -------------------------------------------------- | -------------------------------------- | | Pool eviction | Yes | No (pool is dying) | | `[DONE]` timing | After MCP cleanup (Timeout) / skipped (Disconnect) | Before MCP cleanup | | Grace period | None — immediate cancel | Configurable (`SHUTDOWN_TIMEOUT_SECS`) | # Scratchpad (Context Window Management) Source: https://docs.mezmo.com/aura/scratchpad Intercept large MCP tool outputs, explore them with dedicated tools, and manage token budgets automatically. MCP tools can return responses far larger than an LLM's context window — a single Kubernetes workload listing or log export can be tens of thousands of tokens. Without intervention, this fills the context and degrades reasoning quality. Scratchpad solves this by intercepting large tool outputs and storing them on disk. The LLM gets a summary and eight read-only exploration tools (`head`, `slice`, `grep`, `schema`, `item_schema`, `get_in`, `iterate_over`, `read`) to selectively pull in only the data it needs. Scratchpad works in both single-agent and orchestration modes. ## Configuration Configure at `[agent.scratchpad]` (applies to the single agent, or provides defaults for orchestration workers) and optionally override per worker at `[orchestration.worker..scratchpad]`. Set a top-level `memory_dir` for persistence. See [`[agent.scratchpad]`](/aura/configuration-reference#agent-scratchpad) in the configuration reference for the full field table, and [`[mcp.servers..scratchpad]`](/aura/configuration-reference#per-tool-scratchpad-thresholds) for per-tool interception thresholds. ```toml theme={null} # Top-level — required when scratchpad is enabled. Shared by single-agent # scratchpad and orchestration persistence. memory_dir = "/tmp/aura" [agent.scratchpad] enabled = true context_safety_margin = 0.20 # 20% of context reserved for reasoning/output max_extraction_tokens = 10_000 # cap per extraction tool call turn_depth_bonus = 6 # extra ReAct turns when scratchpad is active [orchestration.worker.data-explorer.scratchpad] # Override just for this worker max_extraction_tokens = 5_000 ``` ## Storage locations * Single-agent: `{memory_dir}/scratchpad/` * Orchestration: `{memory_dir}/{run_id}/iteration-{n}/scratchpad/` (legacy `[orchestration.artifacts].memory_dir` still works as a fallback) ## Per-tool interception thresholds Per-tool interception thresholds are configured at `[mcp.servers..scratchpad]`. Keys are **glob patterns** (default threshold `5_120` if omitted) that are matched against tool names at interception time: ```toml theme={null} [mcp.servers.k8s-sre.scratchpad] "*_list_*" = { min_tokens = 512 } # broad "k8s_list_service_monitors" = { min_tokens = 384 } # specific override "*" = { min_tokens = 4096 } # catch-all ``` When multiple patterns match the same tool, the **longest (most specific) pattern wins**; on length ties the smallest threshold wins. ## Token counting Token counting uses real tokenization, not byte/character heuristics, so `min_tokens` and the context budget reflect actual model token cost. Dispatch is provider-aware: * **OpenAI**: `tiktoken-rs`, using `o200k_base` for GPT-5/4o/o-series models and `cl100k_base` for older models. * **Gemini**: an embedded Gemma 3 SentencePiece model (exact, fully local). * **Anthropic / Bedrock-Claude**: a calibrated `cl100k_base × 1.1` approximation, since Claude ships no public tokenizer. * Everything else falls back to `o200k_base`. ## Per-call extraction limit `max_extraction_tokens` (default `10_000`): every exploration tool checks the size of its result before returning. If a single call would exceed this cap (or the cumulative context budget), the tool returns a structured JSON error like `{"error": "head_too_large", "estimated_tokens": ..., "suggestions": [...]}` instead of the content. The LLM sees this as a successful tool result and retries with smaller params — each retry consumes a turn, which is why `turn_depth_bonus` exists. When an `iterate_over` result is too large, the `iterate_over_too_large` error leads with a ready-to-run [paginated retry over a narrower window](#paginate-iterate_over-with-offset-and-limit) that halves the current window, ahead of suggestions to request fewer fields, use `get_in`, or use `grep`. ## Paginate `iterate_over` with offset and limit `iterate_over` can project the requested fields over a window of a large array instead of the whole array, using two optional arguments. This lets a call that would otherwise exceed `max_extraction_tokens` succeed over a slice. | Argument | Type | Default | Description | | -------- | ------- | ------------------------------------- | --------------------------------------------------------------------------------------------------- | | `offset` | integer | `0` | Index of the first item to return, 0-indexed. Minimum `0`. | | `limit` | integer | *(all remaining items from `offset`)* | Maximum number of items to return, starting at `offset`. Minimum `1`; a `limit` of `0` is rejected. | Windowed rows keep their absolute array position in the `_index` field. That position is `offset` plus the row's position in the window, so paths that `get_in` resolves stay valid no matter which window produced the row. An `offset` past the end of the array returns an empty result `[]` rather than an error, so the agent can correct course. Each exploration tool call ends its result with a one-line footer that summarizes what it returned. When you paginate, that footer reports the window as a half-open interval over the total, with the start index inclusive and the end index exclusive. For example, the footer output `iterate_over: $.results (items [3..7) of 120 total, fields: [id,title])` covers the items at index 3, 4, 5, and 6. With no `offset` or `limit`, the footer reports a full scan instead, for example `2 items`. `get_in` and `item_schema` take the same `offset` and `limit` arguments. `get_in` paginates a large string value by line, and `item_schema` windows a large array by item. ## Budget and usage reporting Each agent (single-agent or orchestration worker) gets a fresh context budget scoped to that agent's effective LLM's `context_window` — workers never share an "orchestrator-level" budget. LLM-reported per-turn token counts feed back into the budget as ground truth, so remaining budget reflects actual context pressure. A per-agent `aura.scratchpad_usage` SSE event (with `agent_id`, `tokens_intercepted`, `tokens_extracted`) is emitted when the agent finishes — the same event name fires for both single-agent and worker contexts (it lives in the base `aura.*` namespace, not `aura.orchestrator.*`). See [Streaming API Guide](/aura/streaming-api-guide) for the full event reference. ## Result artifacts and `read_artifact` In orchestration, large task results are saved to artifact files under `{memory_dir}/{run_id}/artifacts/`. The scratchpad read tools resolve files anywhere under a per-agent read root, not just the scratchpad subdir — for orchestration workers the read root is the session dir, so result artifacts are explorable **in place** without copying. When a worker reads an artifact back with `read_artifact`, the same budget rules apply: an artifact that fits is inlined and recorded against the budget; one that exceeds the limit comes back as a scratchpad pointer that the worker explores in place with the read tools (`head`, `grep`, `slice`, …) — never copied into the scratchpad. The coordinator has no scratchpad, so its `read_artifact` always returns inline content. # Skills (On-Demand Instructions) Source: https://docs.mezmo.com/aura/skills Package task-specific instructions that AURA agents pull in only when a task calls for them. Skills package task-specific instructions that the agent pulls in only when a task calls for them. Each skill is a directory in the [Agent Skills format](https://agentskills.io/specification): a `SKILL.md` file with YAML frontmatter (`name`, `description`) followed by the instructions, plus optional `references/`, `scripts/`, and `assets/` subdirectories for supporting files. Rather than inlining every skill into the system prompt, AURA appends only a catalog of names and descriptions. The LLM calls the `load_skill` tool to fetch a skill's full instructions on demand, and `read_skill_file` to fetch individual resource files. `read_skill_file` resolves symlinks and rejects any path that escapes the skill directory. ## Configuration See [`[agent.skills]`](/aura/configuration-reference#agentskills) in the configuration reference for the full field table. ```toml theme={null} [agent.skills] local = [ { source = "./skills" }, # relative paths resolve from the process CWD { source = "/opt/aura/shared-skills" } ] ``` Each `source` is a directory containing skill subdirectories: ```text theme={null} skills/ └── code-review/ ├── SKILL.md # required: frontmatter (name, description) + instructions ├── references/ # optional resources, fetched via read_skill_file ├── scripts/ └── assets/ ``` ## Discovery and validation Discovery runs at agent build time and validates each skill against the specification; the frontmatter `name` must match the directory name. Directories without a `SKILL.md` are skipped. When two sources provide the same skill name, the first one loaded wins and a warning is logged. Relative sources resolve from the process current working directory in every mode (web server, standalone CLI, and A2A). `CONFIG_PATH` / `--config` locate the TOML file only; they do not change how paths inside TOML are resolved. ## Orchestration inheritance In orchestration mode the coordinator inherits `[agent.skills]`. Workers inherit it too, unless `[orchestration.worker..skills]` provides their own sources; an explicit empty list disables skills for that worker (see [Per-Worker Skills Override](/aura/configuration-reference#per-worker-skills-override) for the exact inheritance/override rules): ```toml theme={null} [orchestration.worker.knowledge.skills] local = [{ source = "./knowledge-skills" }] # worker-specific skills [orchestration.worker.operations.skills] local = [] # no skills for this worker ``` # Streaming API Guide Source: https://docs.mezmo.com/aura/streaming-api-guide OpenAI-compatible Server-Sent Events (SSE) streaming, custom aura.* events, tool result modes, and orchestration events. OpenAI-compatible Server-Sent Events (SSE) streaming for real-time responses. ## Quick Start ```bash theme={null} curl -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"messages": [{"role": "user", "content": "Hello!"}], "stream": true}' ``` ## Configuration ### Tool Result Modes The server supports three streaming modes, configured via CLI or environment variable: | Mode | Tool Call Args | Tool Results | Use Case | | ---------------- | -------------- | ------------------------------- | -------------------------------- | | `none` (default) | Actual JSON | Not streamed | Spec-compliant API clients | | `open-web-ui` | Empty `""` | Streamed via `tool_calls` | OpenWebUI "View Results" support | | `aura` | Actual JSON | Via `aura.tool_complete` events | Custom clients with AURA events | ```bash theme={null} # Spec-compliant mode (default) cargo run --bin aura -- webserver # OpenWebUI compatibility mode cargo run --bin aura -- webserver --tool-result-mode open-web-ui # AURA events mode (requires AURA_CUSTOM_EVENTS=true) AURA_CUSTOM_EVENTS=true cargo run --bin aura -- webserver --tool-result-mode aura # Via environment variable TOOL_RESULT_MODE=aura AURA_CUSTOM_EVENTS=true cargo run --bin aura -- webserver ``` ### Environment Variables | Variable | Default | Description | | -------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `TOOL_RESULT_MODE` | `none` | `none`, `open-web-ui`, or `aura` | | `TOOL_RESULT_MAX_LENGTH` | `1000` | Max chars for tool results (0 = no truncation) | | `STREAMING_TIMEOUT_SECS` | `900` | Request timeout in seconds (0 = no timeout) | | `FIRST_CHUNK_TIMEOUT_SECS` | `90` | Max seconds to wait for first provider chunk before aborting | | `STREAMING_BUFFER_SIZE` | `400` | Chunks to buffer before backpressure | | `AURA_CUSTOM_EVENTS` | `false` | Enable optional custom `aura.*` events. HITL approval lifecycle events are emitted regardless because clients may need to act on them. | | `AURA_EMIT_REASONING` | `false` | Enable `aura.reasoning` events | | `SHUTDOWN_TIMEOUT_SECS` | `30` | Grace period (seconds) for in-flight streams on shutdown | ## Server Info Endpoint `GET /aura/info` is an aura-native introspection endpoint. It returns the default agent and, per agent, its orchestration workers and configured MCP servers. This endpoint is not OpenAI-compatible; it lives under `/aura/` to keep `/v1/models` clean. The CLI uses this at boot to display orchestration workers before the first prompt in HTTP mode. ```bash theme={null} curl http://localhost:8080/aura/info | jq ``` ```json theme={null} { "default_agent": "orch", "agents": [ { "id": "orch", "model": "gpt-4o", "description": "General-purpose assistant with tool access", "workers": [ { "name": "planner", "description": "Plans work" }, { "name": "writer", "description": "Writes summaries", "model": "gpt-4o-mini" } ], "mcp_servers": { "logs": { "transport": "http_streamable", "url": "https://logs.example.com", "description": "Search logs." }, "fs": { "transport": "stdio", "command": "fs-server" } } }, { "id": "solo", "model": "gpt-4o", "mcp_servers": {} } ] } ``` The `description` on each agent object is the agent's own summary. It is distinct from the `description` inside each `workers` entry, which describes an orchestration worker. Each agent's `id` matches the `id` field in `/v1/models` (alias if set, otherwise agent name). Each agent object can also carry a `description`: the agent's human-readable summary from its `[agent].description` config (see the [configuration reference](/aura/configuration-reference#agent)). It is omitted from the response when that config field is unset. Omitting it when unset keeps older servers and clients compatible. This is the same value returned by `/v1/models` (see the [web server reference](/aura/web-server-reference)); use `/aura/info` when a client also wants the worker or MCP-server metadata this endpoint provides. The `workers` array is omitted for non-orchestration agents. Each worker's `model` is included only when it overrides the coordinator model. `mcp_servers` is a credential-free view of the agent's configured MCP servers, keyed by name and tagged by `transport` (`stdio`, `http_streamable`, or `sse`). URLs are reduced to their origin (`scheme://host:port`) — path, query, fragment, and userinfo are all dropped, since any of them can carry a token. A stdio server shows only the executable basename as `command`; its arguments and environment are never included, and neither are `headers` or `headers_from_request`. A URL that cannot be reduced to an origin appears as the sentinel ``, and a command with no extractable file name as ``. An empty object means no configured MCP servers, while a populated one lists them. Older servers omit the field entirely, and the CLI then uses its generic startup call-to-action. ## Custom AURA Events (Optional) Most custom AURA events are optional and require `AURA_CUSTOM_EVENTS=true`. HITL approval lifecycle events are the exception: the server emits them whenever an approval route needs clients to observe or act on approval state. ```bash theme={null} AURA_CUSTOM_EVENTS=true cargo run --bin aura -- webserver ``` ### Custom Event Types | Event | Description | Status | | ------------------------- | -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | | `aura.tool_requested` | LLM decided to call a tool (immediate UI feedback, has arguments) | ✅ Implemented | | `aura.tool_start` | MCP execution actually begins (has `progress_token` for correlation) | ✅ Implemented | | `aura.tool_complete` | Tool execution finished (with `duration_ms`, result/error) | ✅ Implemented | | `aura.reasoning` | LLM reasoning content (requires `AURA_EMIT_REASONING=true`) | ✅ Implemented | | `aura.progress` | MCP progress notifications during long-running tools | ✅ Implemented | | `aura.session_info` | Session metadata (model, context window) emitted at stream start | ✅ Implemented | | `aura.mcp_status` | Per-server MCP connection status emitted at stream start (connected/failed/not\_attempted, with failure reason) | ✅ Implemented | | `aura.worker_phase` | Worker phase transitions in multi-agent mode (planning/executing/analyzing) | ✅ Implemented | | `aura.tool_usage` | Usage snapshot after tool execution (associates tool IDs with token counts) | ✅ Implemented | | `aura.usage` | Final token usage emitted at stream end (prompt/completion/total) | ✅ Implemented | | `aura.scratchpad_usage` | Per-agent scratchpad usage summary (single-agent or worker), emitted when an agent finishes with scratchpad activity | ✅ Implemented | | `aura.approval_requested` | HITL approval request raised for a gated tool or `request_approval` call | ✅ Implemented for webhook and conversational routes | | `aura.approval_pending` | HITL approval is waiting for an attended decision | ✅ Implemented for conversational route | | `aura.approval_completed` | HITL approval reached a terminal outcome | ✅ Implemented for webhook and conversational routes | | `aura.orchestrator.*` | Orchestration lifecycle events (see [Orchestration Events](#orchestration-events) below) | ✅ Implemented | ### Event Flow ```mermaid theme={null} flowchart TD C["MCP execution begins"] F["MCP execution ends"] A[LLM decides to call tool] --> B["aura.tool_requested"] B --> |"Immediate UI feedback (tool_id, tool_name, arguments)"| C C --> D["aura.tool_start"] D --> |"Has progress_token for correlation"| E["aura.progress"] E --> |"MCP server sends updates
(uses progress_token)"| F F --> G["aura.tool_complete"] G --> |"Final result
(duration_ms, success, result\/error)"| H((" ")) ``` ### Event Formats Custom events use the SSE `event:` field to distinguish from standard OpenAI chunks: **Tool requested** (immediate UI feedback when LLM decides to call a tool): ``` event: aura.tool_requested data: ``` ```json theme={null} { "tool_id": "call_abc123", "tool_name": "list_files", "arguments": {"path": "/tmp"}, "agent_id": "main", "session_id": "sess_xyz" } ``` **Tool start** (when MCP execution actually begins): ``` event: aura.tool_start data: ``` ```json theme={null} { "tool_id": "call_abc123", "tool_name": "list_files", "progress_token": 42, "agent_id": "main", "session_id": "sess_xyz" } ``` Note: `progress_token` is included when available from the MCP client. Use it to correlate with `aura.progress` events. **Tool complete (success)**: ``` event: aura.tool_complete data: ``` ```json theme={null} { "tool_id":"call_abc123", "tool_name":"list_files", "duration_ms":1234, "success":true, "result":"file1.txt\nfile2.txt... [truncated]", "agent_id":"main", "session_id":"sess_xyz" } ``` **Tool complete (failure)**: ``` event: aura.tool_complete data: ``` ```json theme={null} { "tool_id": "call_abc123", "tool_name": "failing_tool", "duration_ms": 50, "success": false, "error": "Tool returned an error: Connection refused", "agent_id": "main", "session_id": "sess_xyz" } ``` Note: * Successful tool results include the `result` field (truncated per `TOOL_RESULT_MAX_LENGTH`, default 1000 chars) * Tool errors are automatically detected from Rig's error format prefixes (`ToolCallError:`, `JsonError:`, `Tool returned an error:`) * When detected, `success` is set to `false` and the `error` field contains the full error message **Reasoning** (requires both flags): ```bash theme={null} AURA_CUSTOM_EVENTS=true AURA_EMIT_REASONING=true cargo run --bin aura -- webserver ``` ``` event: aura.reasoning data: ``` ```json theme={null} { "content": "Let me analyze the request...", "agent_id": "main", "session_id": "sess_xyz" } ``` **Progress** (MCP notifications from long-running tools): ``` event: aura.progress data: ``` ```json theme={null} { "message": "Processing step 3 of 5", "phase": "mcp_progress", "percent": 60, "progress_token": 42, "agent_id": "main", "session_id": "sess_xyz" } ``` Note: Progress events are only emitted when: 1. `AURA_CUSTOM_EVENTS=true` is set 2. The MCP server sends `notifications/progress` messages during tool execution **Session info** (emitted once at stream start): ``` event: aura.session_info data: ``` ```json theme={null} { "model": "gpt-5.2", "model_context_limit": 200000, "session_id": "sess_xyz" } ``` Note: `aura.session_info` includes only `CorrelationContext` fields (`session_id`, `trace_id`) — no `agent_id`. `model_context_limit` comes from the `context_window` field in the `[agent.llm]` TOML config section (or `[orchestration.worker..llm]` for per-worker overrides). If `context_window` is not set, `model_context_limit` is omitted from the event. **MCP status** (emitted once when at least one MCP server is configured — at stream start in single-agent mode, or just after the shared manager connects in orchestration mode): ``` event: aura.mcp_status data: ``` ```json theme={null} { "servers": [ { "server_name": "mezmo", "transport": "http_streamable", "status": "connected", "tools_count": 7 }, { "server_name": "pagerduty", "transport": "http_streamable", "status": "failed", "tools_count": 0, "reason": "Connection failed: HTTP MCP server 'pagerduty' authentication failed (401 Unauthorized). Check that your headers, forwarded headers. and/or credentials are correct." } ], "session_id": "sess_xyz" } ``` Note: `status` is one of `connected`, `failed`, or `not_attempted`. This lets a client distinguish a server that is configured but unavailable (`failed`, with a `reason`) from one that connected and legitimately exposes no tools (`connected`, `tools_count: 0`). `reason` is present only for failed servers. The event is omitted entirely when no servers are configured. `aura.mcp_status` includes only `CorrelationContext` fields (`session_id`, `trace_id`) — no `agent_id`. In orchestration mode all workers share a single `McpManager`, so one `aura.mcp_status` reports the whole run's server status. The wire shape is identical to single-agent mode; it just arrives slightly later (after the manager connects, before planning) rather than at stream start. Requires `AURA_CUSTOM_EVENTS=true` in both modes. **Worker phase** (phase transitions in multi-agent mode): ``` event: aura.worker_phase data: ``` ```json theme={null} { "phase": "executing", "task_id": "task_1", "agent_id": "log_worker", "parent_agent_id": "coordinator", "session_id": "sess_xyz" } ``` Possible `phase` values: `"planning"`, `"executing"`, `"analyzing"`. `task_id` and `parent_agent_id` are omitted when not set. **Tool usage** (usage snapshot after tool execution rounds): ``` event: aura.tool_usage data: ``` ```json theme={null} { "tool_ids": ["call_abc123", "call_def456"], "prompt_tokens": 18777, "completion_tokens": 500, "total_tokens": 19277, "session_id": "sess_xyz" } ``` Emitted from the `on_stream_completion_response_finish` hook when usage data is available. Associates the completed tool IDs with a token usage snapshot. No `agent_id` field (only `CorrelationContext`). **Usage** (final token usage at stream end): ``` event: aura.usage data: ``` ```json theme={null} { "prompt_tokens": 21500, "completion_tokens": 342, "total_tokens": 21842, "session_id": "sess_xyz" } ``` Use `prompt_tokens` with `model_context_limit` from `aura.session_info` to calculate context window fill percentage: `(prompt_tokens / model_context_limit) * 100`. No `agent_id` field (only `CorrelationContext`). **Scratchpad usage** (per-agent report when an agent finishes with scratchpad activity): ``` event: aura.scratchpad_usage data: {"agent_id":"main","tokens_intercepted":15840,"tokens_extracted":1200,"session_id":"sess_xyz"} ``` Emitted once per agent that used scratchpad. It fires for both single-agent and orchestration worker contexts (in the latter, `agent_id` is the worker name). `tokens_intercepted` is the total tool output diverted to disk; `tokens_extracted` is what the agent pulled back into context via the scratchpad exploration tools. **Approval requested** (HITL approval request raised): ``` event: aura.approval_requested data: ``` ```json theme={null} { "decision_id": "019edead-beef-7000-8000-000000000001", "tool_name": "restart_deployment", "origin": { "kind": "config_gate", "matched_pattern": "restart_*" }, "scope": { "kind": "worker", "run_id": "019edead-beef-7000-8000-000000000002", "task_id": 3, "worker": "operations", "session_id": "sess_xyz" } } ``` **Approval pending** (conversational HITL is waiting for an attended decision): ``` event: aura.approval_pending data: ``` ```json theme={null} { "decision_id": "019edead-beef-7000-8000-000000000001", "tool_name": "restart_deployment", "arguments": {"namespace": "prod", "deployment": "api"}, "origin": { "kind": "config_gate", "matched_pattern": "restart_*" }, "scope": { "kind": "worker", "run_id": "019edead-beef-7000-8000-000000000002", "task_id": 3, "worker": "operations", "session_id": "sess_xyz" }, "expires_at": "2026-06-23T22:15:30Z" } ``` **Approval completed** (HITL approval reached a terminal outcome): ``` event: aura.approval_completed data: ``` ```json theme={null} { "decision_id": "019edead-beef-7000-8000-000000000001", "outcome": { "kind": "denied", "reason": "maintenance window not open" }, "duration_ms": 1820, "scope": { "kind": "worker", "run_id": "019edead-beef-7000-8000-000000000002", "task_id": 3, "worker": "operations", "session_id": "sess_xyz" } } ``` Approval events are emitted even when `AURA_CUSTOM_EVENTS=false` because they are protocol lifecycle events, not optional telemetry. The webhook route emits `aura.approval_requested` before dispatch and `aura.approval_completed` for all terminal webhook outcomes. The conversational route emits `aura.approval_requested`, then `aura.approval_pending` while the tool call is parked, then `aura.approval_completed` after the decision, timeout, or cancellation. `outcome.kind` is one of `approved`, `denied`, `timed_out`, `cancelled`, or `errored`; `errored` represents channel faults such as transport errors, non-2xx responses, or invalid JSON. `aura.approval_pending` is reserved for the conversational route and is not emitted by the webhook route. ### Client Handling Standard OpenAI clients will ignore these events (they only process `data:` lines without `event:` prefix). Custom clients can filter by event type: ```javascript theme={null} for (const line of chunk.split('\n')) { if (line.startsWith('event: ')) { const eventType = line.slice(7); // Handle aura.tool_start, aura.tool_complete, etc. } if (line.startsWith('data: ')) { const data = JSON.parse(line.slice(6)); // Handle OpenAI chunk or custom event data } } ``` ### Correlation Fields Most custom events include correlation fields for tracing: | Field | Description | | ------------ | ------------------------------------------ | | `session_id` | Chat session ID (from request metadata) | | `trace_id` | OTEL trace ID (when available) | | `agent_id` | Agent identifier (`main` for single-agent) | Approval lifecycle events carry `decision_id` and HITL `scope` instead of the shared `AgentContext` / `CorrelationContext` fields. Use `decision_id` to correlate `aura.approval_requested` with `aura.approval_completed`, and inspect `scope` for the requesting surface (`single`, `worker`, or future `coordinator`). #### Tool Event Correlation Use these fields to correlate tool-related events: | Correlation | Events | Field | | ---------------- | ------------------------------------------------- | ---------------- | | Tool lifecycle | `tool_requested` → `tool_start` → `tool_complete` | `tool_id` | | Progress updates | `tool_start` → `progress` | `progress_token` | Example correlation: ```mermaid theme={null} flowchart TD A["tool_requested
(tool_id: 'call_abc')"] -->|"shows arguments to user"| B B["tool_start
(tool_id: 'call_abc', progress_token: 42)"] -->|"MCP execution begun"| C C["progress
(progress_token: 42, progress: 50, total: 100)"] -->|"correlates via token"| D D["tool_complete
(tool_id: 'call_abc', duration_ms: 1234)"] -->|"final result"| E((" ")) ``` ## Orchestration Events When `orchestration.enabled = true` and `AURA_CUSTOM_EVENTS=true`, the server emits orchestration-specific events covering the Plan/Execute/Continue lifecycle. These events are emitted alongside the standard `aura.*` events above. ### Orchestration Event Types | Event | Description | | ---------------------------------------- | ------------------------------------------------------------------------------------------------- | | `aura.orchestrator.plan_created` | Coordinator decomposed query into a task DAG | | `aura.orchestrator.direct_answer` | Coordinator answered without orchestration | | `aura.orchestrator.clarification_needed` | Coordinator needs user clarification | | `aura.orchestrator.task_started` | Worker began executing a task | | `aura.orchestrator.task_completed` | Worker finished task (success/failure with duration) | | `aura.orchestrator.worker_reasoning` | Worker reasoning content with task/worker attribution | | `aura.orchestrator.iteration_complete` | Iteration finished with replan decision, reasoning, and phase timing (planning/execution/tool ms) | | `aura.orchestrator.replan_started` | Replan cycle triggered (coordinator-routed or task failures) | | `aura.orchestrator.synthesizing` | Coordinator merging worker results (includes iteration number) | | `aura.orchestrator.tool_call_started` | A tool call began (coordinator or worker); see the tool-coverage note below | | `aura.orchestrator.tool_call_completed` | The matching tool call finished (duration, success) | ### Orchestration Event Flow ```mermaid theme={null} flowchart TD A([User query received]) --> R{Coordinator routing} R -->|orchestrated / routed| B["plan_created
(goal, tasks, routing_mode, routing_rationale)"] R -->|simple query| DA["direct_answer
(response, routing_rationale)"] R -->|ambiguous| CL["clarification_needed
(question, options, routing_rationale)"] DA --> Z([Done]) CL --> Z B --> C["task_started
Worker assigned (task_id, worker_id, orchestrator_id)"] C --> D["worker_reasoning
Worker thinking (task_id, worker_id, content)"] D --> E["tool_call_started
Worker calls a tool: MCP, skill, or operation (tool_call_id, tool_name, worker_id)"] E --> F["tool_call_completed
Tool result (duration_ms, success)"] F --> G["task_completed
Worker finished (duration_ms, success, result)"] G --> H["synthesizing
Coordinator merging results (iteration)"] H --> PE{Post-execute routing} PE -->|respond| DA2["direct_answer
(response, routing_rationale)"] PE -->|clarify| CL2["clarification_needed
(question, options, routing_rationale)"] PE -->|replan| J["replan_started
trigger: 'post_execute_create_plan'"] DA2 --> I["iteration_complete
(will_replan: false)"] CL2 --> I J --> I2["iteration_complete
(will_replan: true)"] I2 -->|loop back| B I --> Z ``` **Routing decisions happen twice**: once on initial query (before any work) and again post-execute (after workers finish). Both paths can produce `direct_answer`, `clarification_needed`, or a new plan. The initial routing has no `iteration_complete`; the post-execute routing always emits one. ### Orchestration Event Formats **Plan created** (coordinator decomposed query into tasks): ``` event: aura.orchestrator.plan_created data: ``` ```json theme={null} { "goal": "Calculate (3+7)*2 and list files", "tasks": ["Calculate (3+7)*2", "List files in /tmp"], "routing_mode": "orchestrated", "routing_rationale": "Multi-step: arithmetic + file listing", "agent_id": "coordinator", "session_id": "sess_xyz" } ``` The `routing_mode` field indicates how the coordinator routed the query: * `"routed"` — classified to a single worker * `"orchestrated"` — multi-task DAG with continuation The optional `planning_response` field contains the coordinator's raw planning text and is omitted when empty. **Direct answer** (coordinator answered without orchestration): ``` event: aura.orchestrator.direct_answer data: ``` ```json theme={null} { "response": "The answer is 42", "routing_rationale": "Simple factual query, no tools needed", "agent_id": "coordinator", "session_id": "sess_xyz" } ``` **Clarification needed** (coordinator needs more information): ``` event: aura.orchestrator.clarification_needed data: ``` ```json theme={null} { "question": "Which environment should I check?", "options": ["production", "staging", "development"], "routing_rationale": "Ambiguous target environment", "agent_id": "coordinator", "session_id": "sess_xyz" } ``` Note: `options` is omitted when the coordinator does not suggest choices. **Task started** (worker begins execution): ``` event: aura.orchestrator.task_started data: ``` ```json theme={null} { "task_id": 0, "description": "Calculate (3+7)*2", "worker_id": "arithmetic", "orchestrator_id": "orch-1", "agent_id": "coordinator", "session_id": "sess_xyz" } ``` **Worker reasoning** (worker thinking with attribution): ``` event: aura.orchestrator.worker_reasoning data: ``` ```json theme={null} { "task_id": 0, "worker_id": "arithmetic", "content": "I need to add 15 and 27...", "agent_id": "coordinator", "session_id": "sess_xyz" } ``` Note: requires both `AURA_CUSTOM_EVENTS=true` and `AURA_EMIT_REASONING=true`. Worker reasoning is also emitted as `aura.reasoning` with `agent_id` set to the worker name (e.g., `"arithmetic"`) and `parent_agent_id: "coordinator"` for backward-compatible aggregation. **Tool call started** (coordinator or worker calls a tool): ``` event: aura.orchestrator.tool_call_started data: ``` ```json theme={null} { "task_id": 0, "tool_call_id": "call_abc123", "tool_name": "add", "worker_id": "arithmetic", "arguments": {"a": 3, "b": 7}, "agent_id": "coordinator", "session_id": "sess_xyz" } ``` Note: `task_id` is omitted if it could not be determined. `arguments` is omitted when not available. **Tool call completed** (the matching tool call finished): ``` event: aura.orchestrator.tool_call_completed data: ``` ```json theme={null} { "task_id": 0, "tool_call_id": "call_abc123", "success": true, "duration_ms": 42, "result": "10", "agent_id": "coordinator", "session_id": "sess_xyz" } ``` Note: `task_id` is omitted if it could not be determined. `result` is truncated per `TOOL_RESULT_MAX_LENGTH` and omitted when empty. **Tool coverage:** these events fire for the coordinator (`worker_id: "main"`) as well as workers. They cover MCP tools, the skill tools (`load_skill`, `read_skill_file`), and orchestration operations (`read_artifact`, `submit_result`, `list_prior_runs`). Scratchpad exploration tools are suppressed by default and emit only when `AURA_EMIT_SCRATCHPAD_TOOL_EVENTS` is set. **Task completed** (worker finished with result): ``` event: aura.orchestrator.task_completed data: ``` ```json theme={null} { "task_id": 0, "success": true, "duration_ms": 1500, "orchestrator_id": "orch-1", "worker_id": "arithmetic", "result": "The result is 20", "agent_id": "coordinator", "session_id": "sess_xyz" } ``` **Iteration complete** (replan decision after execution): ``` event: aura.orchestrator.iteration_complete data: ``` ```json theme={null} { "iteration": 1, "will_replan": false, "reasoning": "All tasks completed successfully", "planning_ms": 1180, "execution_ms": 4620, "task_compute_ms": 4500, "tool_ms": 820, "agent_id": "coordinator", "session_id": "sess_xyz" } ``` The `reasoning` and `gaps` fields are included only when non-empty (i.e., when replanning is triggered). **Phase timing fields** (all milliseconds, present on every `iteration_complete`): | Field | Meaning | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `planning_ms` | Prompt → plan created. Includes planning-correction retries. For replanned iterations this is the prior iteration's continuation-decision latency (that call produced this iteration's plan). | | `execution_ms` | Plan ready → continuation-prompt entrypoint — one iteration's execution span (worker waves + persistence drain + result consolidation), measured as wall-clock. | | `task_compute_ms` | Sum of per-task wall durations across the iteration (aggregate compute; exceeds `execution_ms` when tasks run in parallel). | | `tool_ms` | Sum of tool-call durations recorded for the iteration's tasks. | To separate time the LLM spent *deciding what to call* from time spent *executing tools*, compute `execution_ms - tool_ms` ≈ LLM-thinking time. This is exact for single-task routes; for parallel waves it is approximate, since `tool_ms` and `task_compute_ms` are summed compute rather than wall-clock — compare them against `execution_ms` to gauge overlap. The same four fields are written to the run manifest (`phase_timings`) and recorded as `orchestration.{planning,execution,task_compute,tool}_ms` attributes on the `orchestration.iteration` OTel span. **Replan started** (new planning cycle triggered): ``` event: aura.orchestrator.replan_started data: ``` ```json theme={null} { "iteration": 2, "trigger": "post_execute_create_plan", "agent_id": "coordinator", "session_id": "sess_xyz" } ``` Triggers: `"post_execute_create_plan"` (coordinator routed back to planning after evaluating worker results). **Synthesizing** (consolidating task results for coordinator decision): ``` event: aura.orchestrator.synthesizing data: ``` ```json theme={null} { "iteration": 1, "agent_id": "coordinator", "session_id": "sess_xyz" } ``` Fires before the post-execute coordinator call. Bookends with `iteration_complete`, which fires after the coordinator's routing decision. Note: Workers that use scratchpad emit `aura.scratchpad_usage` when they finish — see the [Custom Event Types](#custom-event-types) section above. This is a base `aura.*` event (not orchestration-specific) so the same event fires for single-agent deployments and workers alike. ### Orchestration Correlation | Correlation | Events | Field | | --------------- | ---------------------------------------------------------------------- | ------------------------------------------- | | Task lifecycle | `task_started` → `worker_reasoning` → `tool_call_*` → `task_completed` | `task_id` | | Tool lifecycle | `tool_call_started` → `tool_call_completed` | `tool_call_id` | | Worker identity | `task_*`, `worker_reasoning`, `tool_call_started` | `worker_id` | | Agent hierarchy | All orchestration events | `agent_id` (`"coordinator"` or worker name) | | Replan cycle | `iteration_complete` → `replan_started` → `plan_created` | `iteration` | ## SSE Event Reference ### Event Types by Mode | Event | Description | `none` | `open-web-ui` | `aura` | | --------------- | ----------------------- | :-----------: | :------------------: | :-------------------------: | | **Text chunk** | Token-by-token content | ✅ | ✅ | ✅ | | **Tool call** | Tool name + arguments | ✅ (with args) | ✅ (empty args) | ✅ (with args) | | **Tool result** | Tool execution output | - | ✅ (via `tool_calls`) | ✅ (via aura.tool\_complete) | | **Final chunk** | `finish_reason` + usage | ✅ | ✅ | ✅ | | **\[DONE]** | Stream termination | ✅ | ✅ | ✅ | ### Message Formats **First text chunk** (includes `role`): ```json theme={null} { "choices": [ { "delta": { "role": "assistant", "content": "Hello" } } ] } ``` **Subsequent text chunks**: ```json theme={null} { "choices": [ { "delta": { "content": " world" } } ] } ``` **Tool call (`none` mode)** - includes actual arguments: ```json theme={null} { "choices": [ { "delta": { "tool_calls": [ { "index": 0, "id": "call_xyz", "type": "function", "function": { "name": "list_files", "arguments": "{\"path\":\"/tmp\"}" } } ] } } ] } ``` **Tool call (`open-web-ui` mode)** - empty arguments for UI compatibility: ```json theme={null} { "choices": [ { "delta": { "tool_calls": [ { "index": 0, "id": "call_xyz", "type": "function", "function": { "name": "list_files", "arguments": "" } } ] } } ] } ``` **Tool result (`open-web-ui` mode only)** - sent as second delta with same index: ```json theme={null} { "choices": [ { "delta": { "tool_calls": [ { "index": 0, "id": "call_xyz", "type": "function", "function": { "name": "", "arguments": "{\"files\":[\"a.txt\",\"b.txt\"]}" } } ] } } ] } ``` **Final chunk**: ```json theme={null} { "choices": [ { "delta": {}, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 10, "completion_tokens": 20, "total_tokens": 30 } } ``` **Stream end**: ``` data: [DONE] ``` ### `finish_reason` Values | Value | Meaning | | ------------ | -------------------------------------------- | | `stop` | Normal completion | | `tool_calls` | Response included tool execution | | `length` | Response truncated due to `max_tokens` limit | ## Client Examples ### JavaScript ```javascript theme={null} const response = await fetch('/v1/chat/completions', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ messages: [{ role: 'user', content: 'List files in /tmp' }], stream: true }) }); const reader = response.body.getReader(); const decoder = new TextDecoder(); while (true) { const { done, value } = await reader.read(); if (done) break; for (const line of decoder.decode(value).split('\n')) { if (!line.startsWith('data: ')) continue; const data = line.slice(6); if (data === '[DONE]') break; const chunk = JSON.parse(data); const delta = chunk.choices[0]?.delta; if (delta?.content) { process.stdout.write(delta.content); } if (delta?.tool_calls) { console.log('Tool call:', delta.tool_calls[0].function.name); } } } ``` ### Python ```python theme={null} import httpx import json with httpx.stream('POST', 'http://localhost:8080/v1/chat/completions', json={'messages': [{'role': 'user', 'content': 'Hello!'}], 'stream': True} ) as response: for line in response.iter_lines(): if not line.startswith('data: '): continue data = line[6:] if data == '[DONE]': break chunk = json.loads(data) delta = chunk['choices'][0].get('delta', {}) if content := delta.get('content'): print(content, end='', flush=True) if tool_calls := delta.get('tool_calls'): print(f"\nTool: {tool_calls[0]['function']['name']}") ``` ## Multi-Turn Tool Execution Unlike standard OpenAI API (where tool execution is client-side), this server executes tools server-side and continues streaming. After tool execution completes, text resumes with a `\n\n` separator for readability: ``` I'll check that for you. [tool call: list_files] [tool executes server-side] Here are the files I found: ... ``` The separator is automatically injected when text chunks resume after a `ToolResult` event. ## Connection Behavior | Behavior | Description | | ---------------- | ---------------------------------------------------------------------------------- | | **Timeout** | 900s default (configurable via `STREAMING_TIMEOUT_SECS`, 0 = disabled) | | **Disconnect** | Server detects client disconnect and cancels in-flight operations | | **Backpressure** | Bounded buffer prevents memory exhaustion | | **Cancellation** | Timeout or disconnect triggers MCP tool cancellation via `notifications/cancelled` | ## Graceful Shutdown On SIGTERM or SIGINT, the server performs a two-phase shutdown to let in-flight requests finish: ```mermaid theme={null} flowchart TD A([SIGTERM/SIGINT received]) --> B B["**Phase 1: Gate** _(immediate)_ • New requests rejected with 503 • In-flight streams continue running"] B -->|"grace period _(SHUTDOWN_TIMEOUT_SECS, default 30s)_ in-flight streams may complete naturally"| C C["**Phase 2: Drain** _(after grace period)_ • Remaining streams cancelled • Each stream sends \[DONE\] to client • MCP cleanup (cancel + close)"] C -->|"10s buffer for \[DONE\] delivery + MCP cleanup"| D D([Server exits]) ``` | Phase | Timing | What happens | | ---------------- | ----------------------------------------- | ----------------------------------------------------------------------------------------- | | **Gate** | Immediate | Middleware returns 503 for all new requests (including `/health`) | | **Grace period** | 0 – `SHUTDOWN_TIMEOUT_SECS` (default 30s) | In-flight streams continue running; streams that finish naturally are unaffected | | **Drain** | After grace period | `stream_shutdown_token` cancelled; remaining streams send `[DONE]`, then MCP cleanup runs | | **Exit** | Grace period + 10s buffer | Actix force-closes any remaining connections | Configure the grace period: ```bash theme={null} # Allow 60 seconds for in-flight requests to finish SHUTDOWN_TIMEOUT_SECS=60 cargo run --bin aura -- webserver # Or via CLI flag cargo run --bin aura -- webserver --shutdown-timeout-secs 60 ``` **K8s tip**: Set `terminationGracePeriodSeconds` to at least `SHUTDOWN_TIMEOUT_SECS + 15` (default: 45s). The total shutdown budget is grace period + 10s Actix buffer. During Phase 1, `/health` returns 503 — readiness probes will fail immediately, removing the pod from service endpoints. The `/health` endpoint returns 503 during shutdown (same middleware gate as all routes). This is intentional — it signals load balancers and K8s readiness probes to stop routing traffic to this instance. ## Response Headers ```http theme={null} Content-Type: text/event-stream Cache-Control: no-cache X-Accel-Buffering: no ``` # Telemetry & Privacy Source: https://docs.mezmo.com/aura/telemetry Anonymous, opt-out CLI telemetry — the three-state consent model, exactly what is and isn't collected, kill switches, and how to audit it. The `aura` CLI emits **anonymous** product telemetry to PostHog so maintainers can answer one question: *are people using the CLI, in which modes, and do their chat turns succeed?* Telemetry is **CLI-only** (the server sends nothing), **opt-out**, and **notice-gated**: nothing is sent until you have seen a one-time notice and sent your first message, and it is never sent at all if you opt out. This document is the canonical contract: every state, every event, every value **not** collected, every control, and how to verify all of it yourself. Two principles govern what is here, and any future change (see the [telemetry design ADR](https://github.com/mezmo/aura/blob/main/docs/adr/2026-06-23-cli-product-telemetry.md)): 1. **No tracking without a concrete improvement hypothesis.** Nothing is collected unless we can state how it improves Aura for users. 2. **The why/how is documented in the code**, next to each event. To see exactly what your install has sent (or *would* send), read this file and then `cat ~/.aura/telemetry/events.jsonl`. That local log is written for every captured event in every state, so even a held or disabled install shows you precisely what it is holding. *** ## The three states | State | Meaning | Behaviour | | ------------ | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Unknown** | No preference recorded yet. | **Held.** Events are written to the local inspection log so you can see what *would* be sent, but **nothing is sent**, and held events are **never backfilled** if you later enable. | | **Enabled** | You sent a message after the notice, or `[telemetry] enabled = true` is recorded. | May send. | | **Disabled** | A kill switch or explicit opt-out is in effect. | Held; nothing sent. | ### First-run notice and first-message consent The first time you launch the interactive REPL with no recorded preference, Aura prints a one-time notice: it states that anonymous telemetry is collected, links here, and tells you how to opt out. Telemetry stays **held** (`Unknown`) until you **send your first chat message**: * sending a message is treated as consent → **Enabled**, and `[telemetry] enabled = true` is written to `~/.aura/cli.toml`; * slash commands never grant consent: `/telemetry disable` → **Disabled** (persisted); `/telemetry status`, `/telemetry recent`, `/help`, typos, and unknown commands leave the state **Unknown** (inspect first, decide later); * quitting (`/quit`, Ctrl-D) → stays **Unknown**, so the notice returns next launch. Nothing is sent during the launch in which the notice first appears until that first message, so you always have a chance to opt out (or inspect with `/telemetry status` / `/telemetry recent`) before any telemetry leaves your machine. ### One-shot mode never sends `aura --query "…"` (one-shot) is non-interactive: it cannot show the notice, stays **Unknown**, and only ever writes to the local inspection log. It never transmits. *** ## What is collected Each event carries a small, fixed envelope plus the per-event properties below. Every event has a matching event struct in [`crates/aura-telemetry/src/events.rs`](https://github.com/mezmo/aura/blob/main/crates/aura-telemetry/src/events.rs) whose doc comment states why it exists. **Envelope** (every event): a random anonymous install id (`distinct_id`), a per-process random session id, the Aura version, OS family (`linux`/`macos`/`windows`/`other`), deployment method (`local`/`standalone-cli`/`other`), and `aura_source = "cli"`. PostHog server-side IP capture and geo-IP are explicitly suppressed (`$ip: ""`, `$geoip_disable: true`). | Event | Properties | Why we collect it | | ------------------------ | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cli_session_started` | `interactive`, `standalone_mode`, `client_tools_enabled` (all booleans) | Tells us which run modes to prioritise for UX/perf work, and how often the audit-sensitive client-tools path is used. | | `chat_request_started` | none beyond the envelope | Turn volume, the core adoption signal that justifies continued investment. | | `chat_request_completed` | `success` (boolean) | Turn success rate surfaces reliability regressions to fix. | | `cli_session_ended` | `exit_reason` (`quit` \| `eof` \| `interrupt` \| `error`) | When telemetry remains enabled through REPL exit, provides a best-effort counterpart to `cli_session_started`; their timestamps can yield session length, and the exit path distinguishes clean completion (`quit`/`eof`) from abnormal termination (`interrupt`/`error`). | The `chat_request_*` pair fires once per chat turn, the same way regardless of HTTP or standalone backend (no double-reporting). Session length and turn volume are **derived** in analysis from these events' timestamps and `session_id` — they are not carried as event properties. Session pairing is not guaranteed: `/telemetry disable` during a session suppresses its ending event, and best-effort delivery means transport failures or a full telemetry channel can drop events. ## What is **not** collected No prompts, no responses, no message content of any kind. No model or provider identifier (deferred until a typed representation is settled). No token counts, no latency. No file paths, no working directory, no hostname, IP, MAC, username, CPU arch, kernel, or distro. No free-form strings at all: the property allow-list ([`crates/aura-telemetry/src/properties.rs`](https://github.com/mezmo/aura/blob/main/crates/aura-telemetry/src/properties.rs)) has no `String` variant, so a free-form field **fails to compile**. *** ## Controls | Control | Effect | | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | Send no message / `/quit` on first run | Stays **Unknown**; nothing sent. | | `/telemetry disable` | **Disabled**; persists `[telemetry] enabled = false` to `~/.aura/cli.toml`. | | `[telemetry] enabled = false` in `~/.aura/cli.toml` or a project `.aura/cli.toml` | **Disabled**. `enabled = false` in *either* layer wins. | | `DO_NOT_TRACK=1` | **Disabled** (honours the [Console Do Not Track](https://consoledonottrack.com/) convention). | | `AURA_TELEMETRY_DISABLED=1` | **Disabled**. | | Running under CI (`CI`, `GITHUB_ACTIONS`, …) or `cargo test` | **Disabled** automatically. | | `AURA_TELEMETRY_LOG_EVENTS=0` (or `false`/`no`/`off`) | Disables the local inspection log (telemetry state is unchanged). | Self-hosting: point telemetry at your own PostHog with `AURA_TELEMETRY_ENDPOINT` / `AURA_TELEMETRY_API_KEY` (or the matching `cli.toml` fields). Env wins over file wins over the built-in defaults (`https://us.i.posthog.com` and Aura's bundled project key). That bundled key is a write-only PostHog public key (publish-only, it cannot read any data back), which is why shipping it in source is safe. If a build ever clears the key, requests simply 401 and are logged at debug, and Aura is never affected. Reset your anonymous install id any time: `rm ~/.aura/install-id`. *** ## Inspect what was (or would be) sent * `/telemetry status`: current state, endpoint, install-id path, inspection-log path, and dropped-event count. * `/telemetry recent [N]`: the last N inspection-log rows, each marked `[sent]` or `[not sent: ]`. * `cat ~/.aura/telemetry/events.jsonl`: the raw local log (JSON Lines), written for every captured event in every state. ## Audit guide Everything above is verifiable from source: * [`crates/aura-telemetry/src/properties.rs`](https://github.com/mezmo/aura/blob/main/crates/aura-telemetry/src/properties.rs): the sealed property allow-list and the `IntoTelemetryProperty` gate. * [`crates/aura-telemetry/tests/compile_fail/`](https://github.com/mezmo/aura/tree/main/crates/aura-telemetry/tests/compile_fail): proof that a `String`, integer, or property-value field fails to compile. * [`crates/aura-telemetry/tests/wire_format.rs`](https://github.com/mezmo/aura/blob/main/crates/aura-telemetry/tests/wire_format.rs): the exact PostHog payload, asserted end-to-end against a mock server (run with `--nocapture` to see it). * [`crates/aura-telemetry/src/events.rs`](https://github.com/mezmo/aura/blob/main/crates/aura-telemetry/src/events.rs): every event, with its why/how. * [`crates/aura-telemetry/src/sink.rs`](https://github.com/mezmo/aura/blob/main/crates/aura-telemetry/src/sink.rs): the only code that opens an outbound connection. # Tracing & Span Layout Source: https://docs.mezmo.com/aura/tracing-spans Enable OpenTelemetry, understand AURA's span layout, and interpret OpenInference span kinds. ## Enabling tracing OpenTelemetry support is enabled by default via the `otel` feature. Configure your OTLP endpoint using standard environment variables (for example `OTEL_EXPORTER_OTLP_ENDPOINT`) to export traces — no endpoint set means no OTel layer is installed. AURA emits spans using the [OpenInference](https://github.com/Arize-ai/openinference/tree/main/spec) semantic convention (`llm.*`, `tool.*`, `input.*`, `output.*`) rather than the `gen_ai.*` conventions. Any `gen_ai.*` attributes from underlying provider libraries (Rig.rs) are automatically translated to OpenInference equivalents at export time. This makes AURA traces natively compatible with [Phoenix](https://github.com/Arize-ai/phoenix) and other OpenInference-aware observability tools. ## Trace structure Every request produces two traces: 1. **HTTP trace** — covers the request/response lifecycle 2. **Agent trace** — covers the LLM/tool execution The agent trace is rooted at `agent.stream` with `parent: None` so Phoenix sees it as an independent trace root with all LLM I/O attributes. ### HTTP trace (both modes) ```text theme={null} chat_completions (CHAIN) └── streaming_completion (CHAIN) ``` ### Single-agent mode ```text theme={null} agent.stream (AGENT, ROOT) └── agent.turn (LLM) ├── execute_tool (TOOL) │ └── mcp.tool_call (TOOL) └── execute_tool (TOOL) └── mcp.tool_call (TOOL) ``` ### Orchestration mode ```text theme={null} agent.stream (AGENT, ROOT) └── orchestration (CHAIN) ├── orchestration.planning (CHAIN) │ └── agent.turn (LLM) → execute_tool → mcp.tool_call └── orchestration.iteration (CHAIN) └── orchestration.worker (AGENT) └── agent.turn (LLM) → execute_tool → mcp.tool_call ``` ## Span attributes ### Agent root (`agent.stream`) `user.id`, `session.id`, `metadata`, `input.value`, `output.value`, `llm.token_count.prompt`, `llm.token_count.completion`, `llm.token_count.total` ### Orchestration spans | Span | Attributes | | ------------------------- | ----------------------------------------------------------------------------------------------------------------- | | `orchestration` | `orchestration.goal`, `orchestration.max_iterations`, `orchestration.routing` (direct/clarification/orchestrated) | | `orchestration.planning` | `orchestration.phase` | | `orchestration.iteration` | `orchestration.iteration`, `orchestration.task_count`, `orchestration.quality_score`, `orchestration.will_replan` | | `orchestration.worker` | `orchestration.task_id`, `orchestration.worker`, `orchestration.task` | Token usage (`llm.token_count.*`) is recorded on all orchestration phase spans (planning, worker). ### Tool spans (`execute_tool`) For HITL-gated tool calls, `execute_tool` carries a `decision_id` attribute whose value is the UUID of the approval decision that gated the call. This is the same `decision_id` that appears on the approval webhook payload, the SSE lifecycle events, and the approval ingress endpoint (see [Human-in-the-Loop Approval Gates](/aura/hitl)). The attribute is stamped before the approval outcome is known, so every gated call carries it regardless of outcome — `approved`, `denied`, `timed_out`, `cancelled`, or `errored`. It is stamped for gated calls only; ungated tool calls carry no `decision_id` attribute. `decision_id` sits on the `execute_tool` (TOOL) parent, not on the nested `mcp.tool_call` child, where the tool's own status and errors land. ## OpenInference span kinds | Kind | Spans | | --------- | ---------------------------------------------------------------------------------------------------------------- | | **LLM** | `chat`, `chat_streaming`, `agent.turn` | | **TOOL** | `execute_tool`, `mcp.tool_call` | | **AGENT** | `agent.stream`, `agent.prompt`, `agent.chat`, `orchestration.worker` | | **CHAIN** | `chat_completions`, `streaming_completion`, `orchestration`, `orchestration.planning`, `orchestration.iteration` | `chat`, `agent.prompt`, and `agent.chat` are non-streaming entry points; the trace diagrams above cover the streaming path, which is the one AURA's HTTP/CLI backends use. ## Span parenting * `agent.stream` is created with `parent: None` to break the link from the HTTP handler trace, making it an independent trace root in Phoenix. * The producer task is instrumented with `agent.stream` so Rig's `agent.turn` becomes a direct child. * In orchestration mode, the orchestrator instruments its spawned task with the `agent.stream` span so all orchestration child spans nest under the trace root. * Tool execution propagates the current span into its background task so `mcp.tool_call` nests under Rig's `execute_tool`. ## Content recording When `OTEL_RECORD_CONTENT=true`, prompt/completion text and tool arguments/results are recorded as span attributes, truncated to `OTEL_CONTENT_MAX_LENGTH` (default 1000 bytes, rounded to a UTF-8 boundary). ## Known limitations * **Tool error propagation**: Tool errors are only recorded on the `mcp.tool_call` child span, not on Rig's `execute_tool` parent. This is intentional — `mcp.tool_call` is the canonical TOOL span for Phoenix. # AURA Web Server Reference Source: https://docs.mezmo.com/aura/web-server-reference OpenAI-compatible web API server that exposes AURA agents through a standard chat completions endpoint — endpoints, deployment env vars, and architecture. **Active development** — APIs and configuration may change between releases as AURA evolves. [Issues and feature requests](https://github.com/mezmo/aura/issues) are welcome — we'd love your feedback. **Part of the [AURA Project](https://github.com/mezmo/aura)** - A production-ready framework for building AI agents with declarative TOML configuration. OpenAI-compatible web API server that exposes AURA agents through a standard chat completions endpoint. ## When to run the server The [CLI](/aura/cli-reference) starts an agent per terminal session, which suits interactive work. Running `aura webserver` instead keeps agents resident behind an HTTP API, so a caller that isn't a person at a keyboard can start a run: an alert pipeline reacting to a firing monitor, a scheduled job, a CI step, or a chat client such as LibreChat or OpenWebUI. Requests arrive as OpenAI chat completions, so any system that already speaks that format can call AURA directly. The server exposes no alert-specific receiver, so a monitoring tool that posts its own schema (Alertmanager and PagerDuty webhooks, for example) needs a small adapter to translate its payload into a chat completion request. Long-running investigations can stream progress back over SSE; see the [Streaming API Guide](/aura/streaming-api-guide). ## Features * **OpenAI Compatible**: Implements `/v1/chat/completions` endpoint following OpenAI's API schema * **Multi-Turn Conversations**: Maintains conversation context across requests * **Full Tool Integration**: Supports all MCP transports (HTTP, SSE, STDIO) and client-side tool passthrough * **Health Monitoring**: `/health` endpoint for container health checks * **Production Ready**: Stateless processing with pre-built agent, Docker-ready ## Quick Start ```bash theme={null} # Build cargo build --release --bin aura # Start with default config (config.toml) cargo run --bin aura -- webserver # Or with custom configuration file CONFIG_PATH=my-config.toml cargo run --bin aura -- webserver # Or with a directory of configs (serves multiple agents) CONFIG_PATH=configs/ cargo run --bin aura -- webserver # Custom host/port HOST=0.0.0.0 PORT=3000 cargo run --bin aura -- webserver ``` ## API Endpoints ### Health Check ```bash theme={null} GET /health ``` Response: ```json theme={null} {"status": "healthy"} ``` ### List Models (Agents) ```bash theme={null} GET /v1/models ``` Returns all loaded agents. Each agent's `alias` (or `name` if no alias is set) is its model `id`. The `owned_by` field defaults to the underlying LLM provider (e.g. `"openai"`, `"anthropic"`) and can be overridden with `model_owner` in the agent config. Clients like LibreChat and OpenWebUI use this endpoint to populate their model picker. Each entry can also carry a `description`, a human-readable summary sourced from the agent's `[agent].description` config (see the [configuration reference](/aura/configuration-reference#agent)). This extends the OpenAI model object. The key is omitted when `[agent].description` is unset, so strict OpenAI clients can ignore it. The `/aura/info` endpoint reports this same per-agent `description`; both read the agent's `[agent].description` config value rather than computing it independently. `/v1/models` alone is enough to populate a model picker in an OpenAI-compatible client. Response: ```json theme={null} { "object": "list", "data": [ {"id": "devops", "object": "model", "created": 1677649963, "owned_by": "mezmo", "description": "General-purpose assistant with tool access"}, {"id": "research-assistant", "object": "model", "created": 1677649963, "owned_by": "mezmo"} ] } ``` ### Chat Completions ```bash theme={null} POST /v1/chat/completions Content-Type: application/json ``` The `model` field selects which agent handles the request by matching against agent `alias` or `name`. Agent selection follows this order: 1. If only one config is loaded, it is always used (the `model` field is ignored) 2. Otherwise, `model` is matched first, then `DEFAULT_AGENT` if `model` is absent 3. Returns a 400 error if multiple configs are loaded and neither `model` nor `DEFAULT_AGENT` is supplied at all 4. Returns a 404 error if a `model` or `DEFAULT_AGENT` value is supplied but matches no loaded config Request body: ```json theme={null} { "model": "devops", "messages": [ {"role": "user", "content": "What tools do you have available?"} ] } ``` Response: ```json theme={null} { "id": "chatcmpl-1865d39015e49520", "object": "chat.completion", "created": 1758043845, "model": "openai/gpt-4o-mini", "choices": [{ "index": 0, "message": { "role": "assistant", "content": "I have the following tools available:\n\n1. **Log Analysis**: Export logs, analyze for root causes, and apply time-based filtering.\n2. **Knowledge Base**: Search AWS Bedrock knowledge bases for documentation and procedures.\n3. **Current Time**: Get the current timestamp for time-based operations.\n4. **Pipeline Management**: List and analyze Mezmo pipelines.\n5. **Filesystem**: Read configuration files and logs when needed." }, "finish_reason": "stop" }], "usage": { "prompt_tokens": 42, "completion_tokens": 118, "total_tokens": 160 } } ``` ## Testing with curl ```bash theme={null} # Health check curl -X GET http://127.0.0.1:8080/health # List available agents curl http://127.0.0.1:8080/v1/models # Chat completion (uses DEFAULT_AGENT when model is omitted) curl -X POST http://127.0.0.1:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "messages": [ {"role": "user", "content": "What tools do you have available?"} ] }' # Chat completion with a specific agent curl -X POST http://127.0.0.1:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "devops", "messages": [ {"role": "user", "content": "What tools do you have available?"} ] }' ``` ## Configuration The server uses the AURA TOML configuration system. See the [configuration reference](/aura/configuration-reference) for: * LLM provider configuration (OpenAI, Anthropic, Bedrock, Gemini, Ollama, OpenRouter) * MCP server setup (HTTP, SSE, STDIO) * Vector store and RAG integration * Agent settings and prompts Example configurations are in the [`examples/`](https://github.com/mezmo/aura/tree/main/examples) directory — see [Example Configs](/aura/example-configs). ## Architecture * **Multi-Agent Serving**: Load multiple agents from a config directory, selectable via the `model` field * **Stateless Requests**: Each HTTP request is processed independently * **Multi-Turn Support**: Conversation history passed via messages array * **OpenAI Compatible**: Request/response schemas match OpenAI's format * **Error Handling**: Proper HTTP status codes and error responses ## Deployment **Environment Variables**: | Variable | Default | Description | | -------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `CONFIG_PATH` | `config.toml` | Path to a config file or directory of configs | | `HOST` | `127.0.0.1` | Server bind address | | `PORT` | `8080` | Server port | | `AURA_ENABLE_A2A` | `false` | Enable A2A protocol endpoints (`/a2a/v1/*`, `/.well-known/agent-card.json`). Disabled by default — see [A2A Integration](/aura/a2a-implementation) | | `AURA_SERVER_URL` | host/port | Canonical public origin published in the A2A agent card. Set when running behind a proxy, load balancer, or in Kubernetes — see [A2A Integration](/aura/a2a-implementation#agent-card-url-aura_server_url) | | `DEFAULT_AGENT` | *(none)* | Agent name or alias used when `model` is omitted. Not needed when only one config is loaded. | | `AURA_CUSTOM_EVENTS` | `false` | Emit `aura.*` SSE events alongside OpenAI-compatible chunks | | `AURA_EMIT_REASONING` | `false` | Emit `aura.reasoning` events (requires `AURA_CUSTOM_EVENTS=true`) | | `TOOL_RESULT_MODE` | `none` | How tool results are streamed: `none`, `open-web-ui`, or `aura` | | `TOOL_RESULT_MAX_LENGTH` | `1000` | Truncation limit for streamed tool results (0 = no truncation) | | `STREAMING_TIMEOUT_SECS` | `900` | Max duration for a streaming request before cancellation | | `FIRST_CHUNK_TIMEOUT_SECS` | `90` | Max wait for the first LLM chunk before treating the connection as hung (0 = disabled) | | `STREAM_INACTIVITY_TIMEOUT_SECS` | `0` | Max silence between stream items after the first chunk before the server fails the streaming request (0 = disabled). Applies to single-agent streaming; tool execution is exempt. For orchestrated deployments, use the TOML `stream_inactivity_timeout_secs` instead. See [request lifecycle](/aura/request-lifecycle#timeout-configuration) for tuning guidance, and the [configuration reference](/aura/configuration-reference) for the TOML field. | | `SHUTDOWN_TIMEOUT_SECS` | `30` | Grace period for in-flight streams after SIGTERM/SIGINT | | `STREAMING_BUFFER_SIZE` | `400` | SSE chunk buffer size; higher values reduce latency but increase memory use | | `AURA_DEBUG_PROVIDER_ERRORS` | `false` | **Dev only.** Surface raw upstream provider errors to clients (capped). Keep off when public-facing — error bodies can echo request content. Raw error is always in logs/OTel. | ## See Also * [AURA on GitHub](https://github.com/mezmo/aura) — source, issues, and contributing # Mezmo Ingestion Service Limits Source: https://docs.mezmo.com/docs/Mezmo-ingestion-service-limits Reference the size limits Mezmo enforces during ingestion for body, message, metadata, hostname, app name, log level, tags, and fields. | Log or Log Line Component | Size Limit | | ------------------------- | -------------------------------------------------------------------------------------------------------- | | Body | 10MB

This is the server-enforced maximum body size. Ingestion clients may further reduce this | | Message | 16KB | | Metadata | 32KB | | Hostname length | 256 characters

Domains within hostnames are truncated. FQDN settings available upon request. | | App Name length | 512 characters | | Log Level | 80 characters | | Tags | 80 characters | | Depth of nested fields | 3 | | Number of unique fields | Typically 500 per day | # About Mezmo Log Management Source: https://docs.mezmo.com/docs/about-mezmo-log-analysis Learn how Mezmo centralizes log ingestion, parsing, exclusion rules, alerts, spike protection, and archiving on a single platform. Mezmo Log Management enables enterprises to ingest all of their log data to a single platform, optimize it through a [Telemetry Pipeline](/telemetry-pipelines/getting-started-with-mezmo-telemetry-pipeline), and route it to the appropriate teams so that they can take meaningful action in real time. * Mezmo makes it easy to centralize log data with support for a broad range of [ingestion options](/docs/ingestion) * [Automatic](/docs/log-parsing) and [Custom Parsing](/docs/parse-logs-with-custom-templates) lets you shape log data to become more actionable * Powerful [Exclusion Rules](/docs/excluding-log-lines) let you prioritize the right log data for routing; while [Variable Retention](/docs/variable-retention) allows for finer granularity in prioritizing certain log types over others * [Alerts](/docs/add-alerts-to-views) can be configured using a number of integrations, including [PagerDuty](/docs/pagerduty-alert-integration), [Slack](/docs/slack-alert-integration), and more. * With [Spike Protection features](/docs/manage-usage), you can set dynamic thresholds and alerts when data volume limits are being hit * [Archive](/docs/archiving) log data to longer term storage for compliance purposes, and use [Restoration](/docs/data-restoration) to bring log data back to the Mezmo UI Mezmo provides this level of control on a platform that is loved by customers for its speed, ease of use, and scale. The platform provides flexibility, regardless of the other tools they are using in their observability and security stacks, and gives them more control over their logging spend without sacrificing log data quality. # Access Management for Enterprise Organizations Source: https://docs.mezmo.com/docs/access-management-for-enterprise-organizations Configure sign-in policies, SAML SSO, organization discoverability, and login methods for child organizations in a Mezmo Enterprise account. This feature is only available for customers on an Enterprise plan, and is restricted to **Owner** and **Admin** user roles. Please contact your Customer Support Manager or [support@mezmo.com](mailto:support@mezmo.com) for more information. On the [Access Management](https://app.mezmo.com/enterprise/access-management) page you can configure the type of method that your child organizations will use to log in to the Mezmo Web App. ## Sign In Policy * **Local Sign-in** - Use the credentials they've created in Mezmo. * **Google Sign-in** - Log in with Google credentials. * **Other OAuth Sign-in** - Log in with credentials from other providers like GitHub or Heroku. * **SAML Sign-in** - Set a Security Access Markup Language (SAML) configuration for child organizations within the enterprise. See, [Enterprise SAML SSO](/docs/enterprise-saml-sso) * **Idle Logout** - Set a time to log out account holders after they've been inactive. * **Redirect after logout** - Set a page to redirect account holders to after they've been logged out ## Discoverability Organization discoverability lets users find and join child organizations on the same domain. If you have a Mezmo Enterprise Organization account, you can enable SSO Discoverability for your child domains. Discoverability lets users logging in using SAML, see all child accounts present in the Enterprise Organization that have discoverability enabled. Discoverability is disabled by default. This means users cannot discover child accounts, regardless of the settings made within the child accounts. To enable SSO Discoverability, navigate to the Enterprise dashboard in the Mezmo web app, then toggle the Child organization discoverability setting to the On position. After enabling SSO Discoverability, you can enable discoverability settings within each child account, which will let you control how users can find and join your organization. * **Discover** - Members on the same domain can find and ask to join the organization. Discover is enabled by default. * **Join** - Members on the same domain can join the organization without a request. Your domain is determined by the email address you used when creating your Mezmo account. If you would like to add domains, please contact Mezmo support. ## SAML Configuration Image Use SSO for all child organizations. Learn more about [Enterprise SAML SSO](/docs/enterprise-saml-sso). # Add a Breakdown to a Graph Source: https://docs.mezmo.com/docs/add-a-breakdown-to-a-graph Create histogram and pie chart breakdowns to see how aggregated plot values are distributed across fields like response codes and hosts. A breakdown is a view of your main graph. It shows the distribution of values for the aggregated plots in your graph. For example, you can create a breakdown for a graph on the request field to see the distribution of request values for the HTTP response codes you plotted in Add a Plot to a Graph. You can add different types of breakdowns depending on the kind of information you want to see, in this case you'll create a histogram breakdown. ## Create a Histogram Breakdown 1. Open the graph created in [Create a Graph](/docs/create-a-graph) and [Add a Plot to a Graph](/docs/add-a-plot-to-a-graph). 2. Expand the downward arrow under the center of the graph. 3. Select a breakdown type of **Histogram**. 4. Set field name to **Response**. 5. Click the **Add Breakdown** button. The histogram shows a breakdown of the three plotted queries and which code returns the most HTTP responses. In this example, the response code 404 returns the most. Each graph can have five breakdowns. ## Pie Chart Breakdown With a pie chart breakdown, you can see how the response codes are distributed across hosts. 1. Click **+Add.** 2. Select breakdown type **Pie.** 3. Distribute by **Fields.** 4. Set field to **Host**. Hover over each part of the pie chart to see the hosts. In this example we can see a host that has a large number of 404's compared to the other hosts. There are no 500 response codes returned. # Add a Plot to a Graph Source: https://docs.mezmo.com/docs/add-a-plot-to-a-graph Add additional data points to a graph to compare error levels and response codes using system fields and optional plot filters. You can add additional data points to a graph as a way to compare data. Using the graph created in , you'll add additional plots showing other error levels. 1. Go to the graph created in [Create a Graph](/docs/create-a-graph). 2. Click **Add Plot i**n the lower-left corner of the graph\*\*.\*\* 3. For **System Fields**, select **All Lines**. 4. For **Optional Plot Filter,** enter `response:404` to return all 404 codes. 5. Repeat steps 2 - 4 to add `response:>500` . 6. Click the **Save** button to add the new plot to graph 7. Change the line color by clicking on the colored icon beside the plot value. Next, you'll add a breakdown to show which app is creating the most error messages. # Add Alerts to Views Source: https://docs.mezmo.com/docs/add-alerts-to-views Attach presence or absence alerts to a single view, or build reusable alert presets with custom schedules to apply across multiple views. You can add an alert to a single view, or create an alert template that you can apply to multiple views. ## Add an Alert to a Single View We'll create an email alert for the 404 view created in [Create and Edit Views](/docs/create-and-edit-views). Views can have multiple alerts and multiple alert types. 1. Go to Views and select your Alert. 2. Click the Alert name and select **Attach an Alert**. 3. Under Build My Own, select **View Specific Alert**. 4. Select Email. 5. You can configure presence or absence alerts. Set the alert to **Presence**. 6. For this example, set it to **100 log lines in 15 minutes**. As you update the number of log lines and times, the graph will update to show how many lines for that time. The graph will update reflect your selections, the gray areas show the times you will not receive alerts. 7. You can send an alert at the end of 15 minutes, immediately after 100 lines, or set a custom schedule. For this example, we'll select **at the end of 15 minutes**. 8. Add you email recipients. 9. Set the timezone to send the alerts. Congratulations. You've set an alert, for the 404 View. Now you'll get an alert at the end of 15 minutes if there are 100 or more 404 errors logged. ## Create an Alert Template for Views Preset alerts are templates that can be used on multiple Views. In this example, you'll create a custom schedule for alerts that only notifies those on call Monday, Wednesday, and Friday from 8:00 AM - 5:00 PM. Then you'll assign that alert to a view. ### Create the Preset 1. Go to [Settings > Alerts](https://app.mezmo.com/manage/alerts). 2. Click **Add a Preset**. 3. Select **Email**. 4. Give the preset a name. 5. Set Presence and when 5 log lines appear within 15 minutes. 6. Send the alert at the end of 15 minutes. 7. Toggle Custom Schedule. When you set a custom schedule, alerts will only be send during that time. If your app alerts outside of that time, you will not receive an alert. 8. Select Monday, Wednesday, and Friday as the active days. 9. Restrict the time to 8:00 AM - 5:00 PM. As you update the days and time, you'll notice the graph updating. Areas shaded gray will not send alerts during that time. Image ### Assign Preset to a View 1. Go to Views. 2. Select the View you want to add the alert to. 3. Click the View name in the Viewer, then attach alert. 4. Search for the alert name. ## Tips and Troubleshooting For tips and troubleshooting information, check out [the Mezmo Log Analysis Support Knowledge Base](https://supportkb.mezmo.com/). # Add Enterprise Admins Source: https://docs.mezmo.com/docs/add-enterprise-admins Grant Super Admin privileges from the General page so users can view the Enterprise Dashboard, manage child organizations, and set sign-in options. This feature is only available for customers on an Enterprise plan, and is restricted to **Owner** and **Admin** user roles. Please contact your Customer Support Manager or [support@mezmo.com](mailto:support@mezmo.com) for more information. You can easily add administrators to your Enterprise on the **General** page. These admins will be able to view the Enterprise Dashboard, manage child organizations, and configure sign-in options. When you add a admins for your enterprise, they do not gain any admin privileges for the child organizations in your enterprise. You must be directly added to the organization as an admin to gain additional organization admin capabilities. 1. Log in to [the Mezmo Web App](http://app.mezmo.com/). 2. Select your organization from the organization switcher in the top navigation bar. 3. Click **Enterprise Dashboard**. 4. Click **General**. 5. Under **Super Admins**, click **Add**. 6. Enter the name of the user you want to grant Admin privileges to. 7. Click **Add**. # About Mezmo Agent Exclusion Rule: MEZMO Created - Exclude Agent-Generated Errors Source: https://docs.mezmo.com/docs/agent-generated-errors Understand the exclusion rule Mezmo added to filter excessive, low-value log lines caused by a bug in agent versions earlier than 3.x. Starting in January 2023, an exclusion rule was added to some accounts. Its purpose is to mitigate the impact of a bug in older versions of the Mezmo agent. Most accounts use Mezmo's Agent to send logs to Mezmo's service. The application for the agent can be installed in many different environments and is constantly monitoring for new log lines. When the agent application is running, it generates log lines of its own. In many cases, particularly with versions earlier than 3.x, its log lines are sent to the Mezmo service. If there are problems with the agent, it can be useful to see these agent-generated logs. Versions of the agent earlier than 3.x have a bug, whereby they sometimes generate a very large number of log lines, none of which contain useful information. It is rare, affecting 1-2 accounts a month. It can be avoided by upgrading to a version of the agent later than 3.x. To prevent this problem for customers using older versions of the agent, we proactively added an exclusion rule that targets these "excessive" logs sent by the agent. The rule was added only to accounts that send logs, as best we can determine, from versions of the agent earlier than 3.x. # AI Investigations Source: https://docs.mezmo.com/docs/ai-investigations Read automated root-cause investigations that Mezmo AURA produces, viewable in the Mezmo web app or through the Mezmo MCP server. An AI Investigation is an AI-generated record of a root cause analysis, produced by Mezmo AURA, Mezmo's AI agent. When Mezmo detects a problem, AURA starts an automated investigation and begins the root cause analysis (RCA) on its own, so the analysis is already underway before anyone is paged. Mezmo triggers these investigations internally rather than having you create them, and you read the finished record in the Mezmo web app or through the Mezmo MCP server. ## The Investigation Lifecycle An investigation moves through three states in order: * `triggered`: Mezmo has started the investigation in response to a trigger condition, and AURA is about to begin. * `investigating`: AURA is actively gathering evidence and working toward a root cause. The record updates as the analysis proceeds. * `completed`: AURA has finished. The record now holds the full findings, including the root cause and a suggested resolution. ## What an Investigation Contains A completed investigation record includes: * The trigger source that started the investigation. * The linked alert that fired. * The evidence and trigger conditions AURA evaluated. * A confidence score. * A root cause. * A suggested resolution with remediation steps. * A timeline of events. * Any linked investigations. ## Before You Begin You can view AI Investigations if your account has AI Investigations enabled and your role grants the investigation read permission. Manage roles and permissions from your account's [Roles and Permissions](/docs/rbac) page. If either condition is not met, the AI Investigations entry does not appear in the sidebar. If AI Investigations is not enabled for your account, contact Mezmo to have it turned on. ## How Investigations Are Triggered You do not create an investigation directly. Mezmo starts one for you when a Pipeline node alert is configured to notify AURA. A node alert is an in-stream alert set on a source or processor. Configure a node alert (see [Set In-Stream Alerts for Pipeline Nodes](/telemetry-pipelines/set-in-stream-alerts)), then select **Mezmo AURA Investigation** as its notification service. This service appears as a notification target once AI Investigations is enabled for your account. When that alert fires, it starts an AURA investigation. To confirm the trigger is working, watch for a new card in the [AI Investigations list](#view-ai-investigations-in-mezmo) in the `triggered` state after the alert fires. ## View AI Investigations in Mezmo Open the **AI Investigations** entry in the sidebar. It carries a badge counting the investigations that are currently active. The list page shows one card per investigation. Each card displays the investigation's status, either **Triggered**, **Investigating**, or **Completed**, along with a confidence score, the trigger source, and a summary of the suggested resolution. You can filter the list by status, search it, and page through the results. The list also refreshes automatically as new investigations arrive. Select a card to open the detail page. It shows the fields described in [What an Investigation Contains](#what-an-investigation-contains), plus a live in-progress indicator while the investigation is still running and a plain-language summary. ## Review Investigations With Mezmo MCP The Mezmo MCP server exposes two read-only tools for investigations. Use `list_ai_investigations` to list existing investigations, filtered by state and paginated. Use `get_ai_investigation` to retrieve a single investigation by its id. Both tools read investigations that already exist and do not create them. For setup and the full tool reference, see [Mezmo MCP](/docs/mezmo-mcp). ## Related * [Mezmo MCP](/docs/mezmo-mcp): Connect an AI assistant or IDE chat agent to Mezmo and review investigations from your tools. * [Set In-Stream Alerts for Pipeline Nodes](/telemetry-pipelines/set-in-stream-alerts): Configure the Pipeline node alert used to trigger AURA investigations. * [Roles and Permissions](/docs/rbac): Manage who can access AI Investigations. # Mezmo API Source: https://docs.mezmo.com/docs/api API access to Mezmo The Mezmo API gives you programmatic control over the entire Mezmo platform — from sending and querying log data to building and managing telemetry pipelines to administering your organization at scale. Use it to ingest logs and metrics, configure processors and pipeline nodes for real-time data transformation, set up alerts and exclusion rules to keep your data clean and actionable, export or archive logs for long-term storage and compliance, and manage accounts, members, and API keys across your enterprise. Whether you're automating pipeline deployments, integrating Mezmo into your own tooling, or building custom dashboards on top of your telemetry data, this reference covers every endpoint you need to get started. Download the [API Spec as OpenAPI](/apis/combined-mezmo-api.yaml). # Authenticating With The API When making API requests, it is crucial to authenticate them to ensure secure communication and proper authorization. Mezmo's API utilizes a key-based authentication system. This method involves including a unique access key in the Authorization header of your request, prefixed with the `Token` scheme. ### Quick start Add the Authorization header - Format: `Authorization: Token ` ```bash theme={null} export $MZM_ACCESS_KEY = 'sta_1a2b3c4d5e6f7890abcdef1234567890abcdef12' curl -H "Authorization: Token $MZM_ACCESS_KEY" \ -H "Accept: application/json" \ https://api.mezmo.com/v3/pipeline ``` ```javascript theme={null} const res = await fetch('https://api.mezmo.com/v3/auth/token>', { headers: { 'Authorization': `Token ${process.env.MZM_ACCESS_KEY}`, 'Accept': 'application/json' } }) ``` For enterprise operations, additional, specific account context may be included for delegation as needed. 1. `x-delegate-account-id: ` (enterprise tokens only) ```bash theme={null} export MZM_ENTERPRISE_KEY = 'ste_d710b57dc7dedde45ccc4c35b68cf385b89f8dc3' curl -H "Authorization: Token $MZM_ENTERPRISE_KEY" \ -H "x-delegate-account-id: 507f1f77bc" \ https://api.mezmo.com/v3/pipeline ``` *** ### Key Considerations Choose the Appropriate Access Key Type: Mezmo offers different token types, each designed for specific use cases and levels of access. Carefully select the token type that best aligns with the requirements of your application or integration. This might include: 1. **Service Accounts**: Often used for machine-to-machine communication, background services, or applications that require persistent access. Service accounts have only single access key associated with it 1. **Enterprise Service Accounts**: Similar to a standard service account with the exception that these access keys are not bound to an individual account and may be used to manage an enterprise programmatically, or interface with any of the child accounts associated with an enterprise organization. 2. **Personal Access Keys**: Associated with a specific user with in the organization which may have a shorter lifespan. These keys inherently have the level of access as the user it is associated with. Changing the permissions granted to a user changes the scope of any access keys they may have provisioned. 1. Scoped Access: Additionally, personal access keys may be created with a limited level of access to further restrict what they may be used for. #### Examples ```bash theme={null} export MZM_ACCESS_KEY = 'sta_34b8f6897cd3396e6af781c3bfe34065b690f90b' export MZM_ENTERPRISE_KEY = 'ste_d710b57dc7dedde45ccc4c35b68cf385b89f8dc3' # Account‑scoped curl -H "Authorization: Token $MZM_ACCESS_KEY" \ https://api.mezmo.com/v3/pipeline # Enterprise‑scoped curl -H "Authorization: Token $MZM_ENTERPRISE_KEY" \ https://api.mezmo.com/v3/enterprise/account # Delegation (enterprise keys only) curl -H "Authorization: Token $MZM_ENTERPRISE_KEY" \ -H "x-delegate-account-id: 507f1f77bc" \ https://api.mezmo.com/v3/pipeline ``` ### Security best practices * Rotate tokens regularly. Use expirations and rotate before they expire. * Grant only what you need. Prefer minimal scopes. * Use service accounts for automation. Avoid personal tokens in CI/CD. * Store tokens in environment variables or a secret manager. Do not hard‑code tokens. * Implement a mechanism to quickly revoke tokens if they are suspected of being compromised or are no longer needed. ```bash theme={null} # Good export MZM_ACCESS_KEY="sta_34b8f6897cd3396e6af781c3bfe34065b690f90b" curl -H "Authorization: Token $MZM_ACCESS_KEY" https://api.mezmo.com/v3/pipeline # Avoid curl -H "Authorization: Token sta_34b8f6897cd3396e6af781c3bfe34065b690f90b" \ https://api.mezmo.com/v3/pipeline ``` *** ### Troubleshooting Common HTTP codes | Code | Meaning | What to do | | ----- | ------------------------ | ------------------------------------------------------ | | `401` | Token invalid or expired | Check token format and expiration. Re‑issue if needed. | | `403` | Insufficient permissions | Verify the token has the required scopes. | | `404` | Resource not found | Check the endpoint URL and resource IDs. | Quick validation **Expect 200 (valid) or 401 (invalid)** ```bash theme={null} export MZM_ACCESS_KEY = 'sta_34b8f6897cd3396e6af781c3bfe34065b690f90b' curl -I -H "Authorization: Token $MZM_ACCESS_KEY" https://api.mezmo.com/v3/pipeline ``` Debug headers Responses from the API will include additional metadata in the response headers describing what the system understood about the subject making the request. The can be identified with a common prefix - `x-auth-*` ```bash theme={null} export MZM_ACCESS_KEY = 'sta_34b8f6897cd3396e6af781c3bfe34065b690f90b' curl -v -H "Authorization: Token $MZM_ACCESS_KEY" https://api.mezmo.com/v3/pipeline # Inspect x-auth-* headers in the response ``` ```http theme={null} HTTP/1.1 200 OK content-type: application/json; charset=utf-8 x-request-id: 0e87e821-c93c-4642-a69c-8ac40478239e x-auth-subject-id: 507f1f77bcf86cd799439015 x-auth-account-id: 507f1f77bcf86cd799439012 x-auth-enterprise-id: 68c584136b9b08b8fc53a85e x-auth-subject-email: user@example.com x-auth-account-short-id: d3396ea1f x-auth-access-type: access ``` # AppOptics Alert Integration Source: https://docs.mezmo.com/docs/appoptics-alert-integration Connect Mezmo alerts to the AppOptics Measurement API using a Record Only API token so triggered alerts create measurements in AppOptics. Mezmo’s alert integration with AppOptics interfaces with the [App Optics Measurement API](https://docs.appoptics.com/api/#measurements) and allows alerts set in Mezmo to create measurements in AppOptics. ## Integrating with AppOptics On the alerts creation page, select the AppOptics logo. To find an API token, in your AppOptics dashboard, go to [Settings → API Tokens ](https://my.appoptics.com/organization/tokens)page to copy an existing or create a new API token with “Record Only” permission. Remember to save your alert so that your alerts will now trigger in AppOptics and new alerts will be in yourAppOptics dashboard. Specifically, new alerts can be found by navigating to: AppOptics -> Dashboards and Metrics -> mezmo.alerts. ## Testing your AppOptics alert integration Before saving your AppOptics alert, you can also trigger a test alert with test data. As long as you have pasted a valid API token from AppOptics, you will be able to create a test alert. # Archive Logs Source: https://docs.mezmo.com/docs/archiving Automatically export retained logs hourly to external storage as compressed JSON, with details on file naming, encryption, and restoration support. Archiving is an automated function that exports your logs from Mezmo to an external storage provider. Once archiving is configured for your account, your logs will be exported to the storage provider on an hourly basis in a compressed JSON format `.json.gz`, with the associated metadata for each line preserved. * Only retained logs will be archived, meaning, for example, logs affected by exclusion rules would not be archived * The first time you configure archiving, your archived logs will typically appear within 12-24 hours, but will not include existing logs from the period before you enabled archiving Mezmo's [Data Restoration](/docs/data-restoration) feature only supports restoring logs from [AWS S3 and Google Cloud Storage](/docs/export-logs-to-external-storage) archives. ## Hourly Archiving * Hourly archives create 24+ Gzip JSON files per day. If there are no logs for an hour, then no files will be uploaded for that hour. * Archives are expected to appear within 24 hours, but may take up to 72 hours for larger archives * The file contents will stay the same as the daily archives, except now the file will be stored as `year=YYYY/month=MM/day=DD/.--
.00.json.gz` ,where `HH` is hours in 24 hour format, for all providers. * If log lines are attributed or received 6 hours beyond the hour bucket the log line belongs to, subsequent archive files will be created in the name format of `.--
.00..json.gz`, where `` is an incrementing number starting from`1`, to prevent filename conflicts. * Hourly archiving may create duplication of logs in the storage (1% of the time). You can tell if a line has been duplicated if the lines have the same log line ID. ## Security By default, Mezmo encrypts your archived data in transit, and requests server-side encryption where possible, including using x-amz-server-side-encryption upon upload of logs to S3. ## Related Topics # Azure Entra ID SAML Setup Source: https://docs.mezmo.com/docs/azure-saml Configure SAML single sign-on between Mezmo and Azure Entra ID by exchanging sign-on URLs and uploading the federation metadata XML. This feature is only available for customers on an Enterprise plan, and is restricted to **Owner** and **Admin** user roles. Please contact your Customer Support Manager or [support@mezmo.com](mailto:support@mezmo.com) for more information. ## Step 1: Get Your Mezmo Single Sign On URL 1. In your Mezmo app go to **Settings > Organization > Access Management**. 2. Go to **SAML Configuration** and copy the URL under **Single Sign On URL**. 3. Enable SAML Sign-in 4. Keep this URL available since it will be used in **Step 2: Configure Azure**. ## Step 2: Configure Azure 1. In Azure Portal go to **Enterprise Applications** and then into your **Mezmo SCIM** (if you've created one) or **Create your own application** 2. Click **Set up single sign on**. 3. Under **Basic SAML Configuration** click Edit and enter the following values (replacing 123456789 with your account id 1. **Identifier (Entity ID):** `logdna-saml/123456789` 2. **Reply URL (Assertion Consumer Service URL):** `https://app.logdna.com/auth/saml-consume/123456789` 3. **Sign on URL:** `https://app.logdna.com/auth/saml/123456789` 4. Click Save ## Step 3: Update Mezmo SAML Configuration 1. From section 4 in the Azure setup, please copy the value from Login URL: 2. Paste this value in Mezmo's **Identity provider sign-in URL** field. 3. From section 3 in Azure, download the Federation Metadata XML 4. Drop this file in Mezmo's **Drop metadata.xml here to begin** 5. Finally click **Save Config** ## Finally Setup should now be complete. You can now use the URL you copied in Step 1 to log into Mezmo through Azure Entra ID # Billing Procedures Source: https://docs.mezmo.com/docs/billing-procedures Understand how Mezmo handles failed payments on self-service paid plans, including retry timing, paywalls, and account archiving. ## Resolving Failed Payments For self-service accounts on paid plans where a credit card on file is no longer valid, Mezmo follows this procedure. 1. Attempt billing. If failed, go to next step. 2. Wait 1 day. Retry billing. If failed, go to next step. 3. Wait 5 days. Retry billing. If failed, go to next step. 4. Wait 7 days. Retry billing. If failed, paywall is implemented on the account. Go to next step. 5. Wait until the end of the month. Retry billing. If failed, the account will be archived. Data ingested will no longer be saved. Enterprise accounts are not subject to automatic downgrades. If your Enterprise plan has a failed payment, contact your account representative or Mezmo support to resolve the issue. # Browser Logger Template Source: https://docs.mezmo.com/docs/browser-logger-template Use pre-configured Views, Boards, and Screens to analyze browser-side errors by message, OS, and browser version with the Mezmo Browser Logger. Use the [Mezmo Browser Logger integration](https://github.com/logdna/logdna-browser) combined with this Template to get trends and break down browser-side errors by error message, OS, browser version and more. The Browser Logger Template consists of a collection of pre-configured Views, Boards, and Screens to help you gain insights from your log data. Browse the[ full library of Mezmo Templates](https://app.mezmo.com/manage/template-library). ### Views Views are saved shortcuts to a specific set of filters and search queries. You can also add Alerts to views to notify you when specific conditions are met. Check out the topic [Add Alerts to Views](/docs/add-alerts-to-views) for more information. * **All Messages** - View all messages sent by the Browser Logger * **Errors** - All error-level logs captured by the Browser Logger, including uncaught exceptions ### Boards Boards are collections of graphs. Using boards, you can track trends with response codes and understand how they fluctuate over time at a glance. Drill down using subplots to see which host or path is generating the most errors. Check out the topic [Visualize Log Data with Graphs](/docs/visualize-log-data-with-graphs) for more information. * **Analytics** - View trends of errors over time, expand Subplots to view breakdowns by OS, browser version, error message. Use Board Filters to target a specific error. ### Screens Screens are collections of customized dashboards that can display data in various forms. See the topic [Use Screens and Widgets to Monitor Log Data](/docs/use-screens-and-widgets-to-monitor-log-data) for more information. * **Error Analytics** - Get top error messages, week-over-week error trends, and more # Mezmo Client-Side Logger Source: https://docs.mezmo.com/docs/client-side-logging Use Mezmo's Node.js library to send client-side JavaScript logs directly to Mezmo with multiple log levels, metadata, and tags. You can now use Mezmo’s Node.js library (version 3.0.1+) to log your client-side JavaScript applications to Mezmo. The library sends logs directly from your end users to Mezmo’s ingestion servers, so you can collect vital operational and debugging information without having to print logs to the console. The library offers a complete feature set including multiple log levels, customizable metadata, and the ability to add tags. ## GitHub Repository For complete information about setting up and using the Mezmo Client-Side Logger, check out the source code and documentation in our GitHub repository. [https://github.com/logdna/logger-node](https://github.com/logdna/logger-node) # Code Libraries Source: https://docs.mezmo.com/docs/code-libraries Find official and community logging libraries for sending logs to Mezmo across languages like Go, Node.js, Python, Ruby, Java, and PHP. A logging code library (or logging framework) is code that you embed into your application to create and manage log events. To understand more about code libraries and Mezmo, read [our blog about libraries versus agents](https://mezmo.com/blog/logging-agents-vs-logging-libraries-which-should-you-use/). For instructions on how to set up the code libraries listed on this page, follow the link to their corresponding GitHub repositories in these tables ## Official Code Libraries These libraries are maintained by Mezmo. | Library | GitHub Link | Notes | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Go | [https://github.com/logdna/logdna-go/blob/master/README.md](https://github.com/logdna/logdna-go/blob/master/README.md) | | | JavaScript for Browsers | [https://github.com/logdna/logdna-browser/blob/main/README.md](https://github.com/logdna/logdna-browser/blob/main/README.md) | The LogDNA Browser Logger includes templates to create out-of-the-box Views, Boards, and Screens to analyze your browser logs. After installing the browser logger component, you can [download and import the Template](http://app.logdna.com/manage/import-config?appcue=34863e84-adb0-4ab3-a0dc-7fb47530d47f). | | Node.js | [https://github.com/logdna/logger-node/blob/main/README.md](https://github.com/logdna/logger-node/blob/main/README.md) | | | Pino | [https://github.com/logdna/pino-logdna/blob/main/README.md](https://github.com/logdna/pino-logdna/blob/main/README.md) | The `pino-logdna` "transport" tool was built to facilitate Pino-based logging from Node.js applications.

You can also find pino-logdna listed in [the Transports section of the official Pino site](https://getpino.io/#/docs/transports?id=pino-logdna). | | Python | [https://github.com/logdna/python/blob/master/README.md](https://github.com/logdna/python/blob/master/README.md) | | | Ruby | [https://github.com/logdna/ruby/blob/master/README.md](https://github.com/logdna/ruby/blob/master/README.md) | | | Rust | [https://github.com/logdna/logdna-rust/blob/main/README.md](https://github.com/logdna/logdna-rust/blob/main/README.md) | | ## Community Code Libraries and Integrations | Library | GitHub Repo | Notes | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----- | | Android | [https://github.com/mazenrashed/LogDNA-Android-Client/blob/master/README.md](https://github.com/mazenrashed/LogDNA-Android-Client/blob/master/README.md) | | | Elixr | [https://gist.github.com/timsu/9d23ccda7e05495f764a9b13a9f6f635](https://gist.github.com/timsu/9d23ccda7e05495f764a9b13a9f6f635) | | | Fastly | [https://docs.fastly.com/en/guides/log-streaming-logdna](https://docs.fastly.com/en/guides/log-streaming-logdna) | | | Flutter | [https://github.com/aligorithm/logdna-flutter/blob/master/README.md](https://github.com/aligorithm/logdna_flutter/blob/master/README.md) | | | Go-lang | [https://github.com/evalphobia/go-logdna/blob/master/README.md](https://github.com/evalphobia/go-logdna/blob/master/README.md)

[https://github.com/ctrlrsf/logdna/blob/master/README.md](https://github.com/ctrlrsf/logdna/blob/master/README.md) | | | Go-logrus | [https://github.com/drdaeman/logdna-logrus/blob/master/README.md](https://github.com/drdaeman/logdna-logrus/blob/master/README.md) | | | iOS | [https://github.com/iosphere/ISHLogDNA/blob/master/README.md](https://github.com/iosphere/ISHLogDNA/blob/master/README.md) | | | Java | [https://github.com/robshep/logback-logdna/blob/master/README.md](https://github.com/robshep/logback-logdna/blob/master/README.md) | | | Log4J | [https://github.com/trygve-lie/abslog/blob/master/README.md](https://github.com/trygve-lie/abslog/blob/master/README.md) | | | LogStash | [https://github.com/logdna/logstash-output-logdna/blob/main/README.md](https://github.com/logdna/logstash-output-logdna/blob/main/README.md) | | | .Net 2.0 | [https://github.com/RedBearSys/RedBear.LogDNA/blob/master/README.md](https://github.com/RedBearSys/RedBear.LogDNA/blob/master/README.md) | | | .Net 2.0/4.0 + Log4Net | [https://github.com/landbird111/log4net-logdna/blob/master/README.md](https://github.com/landbird111/log4net-logdna/blob/master/README.md) | | | PHP | [https://github.com/nvanheuverzwijn/monolog-logdna/blob/master/README.md](https://github.com/nvanheuverzwijn/monolog-logdna/blob/master/README.md) | | | WIX Adapter | [https://github.com/braxtonj/wix-logdna-adapter/blob/main/README.md](https://github.com/braxtonj/wix_logdna_adapter/blob/main/README.md) | | # Mezmo Architecture Overview Source: https://docs.mezmo.com/docs/concepts Understand Mezmo's microservice architecture, including ingestion services like Buzzsaw and the retrieval services that power the Web App and API. Mezmo uses a microservice-based architecture to split different tasks into discrete and scalable units. These microservices can be organized into two roles: log ingestion, and log retrieval. Logs sent from your log sources to Mezmo are received by one of many ingestion endpoints. These endpoints route each log to a message queue using a proprietary data pipeline. The message queues manage the delivery of logs to a number of worker pools, which process logs for use in various Mezmo services such as parsing, indexing, live tail, graphs, and alerts. Other microservices provide features such as hosting API endpoints, providing security and authentication, and maintaining log data stores. ## Ingestion Services Ingestion services provide the backend components necessary to ingest, parse, index, and store logs. They receive logs sent by log-generating components, as well as fulfill requests for log data sent from other Mezmo services. Logs arrive into the Mezmo infrastructure through ingestion endpoints, which route logs from your applications and systems to a proprietary message brokering service called Buzzsaw. Buzzsaw is a highly optimized and highly scalable brokering service designed specifically for log data. Buzzsaw routes logs to worker pools, which consist of microservices performing specific actions such as parsing, generating graphs, and running alerts. It also sends logs to a cluster of nodes, which provides Mezmo's search and filtering functionality. Buzzsaw acts as a buffer between ingestion sources, worker pools, and nodes, ensuring that performance problems in any one service don't impact the performance of other services. ## Retrieval Services Retrieval services provide the ability to access, view, and export log data. They act as the gateway between users and the Mezmo ingestion services and are responsible for actions such as responding to user requests, querying for saved log data, and live tailing logs. The primary services provided by retrieval services are the user interfaces, which include the Mezmo Web App and REST API. Actions performed in the Mezmo Web App, command line interface (CLI), or REST API are received by these services and proxied to the relevant infrastructure services before returning the results to the user. These services also perform user authentication and enforce access controls. Other retrieval services include features that export log data automatically, such as alerts and archives. # Create a Graph Source: https://docs.mezmo.com/docs/create-a-graph Build a graph in the Mezmo Web App to aggregate metrics about your log data over time, such as counts of HTTP response codes. You can use graphs to aggregate metrics about your log data over time. In this topic, and the topics on adding plots and breakdowns, you'll learn how to create a graph of HTTP response codes, with breakdowns to show which apps are sending the response codes. You can also add a graph to an existing board without having to create it from scratch. ## Create the Graph 1. In the Mezmo Web App, click the **Boards** icon in the left-hand menu. 2. Click **New Board.** 3. Click **Add a Graph**. 4. Enter `response:(>=200 <300)` into **Graph log line counts**. This will return all lines that contain a response code greater than or equal to 200 and less than 300. 5. Click **Add Graph**. You should see a graph showing the count for HTTP response codes over the selected time period. Next you'll add plots to show other response codes. Next, you'll add plots showing other response codes, then add a breakdown to analyze which apps are creating the response codes. # Create a Parsing Template Source: https://docs.mezmo.com/docs/create-a-parsing-template Parse ingested log lines into fields using extractors, delimiters, trim, concatenate, and convert operations, then validate and activate the template. Must be an [admin or owner](/docs/rbac) to use this feature. You'll learn how to parse ingested log lines. This will cover the parsing screen, the parsing mini map, and using some of the parsing functions available. You can parse all four values or parse one and move onto Validate Templates. You'll use the following log lines throughout and you'll parse: * `ip_address: 111.00.11.10` * `timestamp: 14/Nov/2018:10:35:00 +0000` * `response: 200` * `upstream_length_time: 49/0.008` ```bash theme={null} 111.00.11.10 - [111.00.111.10] - - [14/Nov/2018:10:35:00 +0000] "GET /logdna_test/v1/health HTTP/1.1" 200 44 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; AppInsights)" 586 0.004 [logdna_test-service-80] 111.00.11.10:80 49 0.008 200 230abc56545logdna1238e ``` ## Parse a String You are going to parse `111.00.11.10` from the log line. 1. Go to [Parsing Templates](https://app.mezmo.com/manage/custom-parsing/templates) and click **Create a Template**. 2. In Choose a Log Line, select **Add my own log line**. You'll be using the log line from the introduction. 3. Click **Build a Parsing Template**. You will see the line you entered as a Reference Line. 4. First, you are going to break the text down into smaller parts to so you can use the part you want. In Choose an Extractor, select **Extract Value By Delimiter**. 5. Enter a space and then a dash. 6. Now you should see `111.00.11.10` and `*[111.00.111.10]` as part of the lines parsed. 7. Select `111.00.11.10` and choose the operator,**Capture in Field.** Give a label `ip_address`. 8. The result is shown at the bottom of the parsing page. Image ## Parse a Timestamp You'll parse `14/Nov/2018:10:35:00 +0000` from the log line. 1. Select the circle with the plus sign to create a **Sibling Operator.** When you create a sibling operator, the mini map updates to show the relationship between operators and the template screen updates to have a new Choose an Operator. Learn more in [Edit Parsing Operations with the Mini-Map](/docs/edit-parsing-operations-with-the-mini-map). 2. Check the longer output that includes the timestamp. 3. Choose an operator > **Extract by Delimiter.** 4. Enter a space into the delimiter. 5. Notice the output has split everything by space, including part of timestamp. To fix the timestamp, you need to preserve some of the spaces. 6. Click **Preserve delimiters between.** 7. Start use a left square bracket `[` and end use a right square bracket, `]` . 8. Click **Preserve delimiters between** again\*\*.\*\* Use double quotes `"` for both start and end. You'll notice that the timestamp is now cleaned up, along with some of other output. 9. Now we need to remove the brackets from the timestamp, so it's easier to run diagnosis against. Select the timestamp. `[14/Nov/2018:10:35:00 +0000]` , choose operator, T**rim Value**. 10. Trim Value is 0 based counting. Start, enter `1` , for end enter `-1`. 11. Your output should be the timestamp. 12. Choose operator > **Capture in Field** and label it `timestamp` . 13. So far, you've captured two fields from the log line. Image ## Parse a Number You are going to parse `200` from the log line. 1. Using the mini map, select Trim Value. Trim Value is orange, you can also hover over the icons in the mini map. Using the mini map lets you jump between parsed areas. By starting from Trim Value, you can start from a place where the 200 is already separated from the other values, making it easier to use. 2. Click **Add a Sibling Operator**. 3. Select 200. 4. Choose an operator > **Convert to Number.** 5. Choose an operator > **Capture in Field**. Field name is `response.` Image ## Concatenate Log Lines You are going to parse `49` and `0.008` from the log line. 1. Create a sibling operator to either Trim Value or Convert to Number in the mini map. 2. Select `49` and `0.08` from the output. 3. Choose an operator > **Concatenate Values by Delimiter**. 4. Enter a forward slash `\` as the value. 5. Choose an operator > **Capture in Field**. 6. The field name is `upstream_length_time.` Image ## Validate Template Before you can make a template active, you must check that the log lines you want are working. 1. Add a log line to test against in **Add a Line**. You can use the example line. When testing you want to be sure to test multiple lines by adding lines. 2. You can also test by adding a query to `Apply this parsing template to sample lines matching this query:`. This step is required and is the scope of the parsing template. 3. Mark the log lines as valid or invalid. 1. If a line is marked as invalid you'll be taken back to the Parsing Template step. 4. Activate changes to apply your changes. Active parsing templates are only applied to the lines that come in after the template has been enabled. All log lines that were ingested prior to the template becoming active are not parsed by the parsing template. ## Results The log lines the template applies to will show in the Log Viewer. # Create and Edit Views Source: https://docs.mezmo.com/docs/create-and-edit-views Save filters and search queries as reusable Views, organize them by category, and attach alerts to monitor specific log conditions. Views are saved shortcuts to a specific set of filters and search queries for log lines. You can also [Add Alerts to Views](/docs/add-alerts-to-views) to notify you when specific conditions are met. Mezmo provides alert integrations for several messaging and notification platforms, including [Slack](/docs/slack-alert-integration), [PagerDuty](/docs/pagerduty-alert-integration), and [DataDog](/docs/datadog-alert-integration), and [Templates for Views, Boards, and Screens](/docs/using-templates) for a variety of log data types. The first time you visit Views, you'll see the default **Everything** view, which shows all log lines. ## Create a View For an example of how to create a new screen, you can create a view that shows 404 errors. 1. Select [View](https://app2.logdna.com/logs/view) in your app. 2. In the [Search](/docs/search-and-filter), enter `response:404 request:*`. This will return any 404 response from your web app. 3. You can check the query by selecting a log line and expanding the information. Image 4. Notice that your view has changed from **Everything** to **Unsaved View.** 5. Click **Unsaved View > Save as new view.** 6. Name your view. 7. Select an existing [category](/docs/organize-visualizations-by-category) or add a new one. Categories let you organize Views. You can create new Views and add them to the same Category or create a new one. 8. Select an existing alert or add a new alert. 9. After saving, you'll see your View saved in the menu, in the category you created. # Create Index Rate Alerts Source: https://docs.mezmo.com/docs/create-index-rate-alerts Set max lines/s and z-score thresholds to alert via Slack, email, webhook, or PagerDuty when your log index rate grows unexpectedly. ## Create Alerts 1. Go the [Settings > Organization > Usage > Index Rate Alerts](https://app.mezmo.com/manage/index-rate-alerts). 2. Set the **Max lines/s** and **Max z-score**. You can set one or both. 3. Set the Threshold Alerts. You can send alerts for each threshold or when both thresholds are reached. 4. Alert on each threshold separately - You will get an alert indicating whichever threshold is crossed first, either Max lines/s or Max z-score, or both. 5. Alert only when both thresholds have been exceeded - Mezmo will send one alert (per channel) every 60 minutes/24 hours. The alert will say what thresholds were crossed. 6. Decide how you want to get alerts. Slack, email, webhook or Pager Duty. 7. Set the frequency of the alerts. Alerts will be sent hourly or daily until the index rate is below the threshold. You can also set a custom alert schedule. ## Alert Message The alert message includes information about the top 20 sources and apps that have had the largest index rate growth in the past 2 hours and a list of the 20 newest sources added. The alert also includes a link to download generated lists of all applications and sources. You can view the download the list from Usage in the app. # Restore Log Data Source: https://docs.mezmo.com/docs/data-restoration Re-ingest archived logs from AWS S3 or Google Cloud Storage cold storage so you can search and filter older log data in the Mezmo UI. Mezmo's Log Data Restoration feature provides a way to re-ingest, or restore, archived logs from cold storage so you can [search and filter](/docs/search-and-filter) logs and in the Mezmo user interface. Restoration is useful for troubleshooting older bug tickets, as well as bringing up additional context from older logs beyond your retention period. ## Feature Notes * Your account must have [log archiving](/docs/archiving) enabled to use the Log Data Restoration feature * Mezmo supports restoration from [AWS S3 and Google Cloud Storage ](/docs/export-logs-to-external-storage)archives * On [AWS S3](/docs/export-logs-to-external-storage), object ownership needs to be set to [object writer](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-ownership-new-bucket.html#:~:text=Object%20writer%20\(default\)) * The Mezmo system requires **read** access to the stored logs * By default, each restoration task will be ingested for the same period of time as your normal logs * Data Restoration is included in [ the Pro, and Enterprise plans ](https://www.mezmo.com/pricing) ## Create a New Log Data Restoration Task 1. Log in to [the Mezmo Web App](https://app.mezmo.com). 2. In the left-hand navigation, go to **Settings > Archiving > Log Data Restoration**. 3. Click **New Restoration Task**. Here you can name your task, select a time range for logs, and select the exact files to restore from that time period. 4. Click **Start** to begin the process of restoring logs. Depending on the size of the restored logs, the task may take anywhere from 15 minutes to up to two hours. ## View Restored Log Data When your restoration task is complete, you can view and [search log content](/docs/searching-log-contents) in the same way as you would logs that were directly ingested. 1. Log in to [the Mezmo Web App](https://app.mezmo.com/). 2. In the left-hand navigation, click the **Restored Tasks** icon. 3. Select a **Log Restoration Task** to view and search its contents. # Datadog Alert Integration Source: https://docs.mezmo.com/docs/datadog-alert-integration Send Mezmo alerts to Datadog using the Datadog Metrics API by providing your API key and match and time-span criteria. Mezmo's Datadog integration interacts with the [Datadog Metrics API](https://docs.datadoghq.com/api/?lang=bash#post-timeseries-points). Provide your Datadog API key and input criteria on the number of matches and time span before sending an alert to your Datadog instance: After integrating Mezmo with Datadog, you can now view your metrics in Datadog's metric summary! The metric name will match the name of the view to which you attached an alert. # Edit Parsing Operations with the Mini-Map Source: https://docs.mezmo.com/docs/edit-parsing-operations-with-the-mini-map Use the mini map's rule paths and nodes to navigate, inspect, and jump between operations in a Custom Parsing Template or Extract Fields. When using the Custom Parsing Template or Extract Fields, it includes a mini map. The mini map shows what operators have been used and how they relate to each other. Image The mini map is made of **Rule Paths** that show the chain of operations used to parse the data. The Rule Paths are end in **Nodes** that indicate the operation performed. The mini map can help you navigate complicated parsing structures. Hover over each node to see the operation. To jump to the operation, click each node. Nodes that are side by side are considered siblings and inherit from the parent node. # Email Alert Integration Source: https://docs.mezmo.com/docs/email-alert-integration Trigger presence and absence alerts from log lines and send emails to multiple recipients in your preferred timezone. Email alert integration allows a user to trigger alerts and send emails based off of log lines. Email integration has options for test, presence and absence alerting. The user can select a preferred timezone to receive email alerts and can also provide multiple email recipients. A sample email that showcases email integration with Mezmo: # Enterprise SAML SSO Source: https://docs.mezmo.com/docs/enterprise-saml-sso Set up SAML single sign-on for your Mezmo Enterprise organization through manual configuration or service provider metadata. This feature is only available for customers on an Enterprise plan, and is restricted to **Owner** and **Admin** user roles. Please contact your Customer Support Manager or [support@mezmo.com](mailto:support@mezmo.com) for more information. ## Setup SAML Login 1. Go to **Settings > Organization >** **Access Management**. 2. Toggle **SAML Sign-in** to on. 3. You can either configure SAML manually or upload the XML service provider metadata. ## Setting the URL There are two URLs for SAML configuration: * Single Sign On URL * URL for the Single Sign On Service to Consume or SAML Assertion Consumer Service (ACS) URL Your IDP will specify which URL is needed. You only need one of the URLs. ## Manual Configuration If your IDP doesn't offer an XML file for download, you can enter the information manually. You'll need to provide: * **Identity provider sign-in URL** - This URL users use when they enter their domain email address. * **X.509 certificate** - The security `.pem` file. * You'll also need to enter Mezmo information into your provider. * **IdP Entity ID -** Set this to `https://app.mezmo.com` or the domain you are using for the app. This value is also known as Identifier. * **SP Entity ID -** Set this value to `logdna-saml/`. This value is also known as IdP Audience or Audience URI. * **ACS URL -** Set this value to`https://app.mezmo.com/auth/saml-consume/`. This value is also known as Sign-On URL. ## Provider Instructions * [Okta SAML Setup](/docs/okta-saml-setup) * [OneLogin SAML Setup](/docs/onelogin-saml-setup) ## Service Provider Metadata Most service providers will have an XML data file containing the SAML information. The file should include the following: * EntityDescriptor * IDPSSODescriptor * KeyDescriptor * KeyInfo * X509Certificate * NameIDFormat * SingleSignOnService # Exclude Log Data Source: https://docs.mezmo.com/docs/excluding-log-lines Use Exclusion Rules to filter out unneeded log data by source, app, or query to manage storage costs and reduce usage. You can use Exclusion Rules to filter out log data that you don’t need to store, which can help you manage storage costs and focus your analysis on data that contains meaningful information. Log data that matches an Exclusion Rule is not saved. Be careful when setting exclusion rules that you don't create rules that are too strict, and may result in the loss of data you need. ## Feature Notes * You can filter log data by source, app, or specific queries * Log data that is filtered out with Exclusion Rules will not count toward your usage * Any Admin or organization member can create exclusion rules, and can add as many as needed * Exclusion rules will begin filtering log data within a few minutes of being saved ## Validate Exclusion Rules To validate that your Exclusion Rule is filtering log data as expected before saving it, [create a view](/docs/create-and-edit-views) the same criteria as the rule and save it. In the view you should see the same log data that will be excluded by the rule. When you save the Exclusion Rule, if you **don't** select the option to **Preserve for Live-Tail and Alerts**, then you can check if the rule is applied correctly by seeing if any log data continues to appear in the view. If you do select this option, you may want to monitor the usage from the specified sources in your Usage Dashboard to make sure the rule is being applied. ## Create an Exclusion Rule 1. Go to **Usage > Exclusion Rules**. 2. Click **Add Rule**. 3. Enter a functional title for the rule, like **Exclude daemon.log**. 4. Select any **Sources** from which you want to exclude log data. 5. Select **Apps** from which you want to exclude log data. 6. Enter any **Query** you want to use to exclude log data. 7. Click **Save**. If you select this option, your log data will first come through a live tail and be processed for alerts before the exclusion rule is applied. ### Exclusion Rule Limitations Certain fields are restricted from use within your Exclusion Rule. Utilizing any of these values in your Exclusion Rule will render the rule ineffective. * `_retention` * `_mezmo_line_size` # Set Exclusion Rules Source: https://docs.mezmo.com/docs/exclusion-rules Control which logs are stored by building exclusion rules from hosts, apps, and queries, using one or all three together. Exclusion rules let you control what logs are stored. You can access Exclusion Rules by going to **[Usage > Exclusion Rules](https://app.mezmo.com/manage/exclusion-rules).** There are three parts to a rule. * **Hosts** - Exclude [hosts](/docs/introducing-the-agent) configured on the app. * **Apps** - Exclude [apps](/docs/search-and-filter). * **Query** - Create a [log query](/docs/search-and-filter) You can include one or all to exclude logs. Image # Export Log Lines Source: https://docs.mezmo.com/docs/export-lines Export a local copy of your log lines in JSON line format, delivered as a download link to your Mezmo user email address. This topic describes how to export a local copy of your log lines. Exported files are saved in [JSON line format](http://jsonlines.org/)`.jsonl`, and emailed to your Mezmo user email address. You can also export your log lines programmatically using the [Export API](/docs/api-referencesk9h#export). 1. Log in to the [Mezmo Web App](https://app.mezmo.com). 2. In the left-hand navigation, click **Views** and select the view where you want to search for log lines to export. 3. Enter a [search](/docs/search-and-filter) for the selected view. 4. Enter a time frame to apply to the search results. 5. In the **Unsaved View** menu, select **Export Log Lines.** 6. Select an option to prefer newer or older lines in case the export exceeds our line limit. 7. Click **Request Export**. You will receive an email with a link to download your exported lines. Image The export log lines feature is only available under [the Professional and Enterprise plans](https://www.mezmo.com/pricing). Each plan includes an export limit, which is the number of parallel requests that you can make to the [Export API](/docs/api-referencesk9h#export) at one time. These limits are: **Pro plans with 3 and 7 day retention periods**: 10K log lines per request **Pro plans with 14 and 30 day retention periods**: 20K log lines per request **Enterprise plans**: 20K log lines per request # Export Logs to External Archive Storage Source: https://docs.mezmo.com/docs/export-logs-to-external-storage Configure log archiving to external providers such as AWS S3, Azure Blob, Google Cloud Storage, IBM Cloud, Digital Ocean Spaces, and OpenStack Swift. Mezmo supports exporting logs to external storage providers, including AWS S3, Azure Blob Storage, Digital Ocean Spaces, Google Cloud Storage, IBM Cloud Object Storage, and OpenStack Swift. This topic provides information on configuring both your storage provider and Mezmo Archiving. ## AWS S3 To export your logs to an S3 bucket, make sure that you have an AWS account with access to S3. If you need to create a new S3 bucket for log storage, follow the instructions in the [AWS S3 Getting Started Guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/GetStartedWithS3.html). **Add Mezmo as a Grantee for Your S3 Bucket Access Control List** To set up log archiving for your bucket, follow the AWS instructions for [Using the S3 console to set ACL permissions for a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/managing-acls.html#:~:text=Using%20the%20S3%20console%20to%20set%20ACL%20permissions%20for%20a%20bucket). In those instructions, follow the steps “To grant access to another AWS account” and use this canonical ID for Mezmo: ```none theme={null} 659c621e261e7ffa5d8f925bbe9fe1698f3637878e96bc1a9e7216838799b71a ``` Enable these permissions for Mezmo: * `Objects List` * `Objects Write` * `Bucket ACL Read` * `Bucket ACL Write` To use the [Restore Log Data](/docs/data-restoration) feature, `Object Ownership` should be set to `Object Writer` in the S3 bucket. **Configure Mezmo** 1. In the [Mezmo web app](https://app.mezmo.com), navigate to **Settings > Archiving > Manage**. 2. Toggle **Enable Logging** to **On**. 3. In the **Provider** menu, select **AWS S3**. 4. Enter the name of your S3 bucket, and then click **Save**. ## Azure Blob Storage To export your logs to Azure Blob Storage, make sure that you have an Azure account with access to storage accounts. 1. [Create a Storage Account](https://docs.microsoft.com/en-us/azure/storage/storage-create-storage-account) on Microsoft Azure. 2. Once created, select your storage account, then, under **Settings** click **Access Keys**. 3. Create a key if you do not already have one. 4. In the [Mezmo web app](https://app.mezmo.com), navigate to **Settings > Archiving > Manage**. 5. Toggle **Enable Logging** to **On**. 6. In the **Provider** menu, select **Azure Blob**. 7. Under **Settings** , enter your **Account Name** and **Account Key**, and then click **Save**. ## Digital Ocean Spaces To export your logs to Digital Ocean Spaces, make sure that you have a Digital Ocean account with access to storage. 1. Create a new space (or use an existing one) in [Digital Ocean Spaces](https://cloud.digitalocean.com/spaces). 2. Create a new spaces access key in [Digital Ocean Applications & API](https://cloud.digitalocean.com/settings/api/tokens). Make sure to save the access key and secret key. 3. In the [Mezmo web app](https://app.mezmo.com), navigate to **Settings > Archiving > Manage**. 4. Toggle **Enable Logging** to **On**. 5. In the **Provider** menu, select **Digital Ocean Spaces**. 6. Under **Settings**, input your **Space Name**, **Endpoint**, **AccessKey**, and **SecretKey**, and then click **Save**. You can find your region in your spaces URL. For example`https://my-mezmo-bucket.nyc3.digitaloceanspaces.com` has the region `nyc3`. ## Google Cloud Storage To export your logs to Google Cloud Storage, make that you have a Google Cloud Platform account and project with access to storage. 1. Make sure that [Google Cloud Storage JSON API](https://console.cloud.google.com/apis/library/storage-api.googleapis.com/) is enabled. 2. Create a new bucket (or use an existing one) in [Google Cloud Storage](https://console.cloud.google.com/storage/). 3. Update the permissions of the bucket and add a new member `archiver@logdna-internal-oauth.iam.gserviceaccount.com` with the role of `Storage Admin`. 4. In the [Mezmo web app](https://app.mezmo.com), navigate to **Settings > Archiving > Manage**. 5. Toggle **Enable Logging** to **On**. 6. In the **Provider** menu, select **Google Cloud**. 7. Under **Settings**, enter your **ProjectId** and **Bucket**, and then click **Save**. ## IBM Cloud Object Storage To export your logs to IBM Cloud Object Storage Archiving, make sure that you have an IBM Cloud account with access to storage. 1. Create a new object storage service (or use an existing one) in [IBM Cloud Object Storage](https://console.bluemix.net/catalog/services/cloud-object-storage). 2. Create a new bucket (or use an existing one) in your service for Mezmo dump files. 3. In the [Mezmo web app](https://app.mezmo.com), navigate to **Settings > Archiving > Manage**. 4. Toggle **Enable Logging** to **On**. 5. In the **Provider** menu, select **IBM Cloud Object Storage**. 6. Under **Settings**, enter your **Bucket**, **Public Endpoint**, **API Key**, and **Instance ID**, and then click **Save**. ## OpenStack Swift To export your logs to OpenStack Swift, make sure that you have an OpenStack account with access to Swift. 1. Set up Swift by following [the instructions](https://www.swiftstack.com/docs/cookbooks/swift_usage/auth.html#v2-auth) in the Swift documentation. 2. In the [Mezmo web app](https://app.mezmo.com), navigate to **Settings > Archiving > Manage**. 3. Toggle **Enable Logging** to **On**. 4. In the **Provider** menu, select **OpenStack Swift**. 5. Under **Settings**, enter your **Username**, **Password**, **Auth URL**, **Tenant Name**, and a date for **Expire After**, then click **Save**. # Extract Fields Source: https://docs.mezmo.com/docs/extract-fields Parse additional fields from historical logs without re-ingesting them, using extractors, queries, and time ranges, then export results as CSV. Must be an [admin or owner](/docs/rbac) to use this feature. Extract fields lets you parse additional fields from your historical logs without re-ingesting them. Extracting fields is used as needed and does not change how the logs are seen in the Log Viewer. 1. To Extract fields, select the line you want to extract in the Viewer, then click **Extract Fields**. This will take you to the Extract Fields screen. Image 1. Check the reference line is correct. This is what your parsing will be based on. 2. Choose an extractor type to parse the information needed. 3. Select the auto parsed fields to include. Selecting an auto-parsed field will limit your results to log lines containing that field. 4. Set the time range. The time range is limited based on your plans retention. 5. Enter the query to further refine what logs will be returned. 6. Once you run the query, you'll be able to see the results in a table at the bottom of the screen. 1. The Processed callout shows the total number of log lines processed and if you reached a limit on the number of lines that can be processed 2. The Matched callout shows the percentage of processed logs that were parsed by your template. You can hover either callout to view additional details, including any warnings or errors. 7. You can also sort the columns and drag the column names to sort them. 8. Download the results as a CSV. If you need to parse your logs before they are ingested, create a Custom Parsing Template. # Feature Access Matrix Source: https://docs.mezmo.com/docs/feature-access-matrix Reference table mapping Mezmo features to the Read-only, Basic, Admin, and Owner roles that can access each one. Learn more about members in [Manage Members](/docs/manage-members). | FEATURES | READ-ONLY | BASIC | ADMIN | OWNER | | ---------------------- | --------- | ----- | ----- | ----- | | Live Tail | X | X | X | X | | Search | X | X | X | X | | View Graph | X | X | X | X | | View Organization | X | X | X | X | | Export Logs | X | X | X | X | | View Screens | X | X | X | X | | View Restored Logs | | X | X | X | | Manage Views | | X | X | X | | Manage Alerts | | X | X | X | | Manage Boards | | X | X | X | | Manage Categories | | X | X | X | | Manage Screens | | X | X | X | | View API Keys | | | X | X | | Whitelist Domains | | | X | X | | Archiving | | | X | X | | Restore Logs | | | X | X | | Manage Billing | | | X | X | | Custom Parsing | | | X | X | | Manage Members | | | X | X | | Integrations | | | X | X | | RBAC | | | X | X | | Config Replications | | | X | X | | Generate Keys | | | X | X | | View Ingestion Key | | | X | X | | Create Embedded View | | | X | X | | Manage Exclusion Rules | | | X | X | | Deactivate Account | | | | X | # Format Log Lines with Custom Line Templates Source: https://docs.mezmo.com/docs/format-log-lines-with--custom-line-templates Use custom %LINE templates to reformat log viewer output, display reserved fields, reference the original line, and return data as JSON. You can use custom line templates to format the lines in the log viewer to make it easier for you to identify specific information that is of interest to you. 1. Select an existing View or [Create and Edit Views](/docs/create-and-edit-views). 2. Select **Edit View Properties.** 3. In the **Custom %LINE Template** area, enter your template. ## Display PID, Program, and Log Source If you have log lines that look similar to the example, you can decide to display the information in a more easily parsed format. Image 1. Enter `PID: {{pid}} | Program: {{program}} | Log Source: {{logsource}}` into Custom %LINE Template area. 2. You logs should now look like: ```bash theme={null} Aug 8 11:29:03 samir-Debian-10 daemon.log PID: 468 | Program: logdna-agent | Log Source: ip-12-34-5-67 ``` ## Use Reserved Fields [Reserved fields](/docs/log-parsing) are marked by an underscore. If your data resembles: ```bash theme={null} user 1234 requested endpoint /api/endpoint ``` And contains this field metadata: ```json theme={null} { meta: { first_name: Jane, last_name: Doe } } ``` Enter `{{_meta.first_name}} {{_meta.last_name}}, aka $@` into Custom %LINE Template area. Using `$@` will reference the original line. This will display log messages in that view in this format: ```bash theme={null} Jane Doe, aka user 1234 requested endpoint /api/endpoint ``` ## Return as JSON You can format your data to return as JSON. ```json theme={null} {"index": {{query.index}}, "size": {{query.size}}, "ignore_unavailable": {{query.ignore_unavailable}}, "track_total_hits": {{query.track_total_hits}}, "body": {"query": {{query.body.query}}, "sort": {{query.body.sort}}, "aggs":{{query.body.aggs}}}} ``` ### Formatted Log Line Example ```json theme={null} Aug 8 12:49:20 xxxx-xxxx-xxxxxxxxx-xxxx apiinternal info {"index": ["*:logline.*"], "size": 0, "ignore_unavailable": true, "track_total_hits": true, "body": {"query": { "bool": { "must": [ { "range": { "_ts": { "gte": 1659976890001, "lte": 1659977360647 } } }, { "bool": { "should": [ { "term": { "_app": "localhost" } } ] } } ] } }, "sort": { "_lid": { "order": "desc" } }, "aggs":{ "metrics": { "date_histogram": { "field": "_ts", "interval": "30s" } } }}} ``` # GDPR Compliance Source: https://docs.mezmo.com/docs/gdpr-compliance Understand GDPR obligations for logging and best practices for staying compliant with Mezmo, including data minimization, pseudonymization, and retention. The General Data Protection Regulation (GDPR) is a European Union regulation that grants extensive data privacy protections to EU citizens. It describes the rights of individuals concerning the collection, transmission, and processing of their data. It also describes the obligations of organizations handling this data, including organizations not based in the EU. Although the GDPR is an EU regulation, the scope and restrictiveness of its laws have made it the de-facto standard under which organizations operate. Organizations that handle EU citizen data without following GDPR can be subject to fines of up to [€20 million or 4% of their annual global turnover](https://www.itgovernance.co.uk/dpa-and-gdpr-penalties), investigations, and even a ban on serving EU citizens in the future. In this guide, we’ll explain how to remain compliant with the GDPR as a customer of Mezmo. This document does not provide legal advice but is a general-purpose guide to help you identify and understand your potential obligations. Always check with your legal team before making any changes to your operations. ## The Basics of GDPR The crux of the GDPR is personal data, which is defined under Article 4 as “any information relating to an identified or identifiable natural person.” This includes names, addresses, physiological data, and even online identifiers like IP addresses. Anyone who can be identified through this information is considered a data subject. Essentially, the GDPR protects data subjects by restricting the use of their personal data. Some of the rights that the GDPR gives to data subjects include: * Requiring consent before their data can be collected, processed, or shared (Article 7) * Learning how their personal data is being collected, processed, shared, and safeguarded (Article 15) * Recourse in correcting inaccurate data records (Article 16) * Requesting permanent deletion of their data, also called the “right to erasure” or the “right to be forgotten” (Article 17) * Restricting how their data is processed (Article 18) * Requesting electronic copies of their data (Article 20) * Objecting to certain types of data processing, including automated processing (Article 21) ### Data Controllers and Data Processors The GDPR makes a distinction between [data controllers and data processors](https://ec.europa.eu/info/law/law-topic/data-protection/reform/rules-business-and-organisations/obligations/controller-processor/what-data-controller-or-data-processoren). A data controller determines how and why personal data is collected and processed. A data processor processes personal data on behalf of a data controller. The relationship between a processor and a controller must be outlined in a legal document explaining how the data is to be managed. Note that even though controllers and processors have different obligations under the GDPR, a controller can still be held liable for breaches caused by their processors. In addition, Section 4 requires controllers and processors to designate a Data Protection Officer (DPO) to oversee the organization’s GDPR compliance. ## What GDPR Means for Logging Logs contain a wealth of information about application performance, system operations, user activity, and errors. However, this data creates a substantial risk of personal data making its way into logs, whether deliberately or accidentally. Consider a web server access log. A single entry may contain a user’s IP address, the requested URL, their browser’s user agent, and even their username if they’re logged into your application. If the user causes the application to throw an exception, additional personal data could be logged in a stack trace or variable dump. All of this qualifies as personal data and is enforceable under the GDPR. Under Article 6, this data can be used lawfully in certain scenarios, including when: * The data subject gives their explicit consent * It’s necessary to fulfill a contract or agreement with the data subject * You are complying with other legal obligations * You are protecting the interests of the data subject The GDPR also grants data collectors some flexibility to use personal data as long as those uses are legitimate, don’t conflict with other obligations, and are transparent to data subjects. ## Best Practices for Becoming Compliant The following are general recommendations for mitigating your risk under the GDPR. These are only recommendations and don’t constitute legal advice. Always consult with your DPO before making any changes to your logging strategy. ### Shrink and Reduce Your Logs The easiest way to reduce your risk is to avoid logging personal data in the first place. When reviewing your current logging practices, consider: 1. What personal data is contained within your logs 2. Where your logs are being stored, and for how long Collecting too much data is a liability, especially if it doesn’t contribute to your operations. Some data—such as session IDs—can be useful for auditing and troubleshooting, but you should be able to justify its inclusion in your logs. Article 6 only protects legitimate uses of personal data, so consider whether the data in your logs is essential to your everyday operations. If you’re not sure which data is protected, refer to Article 4(1). Examples of personal data can include IP addresses, geolocation data, and user IDs. Keep in mind that some of this data may be exposed through automated memory dumps and stack traces. If this is a risk, consider modifying your logging implementation to remove or pseudonymize this data if possible. ### Pseudonymize User Data If you need to link a log to a particular data subject, swap out their personal data with pseudonymized data. Pseudonymization is the process of using non-identifying data in place of personal data, while still allowing the non-identifying data to identify an individual. For example, consider an application that logs each successful login. Normally these logs would contain the name of the user that logged in, but if the logs were ever breached, an attacker could identify all of our active users. Instead, we assign a randomly generated number for each user, store it with the rest of their user data, then log this number in place of the user’s name. This way, we can use the number to find the associated user, but an attacker would only see the random number. This is the key difference from anonymized data, which is impossible to link back to an individual. Under Article 25, pseudonymization is considered an appropriate measure for safeguarding data without losing the ability to log user activities. ### Log GDPR-Related Activities Any activities that impact your requirements under the GDPR—including how you can use personal data—should be logged. Certain actions can affect your ability to process data, but it’s your responsibility to show that you have the right to do so. For example, Article 7 allows you to process personal data if the data subjects gives you their consent. However, you must be able to prove that you received this consent, which means storing a record of this consent. Even when a data subject has granted consent, they can revoke that consent or make requests to have their data amended, deleted, or removed from certain types of processing. Not only do you need to prove that you received these requests, but you also need to prove that you honored them. The main purpose of logging GDPR-related activities is to create an audit trail. Article 12 gives data subjects the right to take legal action against controllers if a request has gone unanswered for longer than one month. Logging these activities can protect you in case of a dispute. ### Secure Your Logs Part of protecting personal data means protecting your logs from unauthorized access. Encryption is your first line of defense against a breach by preventing logs from being read by anyone other than its intended recipient. Mezmo uses military-grade encryption to protect your logs both in transit and at rest. Using the [Mezmo agent](/docs/introducing-the-agent), HTTPS endpoint, [syslog with TLS](/docs/ingestion-integrations/rsyslog#tcptls-recommended), or one of our code libraries will automatically encrypt your logs before sending them to Mezmo. If you have [archiving](/docs/archiving) enabled, make sure the storage device receiving your archives is also encrypted. Some of the [largest data leaks](https://businessinsights.bitdefender.com/worst-amazon-breaches) to date resulted from unsecured and unencrypted cloud storage devices. Refer to your cloud storage provider’s documentation to learn how to secure your cloud storage bucket. Lastly, only certain members of your organization should have access to logs containing personal data. Mezmo provides [role-based access controls](/docs/rbac) (RBAC) that lets you restrict certain members’ access to logs. Members who sign into Mezmo are only shown a subset of logs according to their role, reducing the chance of them accidentally viewing or leaking personal data. We recommend you give each member of your Mezmo organization the fewest privileges necessary to perform their tasks. ### Limit Retention The GDPR doesn’t specify a minimum or maximum retention period for log data. Article 5(1)(e) says to retain personal data for no longer than necessary to process it, or when archiving under the conditions outlined in Article 89(1). Generally speaking, stick to shorter retention periods unless you’re certain that your logs contain no personal data. One major exception is audit logging. According to Recital 85, if a breach occurs, you are required to report the breach to a supervisory authority within 72 hours of detecting it, or otherwise prove that no personal data was leaked. Storing a minimum of 72 hours' worth of log data will help you determine the cause and scope of a breach, or prove that there was no unauthorized access to personal data. Mezmo only retains your logs for the duration specified by your plan, after which they are securely deleted. If you want to retain logs for longer, consider using [Mezmo's archive feature](/docs/archiving). ## How Mezmo Supports GDPR Compliance Mezmo works diligently to protect your privacy and your customers’ privacy. You can learn more about our compliance practices on our [GDPR compliance page](https://www.mezmo.com/gdpr). And if you’re not sure how logging with Mezmo might impact your obligations under GDPR, [contact us](https://www.mezmo.com/contact) for more information. # Heroku Template Source: https://docs.mezmo.com/docs/heroku-template Explore the prebuilt Heroku Template of Views, Boards, and Screens for tracking app crashes, HTTP errors, response times, and security trends. The Heroku Template is a collection of Views, Boards, and Screens that enable you to set alerts on your app crashing, view HTTP 500 trends, graph 95th percentile of response times and more. The Heroku Template leverages both the [Heroku platform error codes](https://devcenter.heroku.com/articles/error-codes) as well as router and [system logs](https://devcenter.heroku.com/articles/logging#types-of-logs) emitted by default. Browse the[ full library of Mezmo Templates](https://app.mezmo.com/manage/template-library). ### Views Views are saved shortcuts to a specific set of filters and search queries. You can also add Alerts to views to notify you when specific conditions are met. Check out the topic [Add Alerts to Views](/docs/add-alerts-to-views) for more information. 1. All Heroku HTTP Errors 2. All Heroku Runtime Errors 3. App Crashed (recommended to Alert on) 4. Deployments 5. HTTP 2XX’s 6. HTTP 5XX’s 7. HTTP Forbidden or Unauthorized 8. Memory Quota Exceeded 9. Request Timeout ### Boards Boards are collections of graphs. Using boards, you can track trends with response codes and understand how they fluctuate over time at a glance. Drill down using subplots to see which host or path is generating the most errors. Check out the topic [Visualize Log Data with Graphs](/docs/visualize-log-data-with-graphs) for more information. **Web Server Status** * Volume of HTTP Response Codes Over Time * 95th Percentile of Connection/Service Response Times * Cumulative Response Bytes **Errors** * Frequency of App Crashes * Frequency of Request Timeouts * Frequency of All HTTP Errors * Frequency of all Heroku Runtime Errors ### Screens Screens are collections of customized dashboards that can display data in various forms. See the topic [Use Screens and Widgets to Monitor Log Data](/docs/use-screens-and-widgets-to-monitor-log-data) for more information. **Server** **Overview** * Day Over Day Total Request Volume * Day Over Day Total 5XXs * Top response codes * Top dynos with 5XXs * 95th Percentile Service Response over last day * Day over Day Total Bytes Sent **Server Security** * Top 10 IPs Hitting 401/403s * Day Over Day 401/403s Request Volume * Week Over Week 401/403 Request Volume * Number of 429 Requests * Number of 401/403 Requests * Number of 200 Requests ## Custom Parsing Parsing deployment hash and initiating user into `deploy_user` and `deploy_version.` # HIPAA Compliance Source: https://docs.mezmo.com/docs/hipaa-compliance Understand your logging obligations under HIPAA and HITECH and how Mezmo helps you meet them with encryption, RBAC, retention, and monitoring. The Health Insurance Portability and Accountability Act (HIPAA) was created to improve the way organizations handle healthcare data. Not only does it aim to improve the portability of health information, but also requires organizations to protect and secure it. Any entities handling healthcare data are required to comply with HIPAA, even those handling data on behalf of another entity. If you are a healthcare provider, or an entity providing services to healthcare organizations, you may have compliance obligations under HIPAA. We created this guide to help you understand your potential obligations under HIPAA, and how they may affect your logging strategy. Note that this is not a legal document, but a guide to help you understand your potential obligations. Always check with your legal team before making any changes to your logging procedures or infrastructure. ## An Overview of HIPAA and HITECH Title II of HIPAA, known as the Administrative Simplification provisions, sets standards for protecting and securing private health records (known as protected health information, or PHI). This includes patient records, medical records, and even insurance and billing records. Under HIPAA, any entities that handle this kind of data can be held legally accountable if the data is leaked, stolen, or misused. The Health Information Technology for Economic and Clinical Health Act (HITECH) expands on HIPAA by promoting the use of technology in managing PHI. It offered financial incentives to healthcare providers for adopting electronic health records, while adding tougher penalties for non-compliance. After HITECH was enacted in 2009, the use of electronic records in healthcare organizations jumped from [3.2% in 2008 to 14.2% in 2015](https://www.hipaajournal.com/what-is-the-hitech-act/). ### How Logs Fit Into HIPAA Monitoring and observability play a significant role in HIPAA compliance. Organizations must be capable of: * Auditing employee access to ePHI * Monitoring changes to systems storing ePHI * Identifying and investigating potential security breaches Logs play a vital role in this process as they contain information about application and system operations, user activities, and security incidents. They provide specific and vital details about events including the exact time that it occurred, the application that generated the event, and the users or systems that were involved. Using a log management solution like Mezmo allows you to aggregate logs from all of your systems in a secure central location, as well as providing tools to search, analyze, visualize, and archive your logs to ensure compliance. ## Requirements for Logging The following HIPAA sections outline your requirements for logging. In the following section, we’ll explain how to satisfy these requirements using Mezmo. Before continuing, we need to explain some of the terms used in HIPAA. A [covered entity](https://www.hhs.gov/hipaa/for-professionals/faq/190/who-must-comply-with-hipaa-privacy-standards/index.html) is generally any healthcare provider, health plan, or healthcare clearinghouse that transmits PHI. A [business associate](https://www.hhs.gov/hipaa/for-professionals/privacy/guidance/business-associates/index.html) is any individual, organization, or entity that processes PHI on behalf of a covered entity. If you are a covered entity sending your logs to Mezmo, then we are considered a business associate and have specific obligations for protecting your data. These obligations are described in a Business Associate Agreement (BAA), which must be signed by both the covered entity and business associate before the business associate can provide service. ### Section 164.312: Technical Safeguards This section outlines policies and procedures for protecting and monitoring access to systems containing ePHI. § 164.312(b) specifically addresses auditing, requiring “mechanisms that record and examine activity in information systems that contain or use electronic protected health information.” While this doesn’t specify which activities to log, consider the events that would be important to an audit. Authentication logs, software and hardware modification logs, process logs, and even network logs are all important for monitoring and auditing secured systems. ### Section 164.308: Administrative Safeguards Covered entities are required to “implement policies and procedures to prevent, detect, contain, and correct security violations.” More specifically, § 164.308(a)(1)(ii)(D) requires regular reviews of information system activity “such as audit logs, access reports, and security incident tracking reports.” This section also includes specific monitoring procedures, including: * Malicious software activity (§ 164.308(a)(5)(ii)(B)) * Log-in attempts and discrepancies (§ 164.308(a)(5)(ii)(C)) * Suspected or known security incidents (§ 164.308(a)(6)(ii)) ### Section 164.316: Policies and Procedures and Documentation Requirements § 164.316(b)(2)(1) requires covered entities to retain documentation records for at least six years. However, there’s uncertainty as to whether [audit logs are included in this requirement](https://www.schellman.com/blog/hipaa-audit-log-retention-requirements-do-i-really-need-to-retain-all-my-audit-logs-for-6-years). § 164.316(b)(1)(ii) states that any action, activity, or assessment required to be documented must be recorded, and audit logs technically document activities performed by systems and users. ## Recommendations for HIPAA-Compliant Logging Given these requirements, these are our recommendations for how you can structure your logging strategy to be HIPAA-compliant. Remember that these are recommendations and not legal advice. Always consult with your legal team before changing your logging strategy. ### Avoid Logging PHI Audit logs should rarely if ever contain PHI. Storing PHI in logs increases the risk of a violation, especially if those logs are exported, copied, shared, or archived to a third-party service. If you do need to reference PHI in logs, avoid logging the PHI directly but instead log an abstract identifier that refers back to the PHI. For example, consider an audit log that records access to patient records. Instead of logging protected data such as the patient’s name or Social Security number, log the record number as it appears in the database. This ensures that a patient can’t be identified by logs alone, and can only be identified by accessing both the logs and the patient records. This process is known as de-identification (or pseudonymization) and is a standard under [§ 164.154(a)](https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html). ### Log all Operational and Security Events HIPAA requires strict oversight over systems containing PHI, as described in §§ 164.308(a) and 164.312(b). While there isn’t an exact definition of the types of activities you’re required to log under HIPAA, consider logging important system and security events such as: * Access to protected information and applications * Logins, logouts, and failed attempts * Password changes * Changes to security systems (e.g. firewalls, antivirus software) * Antivirus or antimalware events * Network changes (e.g. new devices connecting to a secure network) * Software installations, uninstallations, and updates * Processes starting and stopping We recommend retaining audit logs for at least six years. Most Mezmo plans retain 30 days of searchable (i.e. “hot”) log data. For longer retention, Mezmo provides an [archiving service](/docs/archiving) that automatically exports older logs to a cloud storage service. Remember to request a BAA from your cloud storage provider and secure your storage bucket before enabling archives. ### Secure Your Logging Infrastructure Logs may not contain PHI, but they do contain sensitive information about HIPAA-regulated systems and applications. Restricting access to your logs and logging infrastructure is an important step in preventing an attacker from finding vulnerabilities in your infrastructure. #### Use Encryption When sending logs to Mezmo’s cloud ingestion servers, use HTTPS or syslog with TLS to encrypt your logs in transit. Otherwise your logs will be sent in plain text, making them trivial to intercept by a malicious third party. Encryption is enabled by default in the Mezmo agent and in our official code libraries. Mezmo also encrypts your logs when storing them, and only allows access to the web application over secure HTTPS. If you are [archiving your logs](/docs/archiving), be sure to encrypt your storage bucket before enabling the archiving process. #### Apply the Principle of Least Privilege The principle of least privilege is a general-purpose principle where users are given the fewest permissions necessary to perform a task. For example, if an engineer needs to access logs from a specific system, that engineer should be given read-only access to only that system’s logs. This is referred to as “access control” under § 164.312(a)(1), which requires protected systems to “allow access to only those persons or software programs that have been granted access rights as specified in § 164.308(a)(4).” Mezmo lets you set granular permissions using [role based access control (RBAC)](/docs/rbac). You can restrict each user’s ability to view, create, or modify Mezmo resources, as well as restrict their access to logs based on source or content. #### Lock Down Your Logging Infrastructure Sending logs to Mezmo requires you to expose certain ports. Mezmo's [Syslog ingestion integrations](/docs/ingestion-integrations/syslog) now recommend forwarding Syslog through an OpenTelemetry Collector to Mezmo's OTEL Source over HTTPS (443) rather than exposing a direct Syslog port (e.g. 6514) to Mezmo. When configuring your firewall, you should only allow outbound traffic from the ports your logging infrastructure actually uses to avoid exposing your systems to unnecessary risk. When using Mezmo on-premise, isolate your logging infrastructure from the public Internet as much as possible to avoid placing your logs in unnecessary risk. In addition, strictly limit system-level access to these systems to specific engineers or members of your operations team. ### Regularly Review Your Logs Under HIPAA, it’s not enough to simply send your logs to Mezmo and store them in an archive for six years. These logs must be constantly monitored and analyzed for signs of any unusual activities, changes in system and application behavior, or potential security incidents. Section 164.308(a)(1)(ii)(D) requires regular reviews of “information system activity, such as audit logs, access reports, and security incident tracking reports.” We recommend designating an employee to check logs on a daily basis, as well as using automated analysis tools such as [views and alerts](/docs/add-alerts-to-views). For example, you can use Mezmo to automatically notify your security team if it detects too many failed logins over a short period of time. #### Auditing Your Logging Systems Regularly reviewing your logs plays another important role by ensuring the logs themselves remain compliant. As with any information system, logs have a risk of becoming lost or corrupt. Remaining compliant means taking measures to protect against this risk and quickly identify it when it happens. For example, imagine one of your systems fails to send its logs to Mezmo. Without adequate monitoring, you may not find out about this until you go to search on the system’s logs. To avoid this scenario, create an [absence alert](https://mezmo.com/blog/mezmo-absence-alerting/) that monitors for any sudden, sharp drops in log volume. An absence alert will immediately notify you of a decrease in log volume, letting you troubleshoot the problem faster. This also applies to archives, as storing archives on a third-party system introduces additional risks of data corruption or loss. This risk is smaller with major providers such as Amazon and Google, but being proactive is much safer than being non-compliant. We recommend periodically checking your archives against the logs stored in your Mezmo account, and if you notice any discrepancies in log volume, [contact us](https://www.mezmo.com/contact-us) to regenerate the archive. As long as the issue is identified within your retention period, we can regenerate the archive with no loss in log volume. ## How Mezmo Supports HIPAA Compliance Mezmo is ready to help you meet your HIPAA compliance requirements. Our systems are audited annually for HIPAA and HITECH compliance by a third-party qualified security assessor. We are committed to protecting your data and will happily sign a Business Associate Agreement (BAA). You can learn more about our commitment to compliance on our [compliance page](https://mezmo.com/compliance/). If you have any questions about Mezmo or about HIPAA compliance with Mezmo, please [contact us](https://mezmo.com/contact-us/). # Troubleshooting With Graphs and Log Data Source: https://docs.mezmo.com/docs/how-to-graph-logs-visualize-data-with-mezmo Walk through real scenarios for using graphs and histograms to detect errors, spot unusual log volume, and analyze performance changes. Visualizing log data provides insights into application availability, performance, resource usage, and errors at a glance. It allows teams to generate statistics, identify trends, and detect patterns in operational behavior. The result is a better overall view of your infrastructure. This article will cover some real scenarios and how to use graphs to understand trends and troubleshoot. ## Detecting Errors If errors are not detected, they can have serious consequences. Logs help us understand the cause and impact of errors, including when they happened, what happened, and what needs to be done to fix them. This information is useful for understanding and fixing errors. This graph shows log lines that contain the word `error` over time. Image You can see a few errors followed by periods of stability. A significant spike occurs shortly after 12pm, which is roughly when the application restarted. We can view the individual log events by clicking on the spike and selecting **Show Logs**. You'll be taken to the Log Viewer. There you'll see all log lines from that date and time. To investigate further, select and individual log line and click **View in Context,** to see the log lines surrounding the error. In this example, someone or something repeatedly tried to log into the PostgreSQL database as an administrator with the wrong credentials. PostgreSQL to logged an error each time the user was denied. Image ## Detecting Unusual Behavior Log volume is a basic but important metric. Sudden changes in log volume can indicate errors, software or hardware failures, performance problems, or problems with the logging infrastructure itself. Restarting a server naturally results in a surge in log volume, which we can easily detect by charting log volume over time. However, any other surges could indicate larger problems. As shown below, our virtual machine experienced two major spikes in log volume followed by a period of relative inactivity: Image To learn more about the cause of these spikes, we can click on the peak of each spike and use the **Show Logs** button to view the logs in the Mezmo dashboard. As it turns out, the first spike is the result of logs generated during the virtual machine’s startup process, which is expected. The second spike, however, consists entirely of logs generated by the popularity-contest service, which records statistics about the software installed on the system. Although popularity-contest only ran for a short time, it generated almost an equivalent volume of log data as the startup process. To get a better idea of the impact that popularity-contest had on log volume, we can add a new plot to the total volume chart and filter the app field to only show events originating from popularity-contest: Image ## Investigate With Histograms You may want to count log volume by a specific field, such as the log-level, host, application, or a custom tag. For example, plot out log volume based on the service that generated each event (stored in the app field). In the example, you can see, the popularity-contest service generates the greatest number of messages, followed by the virtual machines syslog service and daemon log. Image If we create a histogram using the level field instead of the app field, we see that most of these messages are either info or debug logs. There are still a few warning, error, and fatal logs to investigate. Image We can click on the popularity-contest spike and use the **Show Logs** button to view log entries originating from the service. Image The solution to this problem was to disable the popularity-contest service. This resulted in slightly faster startup times, less network traffic, and lower log volumes. ## Analyzing Changes in Performance We can use visualization tools to track application performance metrics, including response times, response sizes, and request volumes. For example, our web application logs the size of each response in bytes. After the restart, the average response size started significantly higher than usual. Shortly after, the response size fell to a low and steady average. Image Using percentiles, we can track metrics that fall far outside of the average, such as unusually large responses. We can do this in Mezmo by changing the metric measured from **Averages to 99th Percentile**, which displays the top 1% of response sizes over the same period of time: Image You can see the log messages for each data point on the chart. For example, the large responses were caused by users having to re-cache an asset-heavy page after the application was restarted. Once users stopped caching this page, the extremes fell to within 2–3 KB of the average. We can mitigate this by reducing the size of the page, updating the web server caching configuration, or by placing the server behind a content delivery network. # Import and Export Organizations Source: https://docs.mezmo.com/docs/import-and-export-organizations Export an organization's views, boards, alerts, exclusion rules, and parsing templates as JSON and import them into another organization. You can export your configuration for an organization as a JSON file, then import the file to another organization. You can export: * [Views](/docs/create-and-edit-views) * [Boards](/docs/visualize-log-data-with-graphs) * [Alerts](/docs/add-alerts-to-views) * [Exclusion rules](/docs/excluding-log-lines) * [Parsing Templates](/docs/parse-logs-with-custom-templates) ## When To Use Import and Export * **Replicate existing account configurations**. Use the existing account configuration for multiple new Mezmo accounts. This makes them easier to manage, support, and keeps things consistent. * **Move configurations** from staging to production. * **Restore account configuration** using the Import option. This will return the configurations to a saved state and overwrite any existing configurations. ## Feature Notes * The file is exported as a JSON file * Administrator privileges are required to use this feature * You can export from the same account and replace in the same account. You can't add to and from the same account. This prevents accidental duplication of views, alerts, and other configurations in the same account. * Importing the same config file twice to one account will create duplicates of every configuration that was originally exported to the file, except for categories. * You can't export alerts without also exporting the views they're attached to. You can export views without alerts. * To stop multiple team members from overwriting each other's work, there is a 10 minute wait between import attempts to the same account. * If an exported configuration file is modified in any way, the import will fail due to a failed checksum action * The configuration file doesn't expire, so you can keep a copy to restore your settings if your account's configuration ever needs to be restored * No log data is exported ## Export Configurations 1. Go to [**Settings > Organization > Export**](https://app.mezmo.com/manage/export-config). By default any configurations you have set are selected. Clear the selection for any you don't want to export. 2. For each type of configuration, click the arrow beside the type to display the full list. Use the arrow to drill down into the categories and view the configuration. 3. For views, you can select whether to also export any alert that's attached to the view. 4. The values shown in the parentheses indicate the total counts of configurations of that type. 5. Click **Export**. 6. A JSON file with the selected configurations downloads to your local machine. The name of the file is prepended with `mezmo-config`. ## Import Configurations 1. Go to [**Settings > Organization > Import**](https://app.mezmo.com/manage/import-config). 2. Upload your JSON configuration file. 3. Once the file is uploaded, choose **Add** or **Replace**. 4. Add to existing configuration 5. Replace existing configuration. **This is irreversible**. 6. Click **Import**. 7. The Import Configuration page reloads, and displays information about the before and after states, including the types and numbers of configurations that were imported. You'll need to refresh the browser to see the new configurations. ## Restore Account Configuration To restore account configuration, store a copy of your exported JSON file and choose **Replace existing configuration**. # Index Rate Alerts Metrics Source: https://docs.mezmo.com/docs/index-rate-alerts-metrics Learn how Mezmo calculates index rate, ingest rate, lines/s, z-score, and standard deviation to set thresholds for Index Rate Alerts. Several metrics are used to calculate the index rate of your log lines, which you can then use to set the thresholds for your Index Rate Alerts. ## Index Rate This rate is the average value of all index rates measured over the last rolling hour. Log lines are indexed every five minutes to get the total number of lines indexed over an hour. The total number of lines is divided by 60 to get the average per minute. That's divided by 60 again to get the average index rate per second for that hour. If no log lines are ingested for over 60 minutes, you will see a Check Later message instead of the index rate. After log lines resume, the Check later message is replaced with the index rate value. Since measurements are taken every five minutes, the first measurement of ingested log lines for that five minutes will be averaged out for the full hour. Any time there is an interruption in ingestion, wait at least an hour before using the index rate value for analysis. ## Ingest Rate The ingest rate measures the average hourly rate at which log lines are ingested. To determine the ingest rate, Mezmo applies the same formula used to determine the index rate average. ## Lines/s The lines-per-second value is the measurement of the average number of lines that were indexed, per second, over the current rolling hour. ## Z-Score The z-score is the count of standard deviations. Standard deviations are measured by assessing the hourly index rate averages of the past 30 rolling days to contrast the historic hourly average against the current hourly average to identify significant spikes. ## Historical Hours Mezmo records the ingest and index rates for fixed hours; for example, June 22 from 1PM to 2PM. These rates for the historical hours are used to calculate a rolling 30 day average index rate and the rolling 30 day std deviation after the collection of 30 days of data. ## Std (standard) Deviation Mezmo collects index metrics for a rolling 30-day period, and calculates the index rate for each hour in that 30-day time frame. Knowing the standard hourly index rate lets Mezmo to identify spikes, or deviations, in volume relative to standard index volume. The severity of a spike is measured in standard deviation or how far from the regular index rate is the spike. ## No Data On the graph for average index rates, any area on the graph for which no data was collected will be marked as no data. # Index Rate Alerts Overview Source: https://docs.mezmo.com/docs/index-rate-alerts-overview Use Index Rate Alerts to track index and ingestion rates, spot volume spikes by source, and monitor anomalous flow with the alerts dashboard. Configuring alerts based on the index rate or retention and storage rate of your log data helps you track unusual behavior in your systems. For example, if there's a sudden spike in volume, Mezmo's Index Rate Alert feature tells you which applications or sources produced the data spike. It also shows any recently added sources. Index rate alerts can also help managers who are responsible for budgets to analyze and predict storage costs. ## Index Rates Alert Dashboard The Index Rate Alert page tells you the rate of indexing and ingestion for your account. You can see the ratio of log lines being ingested to the number of log lines being stored. In other words, you can see how much of what you're taking in and getting indexed. ### Rate Alerts There are two types of index rate alerts: * **Max lines/s** - An excessive flow rate alert which measures, in lines per second, the log lines indexed or stored. * **Max z-score** - An alert for anomalous flow rates, based on the number of standard deviations away from the regular baseline of the last 30 days. ### Average Index Rates Graph The Index Rate Alert page displays a graph that visualizes the index rate for your log data over the past 7 days. You can toggle the graph to view visualizations. * **Index Rate** - The daily average index rates contrasted to the daily minimum and maximum index rates. * **Standard Deviation** - The daily average index rates contrasted to the standard range for the past 30 days. You can also toggle between viewing index rates for the past 30 days, the past seven days, or the past one day. # Mezmo Log Ingestion Methods Source: https://docs.mezmo.com/docs/ingestion Compare ways to send logs to Mezmo, including the Mezmo Agent, client-side logger, platform integrations, code libraries, and the REST API. **Ingestion** refers to the process of formatting and uploading log data from external sources like hosts, applications, and cloud-based logging services. In the process of ingesting log data, Mezmo also parses the information in the log lines following [automatic](/docs/log-parsing) and [custom rules](/docs/parse-logs-with-custom-templates) to make it available for searching, and for use in data analysis. ## Mezmo Ingestion Methods Mezmo provides several methods for ingesting log data. #### Mezmo Agent You can install [the Mezmo Logging Agent](/docs/introducing-the-agent) directly on the host where your logs are generated, and it will maintain a persistent, HTTPS-encrypted connection to the Mezmo ingestion servers. You can deploy the Mezmo Agent on Linux, Windows, macOS, Kubernetes, and OpenShift systems. #### Mezmo Client-Side Logger Built on the Mezmo Node.js library, the [Mezmo client-side logger](/docs/client-side-logging) sends logs from your client-side JavaScript applications to Mezmo's ingestion servers. #### Platform Integrations Mezmo has developed [integrations](/docs/ingestion-integrations) to send logs directly from several popular logging platforms, including Akamai Cloud Monitor, AWS CloudWatch, Docker, Heroku, and varieties of Syslog. #### Code Libraries There are both Mezmo supported and community [Code Libraries](/docs/code-libraries) available for sending logs directly from applications built in Go, Python, Ruby, Rust, Android, and iOS. #### REST API You can use the [Mezmo API](/docs/api-referencesk9h) to send log lines to Mezmo, as well as programmatically manage starting and stopping ingestion. ## Troubleshooting and Tips For tips and troubleshooting information for the Mezmo Agent and other ingestion methods, check out the [Mezmo Log Analysis Support Knowledge Base](https://supportkb.mezmo.com). # Mezmo Log Ingestion Integrations Source: https://docs.mezmo.com/docs/ingestion-integrations Browse the log ingestion integrations Mezmo develops and maintains for popular platforms. Mezmo develops and maintains log ingestion integrations for multiple popular platforms. You can find a complete listing in this section. * [Akamai Cloud Monitor](/docs/ingestion-integrations/akamai-cloud-monitor-logs) * [Aptible](/docs/ingestion-integrations/aptible-logs) * [AWS CloudWatch](/docs/ingestion-integrations/cloudwatch) * [AWS Elastic Beanstalk](/docs/ingestion-integrations/elastic-beanstalk) * [AWS S3](/docs/ingestion-integrations/amazon-s3-logs) * [Cloud Foundry](/docs/ingestion-integrations/cloudfoundry) * [Docker](/docs/ingestion-integrations/docker) * [Fluent Bit](/docs/ingestion-integrations/fluentbit) * [Fluentd](/docs/ingestion-integrations/fluentd) * [Flynn](/docs/ingestion-integrations/flynn) * [GitHub Events](/docs/ingestion-integrations/github-events) * [Heroku Logs](/docs/ingestion-integrations/heroku) * [LaunchDarkly](/docs/ingestion-integrations/launchdarkly) * [NXLog](/docs/ingestion-integrations/nxlog-for-windows) * [OpenTelemetry Exporter](/docs/ingestion-integrations/opentelemetry-exporter) * [Rsyslog](/docs/ingestion-integrations/rsyslog) * [Syslog](/docs/ingestion-integrations/syslog) * [Syslog-ng](/docs/ingestion-integrations/syslog-ng) * [SaltStack](/docs/ingestion-integrations/saltstack) * [Vercel](/docs/ingestion-integrations/vercel) * [Vector](/docs/ingestion-integrations/vector) # Akamai Cloud Monitor Source: https://docs.mezmo.com/docs/ingestion-integrations/akamai-cloud-monitor-logs Configure Akamai Cloud Monitor data delivery to forward logs to Mezmo and learn how Akamai datasets and fields are parsed and formatted. Follow the [Cloud Monitor Implementation Guide](https://control.akamai.com/dl/customers/ALTA/Cloud-Monitor-Implementation.pdf) to configure Akamai Cloud Monitor to forward logs to Mezmo. Use the parameters listed here to set the configuration options. ## Cloud Monitor Data Delivery | Genera Configuration Option | Parameter | | --------------------------- | ------------------------------------------------------------------------------------------------------------- | | **Origin Server Hostname** | `logs.logdna.com` | | **HTTP Port** | `443` | | **HTTPS Port** | `443` | | **Content Provider Code** | Select **812828 - Cloud Monitor to LogDNA**

If this option isn't available, contact Akamai Support | | **Use SNI TLS Extension** | Yes | | Origin SSL Certificate CN Options | | | --------------------------------- | ------------------------------------ | | **Verification Settings** | **Choose Your Own** | | **Trust** | **Specific Certificates (pinning)**; | | **Match CN/SAN** | `{{Origin-Hostname}}`; | | **Hostname/IP** | `logs.logdna.com` | | **HTTPS Port** | '443' | ## Dynamic Site Accelerator In your **Dynamic Site Accelerator** configuration, set the **Cloud Monitor Instrumentation** behavior with the these parameters: | | | | ----------------------------------- | ------------------------------------------------------------------------------------------------ | | **Cloud Monitor Delivery Hostname** | `logs.logdna.com` | | *Data Sets to Include\** | Specify the datasets for your logs | | **Delivery URL Path** | `/akamai/ingest/`

Use your Mezmo **Access Key** for '\' | Use querystring parameters to override **hostname**, **app**, or to add **tags**: `/akamai/ingest/?hostname=newHost&app=newTest`; `/akamai/ingest/?tags=akamai,logdna&app=newTest`; `/akamai/ingest/?tags=tag1&hostname=host1`, etc. ## How Akamai Logs are Parsed You can find information for each of your datasets in these log entries: | | | | ------------------------------------- | ----------- | | **Base Log Line Data** | `meta.*` | | **Geographic Data** | \`geo.\*' | | **HTTP 1.0** | `http.*` | | **Message Exchange Data** | `http.*` | | **Network Data 1.0** | `network.*` | | **Network Performance Data** | `netPerf.*` | | **Request Header Data** | `reqHdr.*` | | **Response Header Data** | `respHdr.*` | | **Web Application Firewall Data 2.0** | `waf_2.*` | The information about each dataset and the definition of each field can be found in Appendix A: Cloud Monitor Default Connectors section of [Cloud Monitor Implementation Guide](https://control.akamai.com/dl/customers/ALTA/Cloud-Monitor-Implementation.pdf). Unless specified, **hostname** is always `http.reqPath` and **app** is always AkamaiCloudMonitor. The log line is formatted using some `http.*` fields as in this example: `cliIP reqMethod reqPath?reqQuery proto/protoVer status bytes reqCT reqLen respCT respLen UA` **Notes**: * `-` is used if the optional field is missing; * Just `reqPath` is used if there is no `reqQuery`; * In order to format the line, the fields are joined together using whitespace delimiter in this order: * `http.cliIP`; * `http.reqMethod`; * `http.reqPath?http.reqQuery` or just `http.reqPath`; * `http.proto/http.protoVer`; * `http.status`; * `http.bytes`; * `http.reqCT`; * `http.reqLen`; * `http.respCT`; * `http.respLen`; * `http.UA`. # AWS S3 Source: https://docs.mezmo.com/docs/ingestion-integrations/amazon-s3-logs Use an AWS Lambda function to subscribe to an S3 bucket, transform retrieved log data, and route your logs from S3 into Mezmo. The Mezmo Amazon S3 integration uses [AWS Lambda](https://docs.aws.amazon.com/lambda/index.html) to process and transform your log data as it is retrieved from [S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html), and then routes your logs from S3 to Mezmo. Using our Mezmo integration, you can configure a Lambda function to subscribe to your AWS S3 bucket and then optionally use environment variables to fine-tune the function. For more detailed information about the Mezmo AWS Lambda function, refer to the documentation in our public [GitHub repository](https://github.com/logdna/logdna-s3/tree/v2.0.0). # Aptible Source: https://docs.mezmo.com/docs/ingestion-integrations/aptible-logs Set up Aptible log ingestion from the Mezmo Web App using your ingestion key, and learn why multi-line parsing is not supported. ## Set Up Aptible Log Ingestion Follow the instructions in the Mezmo Web App to set up Aptible log ingestion using your Mezmo ingestion key. 1. Log in to [the Mezmo Web App](https://app.Mezmo.com/account/signin). 2. In the bottom section of the left-hand navigation, click **Help**. 3. Select **Add Log Sources**. 4. Under **Via platform**, click **Aptible**. 5. Follow the instructions to set up Aptible log ingestion. Note that your ingestion key is automatically inserted into the configuration code. Because of the way Aptible interacts with the Mezmo Ingestion API, our integration does not support multi line parsing. This means that stack traces may be split up into several log lines, instead of being represented as a single entity. # Cloud Foundry Source: https://docs.mezmo.com/docs/ingestion-integrations/cloudfoundry Stream events from your entire Cloud Foundry deployment to Mezmo using the Mezmo nozzle and your ingestion key. The [Mezmo nozzle](https://github.com/logdna/logdna-cloudfoundry) streams events from your entire Cloud Foundry deployment to Mezmo. The nozzle runs as an app within Cloud Foundry, making it easy to deploy and manage. In minutes, all of your Cloud Foundry app logs will begin appearing in Mezmo. ## Set Up Cloud Foundry Log Ingestion Follow the instructions in the Mezmo Web App to set up Cloud Foundry log ingestion using your Mezmo ingestion key. 1. Log in to [the Mezmo Web App](https://app.mezmo.com/account/signin). 2. In the bottom section of the left-hand navigation, click **Help**. 3. Select **Add Log Sources**. 4. Under **Via platform**, click **Cloud Foundry**. 5. Follow the instructions to set up Cloud Foundry log ingestion. Note that your ingestion key is automatically inserted into the configuration code. # AWS CloudWatch Source: https://docs.mezmo.com/docs/ingestion-integrations/cloudwatch Route and transform CloudWatch log data to Mezmo using the AWS Lambda-based Mezmo CloudWatch integration. The Mezmo Amazon [CloudWatch](https://aws.amazon.com/cloudwatch/) integration uses AWS [Lambda](https://docs.aws.amazon.com/lambda/index.html) to process and transform your CloudWatch log data, and then routes your logs from CloudWatch to Mezmo. For more detailed information about the Mezmo Cloudwatch integration, refer to [the documentation in our public GitHub repository](https://github.com/mezmo/mezmo-cloudwatch). # Docker Source: https://docs.mezmo.com/docs/ingestion-integrations/docker Send logs from Docker, Docker Cloud, ECS, and other Docker-based platforms to Mezmo using the Logspout-based Mezmo Docker container. The [Mezmo Docker container](https://github.com/logdna/logspout) lets you send logs from Docker, Docker Cloud, Amazon Elastic Container Service (ECS), and other Docker-based platforms. The container automatically collects logs generated by other containers running on the host and sends them to Mezmo's ingestion servers. The container is based on [Logspout](https://hub.docker.com/r/logdna/logspout/) and supports custom metadata, such as a custom hostname and tags. To learn how to send logs from orchestration tools like Kubernetes, Docker Swarm, and Rancher, see [the Kubernetes documentation page](https://github.com/logdna/logdna-agent-v2/blob/master/docs/KUBERNETES.md). As part of your container logging strategy, all of your containers should log to STDOUT and STDERR (standard output). Not only is this required for Logspout to work, but Docker itself expects containers to log to these outputs. Logging your containers to standard output is both a best practice and essential for the Logspout container to function. ## Set Up Docker Log Ingestion Mezmo supports log ingestion from these Docker platforms: * Docker * Docker Cloud * Docker Compose * Docker Swarm * Amazon Elastic Container Service (ECS) * Rancher * Convox Follow the instructions in the Mezmo Web App to set up Docker log ingestion using your Mezmo ingestion key. 1. Log in to [the Mezmo Web App](https://app.mezmo.com/account/signin). 2. In the bottom section of the left-hand navigation, click **Help**. 3. Select **Add Log Sources**. 4. Under **Via platform**, click **Docker**. 5. Follow the instructions to set up Docker log ingestion. Your ingestion key is automatically inserted into the configuration code. # AWS Elastic Beanstalk Source: https://docs.mezmo.com/docs/ingestion-integrations/elastic-beanstalk Configure the Mezmo agent on AWS Elastic Beanstalk by adding a .config file to your app's .ebextensions directory with your ingestion key. For the best user experience, we recommend logging into the [Mezmo web app](https://app.Mezmo.com/) and following the account-tailored add log source instructions. You may also follow the more generic instructions below Copy the configuration for your version of the Mezmo agent into a .config file in your Elastic Beanstalk app's .ebextensions directory. Before redeploying your Elastic Beanstalk app, be sure to insert your [Mezmo Ingestion Key](https://app.Mezmo.com/manage/api-keys) and specify any custom directories you want the Mezmo agent to monitor. ## Mezmo Agent v2 ```yaml theme={null} files: "/etc/yum.repos.d/logdna.repo": mode: "000644" owner: root group: root content: | [logdna] name=logdna packages baseurl=https://assets.logdna.com/el6/ enabled=1 gpgcheck=1 gpgkey=https://assets.logdna.com/logdna.gpg "/home/ec2-user/logdna.sh": mode: "000755" owner: root group: root content: | #!/bin/bash rpm --import https://assets.logdna.com/logdna.gpg yum -y install logdna-agent systemctl start logdna-agent systemctl enable logdna-agent "/etc/logdna.env": mode: "000644" owner: root group: root content: | LOGDNA_INGESTION_KEY=YOURAPIKEY LOGDNA_TAGS=TEST commands: 1_install_agent: command: "/home/ec2-user/logdna.sh" 2_enable_agent: command: "systemctl restart logdna-agent" ``` # Fluent Bit Source: https://docs.mezmo.com/docs/ingestion-integrations/fluentbit Set up log ingestion to Mezmo using the Fluent Bit Mezmo output plugin and your Mezmo ingestion key. Fluent Bit has developed a Mezmo output plugin that enables you to set up an integration with Mezmo by creating a new Fluent Bit configuration file. You can find complete instructions and an example file in [the Fluent Bit documentation for the Mezmo output plugin](https://docs.fluentbit.io/manual/pipeline/outputs/logdna). You will need to [log in to the Mezmo Web App](https://app.mezmo.com/) to get your **Ingestion Key** to complete the configuration. # Fluentd Source: https://docs.mezmo.com/docs/ingestion-integrations/fluentd Send Fluentd logs to Mezmo using the Mezmo plugin, set up ingestion in the Web App, and configure tailing in td-agent.conf. Mezmo has developed a plugin to send your Fluentd logs to Mezmo. For more detailed information regarding the Fluentd plugin and configuration options, check out our [Github repo](https://github.com/logdna/fluent-plugin-logdna). ## Set Up FluentD Log Ingestion Follow the instructions in the Mezmo Web App to set up FluentD log ingestion using your Mezmo ingestion key. 1. Log in to the [Mezmo Web App](https://app.mezmo.com). 2. In the bottom section of the left-hand navigation, click **Help**. 3. Select **Add Log Sources**. 4. Under Via platform, click **FluentD**. 5. Follow the instructions to set up FluentD log ingestion. Note that your ingestion key is automatically inserted into the configuration code. ## Using FluentD to Tail Logs Add code to `td-agent.conf`. ```none theme={null} @type tail path C:\tmp\*.log #change to any path pos_file C:\tmp\httpd-access.log.pos #change to any temp path as needed tag tail #could be any tag @type none ``` # Flynn Source: https://docs.mezmo.com/docs/ingestion-integrations/flynn Set up Flynn log ingestion to Mezmo by following the Web App instructions using your Mezmo ingestion key. ## Set Up Flynn Log Ingestion Follow the instructions in the Mezmo Web App to set up Flynn log ingestion using your Mezmo ingestion key. 1. Log in to [the Mezmo Web App](http://app.mezmo.com). 2. In the bottom section of the left-hand navigation, click **Help**. 3. Select **Add Log Sources**. 4. Under **Via platform**, click **Flynn**. 5. Follow the instructions to set up Flynn log ingestion. Note that your ingestion key is automatically inserted into the configuration code. # GitHub Events Source: https://docs.mezmo.com/docs/ingestion-integrations/github-events Connect the Mezmo GitHub integration to collect repository events, choose which event types to monitor, and filter them in the Web App. The Mezmo integration for GitHub Events lets you collect information associated with GitHub events, like the repository where an event occurred or the user who triggered the event. The Mezmo integration for GitHub events monitors these GitHub Event types by default: * PushEvent * CreateEvent * DeleteEvent * Commit * CommentEvent * ReleaseEvent * ForkEvent * PullRequest * EventPull * RequestReview * CommentEvent You also have options to include Extended Repository Events, Issues Events, and Deployment Events. ## Set Up GitHub Event Logging 1. Log in to the [Mezmo Web App](http://app.mezmo.com). 2. Go to **Settings > Integrations > GitHub**. 3. Click **Connect to GitHub**. If prompted, sign in to GitHub. 4. Click **Authorize Application**. 5. Select the GitHub repositories that you want to monitor. 6. Select any optional event types that you want to monitor for your repository. If you select the GitHub source in the **All Sources** filter, you can see all GitHub events that were sent to Mezmo. You can also filter by specific repositories in **All Apps > Repositories** . # Heroku Logs Source: https://docs.mezmo.com/docs/ingestion-integrations/heroku Ingest Heroku logs into Mezmo using the Heroku add-on or log drains, with notes on user roles, multi-app plans, and Private Space logging. You can set up Heroku log ingestion for Mezmo either by installing the Mezmo Heroku Add-on available in [the Heroku Elements Marketplace](https://elements.heroku.com/addons/logdna), or by setting up Heroku Log Drains. You may prefer to use Heroku log drains if you have a mixed distribution of Heroku apps and non-Heroku services. Instructions for both set up configurations are available in the Mezmo Web App. ## User Roles for Heroku-Provisioned Accounts When you provision the Mezmo add-on through Heroku, user roles are assigned automatically. If your email matches the account owner email, you receive the Owner role with full access to all Mezmo features, including the UI and API. Learn more about [Role-based Access Control](/docs/rbac). ## Set Up Heroku Log Ingestion Follow the instructions in the Mezmo Web App to set up Heroku log ingestion using your Mezmo ingestion key. 1. Log in to [the Mezmo Web App](https://app.mezmo.com/account/signin). 2. In the bottom section of the left-hand navigation, click **Help**. 3. Select **Add Log Sources**. 4. Under **Via platform**, click **Heroku**. 5. Follow the instructions to set up Heroku log ingestion using the method you prefer. Note that your Mezmo ingestion key is automatically inserted into the configuration code. If you have more than one Heroku app using the Mezmo Heroku add-on: * The features from the most expensive Mezmo plan will apply to all of your apps * The data for each Mezmo plan is additive, so if you have a plan with 10 MB and another with 20 MB, your total volume would be 30 MB * The data for each Mezmo plan is part of a total pool available to all your apps We don't recommend that you use this option. Enabling it will cause all of your logs to be grouped under the original Heroku app where you installed the add. This makes it difficult and confusing to distinguish which log lines came from which Heroku app. Instead, we strongly recommend that you install the add-on for each Heroku app that you want to send logs to Mezmo. You must enable logging for Private Spaces when you create them to use Private Space Logging. If you expand the context for a log line generated by a Heroku dyno, you can see the dyno information listed there. You can even filter by dyno using [a dyno field search](/docs/search-and-filter#field-search). dyno:mydyno # LaunchDarkly Source: https://docs.mezmo.com/docs/ingestion-integrations/launchdarkly Send and aggregate LaunchDarkly audit log events in Mezmo by configuring the integration with your Mezmo ingestion key. The LaunchDarkly integration for Mezmo sends and aggregates LaunchDarkly audit log events in Mezmo. This integration is set up from within the LaunchDarkly web interface, but you will need to [log in to the Mezmo Web App ](https://app.mezmo.com/)and get your **Ingestion Key** to use in the integration set up. [The LaunchDarkly documentation for the Mezmo integration](https://docs.launchdarkly.com/integrations/logdna) provides complete set up instructions. # NXLog Source: https://docs.mezmo.com/docs/ingestion-integrations/nxlog-for-windows Configure NXLog on Windows to forward Windows event and file logs to an OpenTelemetry Collector, which relays them to the Mezmo OTEL Source. NXLog is the workhorse of Windows logging plugins. Mezmo ingests NXLog data by placing an OpenTelemetry (OTel) Collector between your Windows hosts and Mezmo. NXLog forwards Syslog-formatted events to the Collector's `syslog` receiver over TLS, and the Collector relays them to Mezmo's [OpenTelemetry Source](/telemetry-pipelines/open-telemetry-source) with the `otlphttp` exporter. ``` NXLog (om_ssl output) --> OpenTelemetry Collector (syslog receiver) --> Mezmo OTEL Source (otlphttp exporter) ``` ## Set Up the OpenTelemetry Collector 1. Log in to the [Mezmo Web App](https://app.mezmo.com), create (or open) a Pipeline, and add an [OpenTelemetry Source](/telemetry-pipelines/open-telemetry-source). Note the ingestion key — you'll use it as `apikey`. 2. Download the `otelcol-contrib` distribution, which includes the `syslog` receiver, from [the OpenTelemetry website](https://github.com/open-telemetry/opentelemetry-collector-releases). 3. Create a `config.yaml`: ```yaml theme={null} receivers: syslog: protocol: rfc5424 tcp: listen_address: "0.0.0.0:CUSTOM_PORT" tls: cert_file: /path/to/cert.pem key_file: /path/to/key.pem exporters: otlphttp/mezmo: endpoint: "https://logs.mezmo.com/otel" compression: gzip headers: apikey: "" service: pipelines: logs: receivers: [syslog] exporters: [otlphttp/mezmo] ``` 4. Start the Collector: ```bash theme={null} ./otelcol-contrib --config /path/to/config.yaml ``` ## Configure NXLog Point NXLog's `om_ssl` output at the host and port where the Collector's `syslog` receiver is listening, using the same certificate you configured on the Collector. ```none theme={null} Panic Soft #NoFreeOnExit TRUE define ROOT C:\\Program Files (x86)\\nxlog define CERTDIR %ROOT%\\cert define CONFDIR %ROOT%\\conf define LOGDIR %ROOT%\\data define LOGFILE %LOGDIR%\\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\\modules CacheDir %ROOT%\\data Pidfile %ROOT%\\data\\nxlog.pid SpoolDir %ROOT%\\data Module xm_syslog Module xm_exec Module xm_json Module im_internal Exec $Message = to_json(); ####################################################################### ##### This is just explicit version of internal input above ########### ####################################################################### # # Module im_file # File '%LOGFILE%' # # $Message = $raw_event; # if $Message == '' drop(); # $SourceName = substr(file_name(), size('%LOGDIR%') + 1); # # ####################################################################### # Define Directory for Making Substring Operation define LOGFOLDER C:\\ProgramData\\logs Module im_file File '%LOGFOLDER%\\*.log' Recursive TRUE $Message = $raw_event; if $Message == '' drop(); $SourceName = substr(file_name(), size('%LOGFOLDER%') + 2); Module im_msvistalog Exec $Message = to_json(); Module pm_buffer MaxSize 102400 Type disk Module om_ssl Host Port CUSTOM_PORT CAFile %CERTDIR%\ca.pem Exec to_syslog_ietf(); Path internal, filelog, eventlog => buffer => out ``` You can add additional logfiles by creating a new `` section that imitates the previous ones, and adding the name of that section to `` at the end. ## Example for Tailing Additional Log Files ```none theme={null} Module im_file File '%LOGDIR%\\example.log' Exec $Message = to_json(); ``` # OpenTelemetry Exporter Source: https://docs.mezmo.com/docs/ingestion-integrations/opentelemetry-exporter Send log data to Mezmo by adding the Mezmo exporter to an OpenTelemetry Collector pipeline, with a quick start config example. You can export your log data directly to Mezmo with the OpenTelemetry Exporter. If you are already using OpenTelemetry Collector, you can start sending log data by adding the mezmo exporter to your existing pipelines. If you are new to OTEL, you can use this quick start example to get you up and running. ### Set Up OpenTelemetry Exporter The Mezmo Exporter only supports capturing log data. **Metrics** and **Traces** from OTEL will not be captured. 1. Download the appropriate OTEL collector for your environment from [the OpenTelemetry website](https://github.com/open-telemetry). 2. Create a configuration file called `config.yaml` with these contents: ```yaml theme={null} receivers: filelog: include: [ ./*.json ] operators: - type: json_parser exporters: mezmo: ingest_url: "https://logs.mezmo.com/otel/ingest/rest" ingest_key: "" service: pipelines: logs: receivers: [ filelog ] exporters: [ mezmo ] ``` 3. Start the collector with this command: ```bash theme={null} --config=/path/to/config.yaml ``` 4. Create`.json` files in this directory. 5. As you add json lines to this file they will appear in the Log Viewer in Mezmo. For complete information on setting up and configuring the OpenTelemetry Export for Mezmo log ingestion, check out [the GitHub repo for the mezmoexporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/mezmoexporter/README.md). For `ingest_key`in the `config.yaml` file, enter your Mezmo Ingestion Key. You can find this by logging into [the Mezmo Web App](https://app.mezmo.com), and navigating to **Settings > Organization > API Keys**. `otel`will be set at the default hostname for collector logs only if your logs don’t have a hostname set in the log metadata. You cannot set this metadata hostname in your collector configuration. Instead, recent changes to the collector configuration will enable the collector to recognize the hostname data from incoming logs, and add that data to the internal OTEL representation of the log. The Mezmo OTEL Exporter will then attach the hostname data to the outgoing logs. For more information, see [the OTel Exporter changelog in GitHub](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CHANGELOG.md). # Rsyslog Source: https://docs.mezmo.com/docs/ingestion-integrations/rsyslog Forward Rsyslog logs to an OpenTelemetry Collector over TCP+TLS, TCP, or UDP, which relays them to the Mezmo OTEL Source. Mezmo ingests Rsyslog data through an OpenTelemetry (OTel) Collector placed between your Rsyslog hosts and Mezmo. Rsyslog forwards logs to the Collector's `syslog` receiver over TCP+TLS, TCP, or UDP, and the Collector relays them to Mezmo's [OpenTelemetry Source](/telemetry-pipelines/open-telemetry-source) with the `otlphttp` exporter. The Collector's `syslog` receiver accepts the Rsyslog default format, [RFC 5424](https://datatracker.ietf.org/doc/html/rfc5424) and [RFC 3164](https://datatracker.ietf.org/doc/html/rfc3164), for auto parsing. ``` Rsyslog (omfwd) --> OpenTelemetry Collector (syslog receiver) --> Mezmo OTEL Source (otlphttp exporter) ``` ## Set Up the OpenTelemetry Collector 1. Log in to the [Mezmo Web App](https://app.mezmo.com), create (or open) a Pipeline, and add an [OpenTelemetry Source](/telemetry-pipelines/open-telemetry-source). Note the ingestion key — you'll use it as `apikey`. 2. Download the `otelcol-contrib` distribution, which includes the `syslog` receiver, from [the OpenTelemetry website](https://github.com/open-telemetry/opentelemetry-collector-releases). 3. Create a `config.yaml`: ```yaml theme={null} receivers: syslog: protocol: rfc5424 tcp: listen_address: "0.0.0.0:514" # tls: # cert_file: /path/to/cert.pem # key_file: /path/to/key.pem exporters: otlphttp/mezmo: endpoint: "https://logs.mezmo.com/otel" compression: gzip headers: apikey: "" service: pipelines: logs: receivers: [syslog] exporters: [otlphttp/mezmo] ``` Uncomment the `tls` block to require TCP+TLS, or replace the `tcp` block with a `udp` block to accept UDP instead. 4. Start the Collector: ```bash theme={null} ./otelcol-contrib --config /path/to/config.yaml ``` ## Point Rsyslog at the Collector Configure Rsyslog's forwarding output (`omfwd`) to send to the host and port where the Collector's `syslog` receiver is listening, instead of directly to Mezmo. For example, in `/etc/rsyslog.d/22-mezmo.conf`: ```none theme={null} *.* action(type="omfwd" target="" port="514" protocol="tcp") ``` If possible, we highly recommend setting up a keepalive inside your rsyslog forwarding configuration. This helps make sure that bad connections are properly terminated and re-initiated, and increases the reliability of log delivery. You can learn about rsyslog keepalive options [here](https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html#keepalive). ## Tagging Hosts Authentication now happens between the Collector and Mezmo, via the `apikey` header on the `otlphttp` exporter, rather than being embedded in each Syslog line. As a result, the legacy `logdna@48950` STRUCTURED-DATA tagging format is no longer needed. To group or tag hosts, add a processor (such as an [`attributes` or `resource` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor)) to your Collector's logs pipeline, and use fields the `syslog` receiver already extracts from RFC 5424/3164 messages, such as `HOSTNAME` and `APP-NAME`, to set attributes for grouping in Mezmo. # SaltStack Source: https://docs.mezmo.com/docs/ingestion-integrations/saltstack Set up the Mezmo Salt deployment integration to listen for Salt state events and send the event information to Mezmo. The Mezmo Salt deployment integration listens for your Salt state events and sends the event information to Mezmo. ## Set Up SaltStack Log Ingestion Follow the instructions in the Mezmo Web App to set up SaltStack log ingestion using your Mezmo ingestion key. 1. Log in to [the Mezmo Web App](https://app.mezmo.com/account/signin). 2. In the bottom section of the left-hand navigation, click **Help**. 3. Select **Add Log Sources**. 4. Under **Via platform**, click **SaltStack**. 5. Follow the instructions to set up SaltStack log ingestion. Note that your Mezmo ingestion key is automatically inserted into the configuration code. If you select the **Salt** app inside the [All Apps filter menu](/docs/search-and-filter)), you can see the states applied across all your hosts. Filtering by a specific source will only show Salt states applied to that particular host. # Syslog Source: https://docs.mezmo.com/docs/ingestion-integrations/syslog Route Syslog log ingestion to Mezmo through an OpenTelemetry Collector, which parses Syslog with RFC 5424 and RFC 3164 support and forwards it to the Mezmo OTEL Source. Mezmo ingests Syslog data by placing an OpenTelemetry (OTel) Collector between your Syslog senders and Mezmo. The Collector receives Syslog over TCP or UDP with its `syslog` receiver, then forwards the parsed logs to Mezmo's [OpenTelemetry Source](/telemetry-pipelines/open-telemetry-source) using the `otlphttp` exporter. ``` Syslog sender --> OpenTelemetry Collector (syslog receiver) --> Mezmo OTEL Source (otlphttp exporter) ``` ## Set Up Syslog Log Ingestion 1. Log in to the [Mezmo Web App](https://app.mezmo.com), create (or open) a Pipeline, and add an [OpenTelemetry Source](/telemetry-pipelines/open-telemetry-source). Note the ingestion key — you'll use it as `apikey`. 2. Download the `otelcol-contrib` distribution, which includes the `syslog` receiver, from [the OpenTelemetry website](https://github.com/open-telemetry/opentelemetry-collector-releases). 3. Create a configuration file `config.yaml`: ```yaml theme={null} receivers: syslog: protocol: rfc5424 tcp: listen_address: "0.0.0.0:514" exporters: otlphttp/mezmo: endpoint: "https://logs.mezmo.com/otel" compression: gzip headers: apikey: "" service: pipelines: logs: receivers: [syslog] exporters: [otlphttp/mezmo] ``` Set `protocol` to `rfc5424` or `rfc3164` depending on the format your Syslog sender uses, and swap the `tcp` block for a `udp` block if you're sending over UDP instead. Consult your local Syslog man page and the [Collector `syslog` receiver documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/syslogreceiver) for full configuration details. 4. Point your Syslog senders at the host and port where the Collector's `syslog` receiver is listening. 5. Start the Collector: ```bash theme={null} ./otelcol-contrib --config /path/to/config.yaml ``` Logs will begin appearing in the Mezmo Log Viewer once the Collector starts forwarding data. # Syslog-ng Source: https://docs.mezmo.com/docs/ingestion-integrations/syslog-ng Forward syslog-ng logs to an OpenTelemetry Collector over TCP+TLS, TCP, or UDP, which relays them to the Mezmo OTEL Source, including ready-to-use configuration examples. Mezmo ingests syslog-ng data through an OpenTelemetry (OTel) Collector placed between your syslog-ng hosts and Mezmo. syslog-ng forwards logs to the Collector's `syslog` receiver over TCP+TLS, TCP, or UDP, and the Collector relays them to Mezmo's [OpenTelemetry Source](/telemetry-pipelines/open-telemetry-source) with the `otlphttp` exporter. The Collector's `syslog` receiver accepts the syslog-ng default format, [RFC 5424](https://datatracker.ietf.org/doc/html/rfc5424) and [RFC 3164](https://datatracker.ietf.org/doc/html/rfc3164), for auto parsing. ``` syslog-ng (destination) --> OpenTelemetry Collector (syslog receiver) --> Mezmo OTEL Source (otlphttp exporter) ``` ## Set Up the OpenTelemetry Collector 1. Log in to the [Mezmo Web App](https://app.mezmo.com), create (or open) a Pipeline, and add an [OpenTelemetry Source](/telemetry-pipelines/open-telemetry-source). Note the ingestion key — you'll use it as `apikey`. 2. Download the `otelcol-contrib` distribution, which includes the `syslog` receiver, from [the OpenTelemetry website](https://github.com/open-telemetry/opentelemetry-collector-releases). 3. Create a `config.yaml`: ```yaml theme={null} receivers: syslog: protocol: rfc5424 tcp: listen_address: "0.0.0.0:514" # tls: # cert_file: /path/to/cert.pem # key_file: /path/to/key.pem exporters: otlphttp/mezmo: endpoint: "https://logs.mezmo.com/otel" compression: gzip headers: apikey: "" service: pipelines: logs: receivers: [syslog] exporters: [otlphttp/mezmo] ``` Uncomment the `tls` block to require TCP+TLS, or replace the `tcp` block with a `udp` block to accept UDP instead. 4. Start the Collector: ```bash theme={null} ./otelcol-contrib --config /path/to/config.yaml ``` ## Point syslog-ng at the Collector Configure syslog-ng's `destination` to send to the host and port where the Collector's `syslog` receiver is listening, instead of directly to Mezmo. If possible, we highly recommend setting up a keepalive inside your syslog-ng forwarding configuration. This helps make sure that bad connections are properly terminated and re-initiated, and increases the reliability of log delivery. You can learn about rsyslog/syslog-ng keepalive options [here](https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html#keepalive). ### syslog-ng TCP+TLS with Custom Port Settings ```none theme={null} destination d_otelcol { tcp("" port(CUSTOM-PORT-NUMBER) tls(peer-verify(required-trusted) ca_dir("/etc/ssl/certs"))); }; log { source(s_src); destination(d_otelcol); }; ### END syslog-ng OTel Collector logging directives ### ``` ### syslog-ng TCP with Custom Port Settings ```none theme={null} destination d_otelcol { tcp("" port(CUSTOM-PORT-NUMBER)); }; log { source(s_src); ### this could be (s_sys) for RedHat/CentOS destination(d_otelcol); }; ### END syslog-ng OTel Collector logging directives ### ``` ### syslog-ng UDP ```none theme={null} destination d_otelcol { udp("" port(CUSTOM-PORT-NUMBER)); }; log { source(s_src); ### this could be (s_sys) for RedHat/CentOS destination(d_otelcol); }; ### END syslog-ng OTel Collector logging directives ### ``` ## Tagging Hosts Authentication now happens between the Collector and Mezmo, via the `apikey` header on the `otlphttp` exporter, rather than being embedded in each syslog-ng line. As a result, the legacy `logdna@48950` STRUCTURED-DATA tagging format is no longer needed. To group or tag hosts, add a processor (such as an [`attributes` or `resource` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor)) to your Collector's logs pipeline, and use fields the `syslog` receiver already extracts from RFC 5424/3164 messages, such as `HOSTNAME` and `APP-NAME`, to set attributes for grouping in Mezmo. # Vector Source: https://docs.mezmo.com/docs/ingestion-integrations/vector Ingest observability data into Mezmo from Vector pipelines by installing Vector and configuring the Mezmo output sink. Mezmo ingests observability data from Vector pipelines through a Mezmo output sink. To set up the Vector integration, you will need to [install Vector on the platform](https://vector.dev/docs/setup/installation/) you want to monitor if you haven’t already done so, then [set up a Mezmo Sink Configuration file](https://vector.dev/docs/reference/configuration/sinks/logdna/). # Vercel Source: https://docs.mezmo.com/docs/ingestion-integrations/vercel Stream Vercel logs to Mezmo by pointing the Vercel Syslog integration at an OpenTelemetry Collector, which relays them to the Mezmo OTEL Source. Vercel has created [a Mezmo integration](https://vercel.com/integrations/logdna) that you can set up as a Vercel Add-on to start streaming Vercel logs. Vercel's integration sends Syslog to a URL you configure, so point it at an OpenTelemetry (OTel) Collector running a `syslog` receiver. The Collector then relays the logs to Mezmo's [OpenTelemetry Source](/telemetry-pipelines/open-telemetry-source) with the `otlphttp` exporter. ``` Vercel (Syslog integration) --> OpenTelemetry Collector (syslog receiver) --> Mezmo OTEL Source (otlphttp exporter) ``` ## Set Up the OpenTelemetry Collector 1. Log in to the [Mezmo Web App](https://app.mezmo.com), create (or open) a Pipeline, and add an [OpenTelemetry Source](/telemetry-pipelines/open-telemetry-source). Note the ingestion key — you'll use it as `apikey`. 2. Download the `otelcol-contrib` distribution, which includes the `syslog` receiver, from [the OpenTelemetry website](https://github.com/open-telemetry/opentelemetry-collector-releases). 3. Create a `config.yaml` and run the Collector somewhere reachable from Vercel's infrastructure over the public internet: ```yaml theme={null} receivers: syslog: protocol: rfc5424 tcp: listen_address: "0.0.0.0:514" exporters: otlphttp/mezmo: endpoint: "https://logs.mezmo.com/otel" compression: gzip headers: apikey: "" service: pipelines: logs: receivers: [syslog] exporters: [otlphttp/mezmo] ``` 4. Start the Collector: ```bash theme={null} ./otelcol-contrib --config /path/to/config.yaml ``` ## Configure the Vercel Integration 1. In the Vercel dashboard, add the [Mezmo integration](https://vercel.com/integrations/logdna) as an Add-on. 2. When prompted for a Syslog URL, enter the address and port where your OTel Collector's `syslog` receiver is publicly reachable. Be aware that it can take up to 30 minutes before your logs start arriving after the integration is configured. Check the Collector's logs if you're having issues receiving data from Vercel. # Generate Service and Ingestion Keys Source: https://docs.mezmo.com/docs/ingestion-key Create and manage Mezmo ingestion keys and IAM access keys (personal, service, and enterprise), with security best practices for handling tokens. Generation 1 service keys are being deprecated in favor of Generation 2 IAM access keys. Gen 2 access keys can clearly be identified by its well known prefix format. Support for generation 1 Service keys will be removed January 31, 2026. You use personalized, private service and ingestion keys associated with your account to connect Mezmo to third-party applications and services. They enable users, and programs such as log collectors to authenticate with Mezmo without requiring you to share your account details. Anyone who has access to your service and ingestion keys can send or retrieve logs to or from your account with no additional authentication. Be sure to keep your API keys secret. ### Security Best Practices * Rotate tokens regularly. Use expirations and rotate before they expire. * Grant only what you need. Prefer minimal scopes. * Use service accounts for automation. Avoid personal tokens in CI/CD. * Store tokens in environment variables or a secret manager. Do not hard‑code tokens. ```bash theme={null} # Good export MZM_ACCESS_KEY="sta_34b8f6897cd3396e6af781c3bfe34065b690f90b" curl -H "Authorization: Token $MZM_ACCESS_KEY" https://api.mezmo.com/v3/pipeline # Avoid curl -H "Authorization: Token sta_34b8f6897cd3396e6af781c3bfe34065b690f90b" \ https://api.mezmo.com/v3/pipeline ``` *** There are two primary types of keys you can manage, Ingestion Keys and IAM Access Keys. ### Ingestion Keys Ingestion keys are used by log collectors like the Mezmo Agent to send log data to Mezmo, and are also used in commands for the Ingest API. You can have up to 10 ingestion keys active at a time. ### IAM Access Keys Identity and Access Management (IAM) access keys represents a significant step forward in enhancing the security of your interactions with our services. IAM access tokens offer several key advantages over the Generation 1 service keys, including: * **Enhanced Security:** IAM access keys provide more granular control over permissions and integrate with advanced security features, reducing the risk of unauthorized access. * **Improved Auditing Capabilities**: IAM Access keys offer enhanced auditing over legacy service keys, detailing *who*, *what*, *when*, and *where* actions occur. This improves security breach identification, suspicious activity investigation, and audit trails for compliance, offering clearer visibility for proactive security and efficient incident response. * **Improved Flexibility:** The new access key system allows for more flexible and dynamic management of access rights, enabling you to manage your integrations with greater precision. * **Future-Proofing:** This change aligns with industry best practices for secure access management, ensuring that our security infrastructure remains robust and adaptable to evolving threats. There are three distinct types of IAM Access Keys, each of which can be identified by a unique prefix | Prefix | Key Type | Example | | ------ | -------------------------- | ---------------------------------------------- | | `sta_` | Personal Access Key | `sta_1a2b3c4d5e6f7890abcdef1234567890abcdef12` | | `sts_` | Service Account | `sts_9876543210fedcba0987654321fedcba09876543` | | `ste_` | Enterprise Service Account | `ste_d710b57dc7dedde45ccc4c35b68cf385b89f8dc3` | *** #### Personal Access Keys (`sta`) Personal Access Keys will be generally available End of Q1 2026 Personal Access Keys provide user-specific authentication for API operations. These keys are tied to individual user accounts and inherit the permissions of the user who created them. Personal Access Keys can be used for: * Accessing APIs with user-level permissions * Automating tasks that require user authentication * Integrating with third-party tools and services * Performing operations within the scope of the user's access Personal Access Keys are scoped to the user's permissions and cannot exceed the access level of the user the access token is associated with. Individual users looking to explore the Mezmo Platform APIs should be encouraged to create Personal Access Keys to do so. #### Service Accounts (`sts`) A service account is a non-user identity that has its own key and its own permission set. Service accounts are associated with a Service Account Key for automation and CI/CD. These tokens are not associated with a normal user with in your organizations and as such, its level of access is only limited by the access it is granted. For this reason it is highly recommended that the ability to create service account and keys be reserved for account administrators. You can have up to 50 service accounts active at a time. #### Enterprise Service Accounts (`ste`) Enterprise Service Accounts and their access keys are associated with an enterprise rather than an individual Mezmo account. These types of access keys are enabled to perform operations and interactions across many accounts in an effort to streamline and optimized account management for large customers who may have many dozens to hundreds of accounts. Enterprise Service Accounts are only available through our enterprise dashboard. *** ## Access and Generate New Keys 1. Log in to [the Mezmo Web App](https://app.logdna.com/account/signin). 2. Go to **Settings >** **Organization > API Keys**. 3. To generate additional ingestion keys, click **Generate Ingestion Key** for up to a total of 10 keys. 4. To generate additional service accounts, click **Create Service Account** for up to a total of 50 keys. 5. Remove an ingestion key by clicking the X next to it. Note that any applications actively using this key will no longer be able to send logs to your account Image ## Manage Service Accounts and Keys The keys table and its actions apply to your IAM Access Keys: personal access tokens, service accounts, and enterprise service accounts. Manage ingestion keys with the steps in [Access and Generate New Keys](#access-and-generate-new-keys) above. After you create a key, it appears in a table with **Name**, **Access**, **Created**, and **Actions** columns. The **Name** column shows only a masked trailer of the secret, displayed as `····`, so the full key value never appears in the table. The **Access** column lists the key's assigned roles or permissions. From the **Actions** column, the row actions available to you depend on the key type: * **Rename** is available for all key types. * **Edit permissions** is available for service accounts and enterprise service accounts, and only when role-based access management is enabled for your organization. It is not available for personal access tokens. * **Rotate** is available for service accounts only. It is not available for enterprise service accounts or personal access tokens. * **Delete** is available for all key types. When you rotate a key, you confirm the action, then the current key stops working immediately and a new key is generated and shown to you once. Rotating or deleting a key takes effect immediately and cannot be undone. Any integration or automation still using the old key stops working until you update it with the new key. After you create or rotate a key, the full key value appears exactly once in a window with a copy control. It is not shown again, so copy and store it right away. This applies to all key types. The following table summarizes what each key type supports: | Capability | Personal Access Tokens | Service Accounts | Enterprise Service Accounts | | ---------------- | ---------------------- | ---------------- | --------------------------- | | Create | Yes | Yes | Yes | | Rename | Yes | Yes | Yes | | Edit permissions | No | Yes | Yes | | Rotate | No | Yes | No | | Delete | Yes | Yes | Yes | | One-time reveal | Yes | Yes | Yes | For information on using IAM Access Keys to interact with the Mezmo platform APIs, see: [Authenticating With The API](/docs/api#authenticating-with-the-api) # Mezmo Logging Agent Source: https://docs.mezmo.com/docs/introducing-the-agent Learn how the Rust-based Mezmo Agent reads and uploads log files over HTTPS, plus configuration, host tags, hostnames, and release notes. The Mezmo Agent reads log files from the host where it is installed, and uploads the log data to Mezmo via a secure HTTPS connection, where it is parsed and processed according to the parameters set in the Mezmo Agent configuration file. The Mezmo Agent runs on Kubernetes, Openshift, Linux, and Windows. The Mezmo Agent is written in Rust, and uses the Linux kernel to monitor the log files and directories for changes, rather than having to poll these files constantly. This implementation frees up CPU utilization and improves stability. As of 5/1/2025, versions below 3.8 will be deprecated. Deprecated agents should continue to work, but Mezmo will only debug issues with ingestion on supported versions. ### GitHub Repository For complete information about installing and configuring the Mezmo Agent, check out the source code and documentation in our GitHub repository. [https://github.com/logdna/logdna-agent-v2](https://github.com/logdna/logdna-agent-v2) ### Default Configuration File You can find the default configuration `yaml` file in the Agent directory at `/etc/logdna/config.yaml`. You can find more information in [the Configuration section](https://github.com/logdna/logdna-agent-v2#configuration) of the GitHub documentation. ## Agent 1.x-2.x (Deprecated) GitHub Repository For complete information about installing and configuring the older Mezmo Agent, check out the source code and documentation in our GitHub repository. [https://github.com/logdna/logdna-agent](https://github.com/logdna/logdna-agent) The certificates in versions \<2.2.1 have expired, and it is not possible to update the certificate model used in these versions. Mezmo recommends that you to use latest version of version 2, or agent versions above 2.2 on version 1. You can find more information[ in this blog post](https://www.mezmo.com/blog/announcing-mezmo-agent-3-6-ga-windows-support-and-updated-agent-deprecation-schedule). Starting from version 3.6 GA, we recommend upgrading to version 3.x, which is a more performant Agent that can handle all the types of logs supported by Version 1. ## Mezmo Agent FAQS ### How do I tell the Mezmo Agent what to log? By default, the Mezmo Agent automatically logs all .log and extensionless files located under /var/log/, but if you want to log other directories or files, you can use these commands in a host terminal to specify additional directories or files: For directories: ```bash theme={null} sudo LogDNA-agent -d "/path/to/my/logs" ``` For files: ```bash theme={null} sudo LogDNA-agent -f "/path/to/my/logfile" ``` If you need more complex logic, you can also view and set specific logging paths, as well as use glob patterns by editing `/etc/LogDNA.conf` or /etc/LogDNA.env\`, depending on the agent version. ### Is the Mezmo Agent open source? Yes! You can view, and contribute to, the source code on GitHub for both Agentv1 and Agent v2, and even build the agent yourself. We also love the open source community, so please feel welcome to submit PRs or report any issues you find. Check out our contributing guide for more info. ### How do I use host tags? Host tags let you automatically group hosts into dynamic host groups without having to explicitly assign a host to a group within the Mezmo web app. To use tags, make sure the agent is installed on your host, and use this command to add a tag or tags: ```bash theme={null} sudo LogDNA-agent -t mytag,myothertag,anothertag ``` You can also edit the LogDNA configuration file, `etc/LogDNA.conf` or /etc/LogDNA.env, depending on the agent version, and specify the tags there. Even if your hostname or host machine changes, as long as the agent is running with the same tags configured, that host will automatically be added to the dynamic group located under the **Hosts** filter. ### How do I override the default hostname? The Mezmo agent automatically uses the machine's default OS hostname. If you wish to use a different hostname, you can edit the LogDNA configuration file, `/etc/LogDNA.conf` or /etc/LogDNA.env\`, depending on the agent version, and specify the hostname there. ### Why Does the Mezmo Agent v1 need root access? The Mezmo Agent v1 works by monitoring changes in local log files (read only access) and sending new lines to the designated ingestion endpoint. These are the primary reasons why Mezmo's agent requires root access: 1. The agent listens to kernel-level file events to detect new lines in log files. 2. The agent monitors default log file paths, such as `/var/log` or `/var/data`. 3. Container-based frameworks such as Kubernetes also use these file paths. Kubernetes centralized logging, which writes all container logs to `/var/log`, recommends using a DaemonSet and node-level logging, which is how we implemented the Mezmo agent. # View Kubernetes Events and Metrics Source: https://docs.mezmo.com/docs/kubernetes-enrichment Use Kubernetes Enrichment to view cluster events, pod and node metrics, and logs in the Log Viewer, and tag lines to distinguish clusters. Mezmo Kubernetes Enrichment centralizes Kubernetes events, resource metrics, and logs behind a single pane of glass to enable end-to-end visibility into your Kubernetes cluster. With Kubernetes Enrichment, you can troubleshoot deployment issues from within the Log Viewer, and view and set alerts for events like crashing pods or failing health checks in your production environments. ## Kubernetes Events Knowing what is happening in your cluster alongside of your application log lines can be a valuable tool in debugging problems in your deployments. With the Mezmo Agent, you can now do this. To set this up in your cluster, head over to our [setup instructions on GitHub](https://github.com/logdna/logdna-agent-v2/blob/master/docs/KUBERNETES.md#enabling-k8-events). ## Kubernetes Metrics Reporter Enriched Lines This feature is only available for customers on an Enterprise plan, and is restricted to **Owner** and **Admin** user roles. Please contact your Customer Support Manager or [support@mezmo.com](mailto:support@mezmo.com) for more information. The previous Mezmo Reporter that enabled metrics collection will soon be deprecated. We recommend all users to upgrade to the latest 3.7+ Mezmo Agent with built-in metrics support. If you would like to see more detailed statistics about the pod your application is running on, edit your YAML config to enable metrics collection. With this feature, you can correlate performance issues with log lines that may be causing those issues. Please note that non-enterprise customers will still incur the costs of storing metrics, but without the ability to see the metrics in Log Viewer. After you enabling this feature, you can start viewing Kubernetes metrics in the Log Viewer. These metrics and metadata all show a point-in-time capture of the Kubernetes cluster state when that log line was sent, so even if an event happened a few days ago, the metrics will still reflect what was happening during that event. 1. In the Mezmo Web App, under **Views**, select a view that contains lines from your Kubernetes cluster. 2. Click the arrow next to a log line to open the information pane for that line. 3. Click the arrow next to **Kube Stats** to view metrics including CPU usage and memory for the pod and node associated with the log entry. All metrics have the same retention as your log retention, and metrics are collected every 30 seconds. Image Image Other information you can view includes: * Events associated with the log line * Line identifiers * Labels * Diagnostics ## Cluster Tag to Distinguish between Multiple Clusters When you are sending log lines to Mezmo from multiple similarly named namespaces and pods you will need to add a value to the `MZ_TAGS` environment variable. Please add a tag that is prefixed `k8sclusterkey` (for example `k8sclusterkey_prod-clstr` where `prod-clstr `is a unique cluster name). This will allow enrichment to distinguish between the clusters when the nodes are similarly named. This will also add a new line identifier called `Cluster` to your log lines that you can search for. # Log Ingestion Delays Source: https://docs.mezmo.com/docs/log-ingestion-delays Understand the two types of ingestion delays in Mezmo, Live Tail latency and indexing latency, and the typical timing for each. Periodically there will be delays in processing new log data due to the volume of incoming logs. This topic describes the two types of delays you may experience during log ingestion: **Live Tail Latency** and Indexing **Latency** . ## Live Tail Latency Mezmo aims for an operational standard of 1s for Live Tail latency. The typical average for Live Tail latency is approximately 10s. You can view the current latency at [https://status.mezmo.com](https://status.mezmo.com). ## Indexing Latency Indexing refers to the time between when a log line is ingested, and when it's available for search. By default, the Mezmo indexing process updates every 30s. As soon as a line is available in Live Tail, it should be indexed by Mezmo within 30s. # Automatically Parsed Log Line Components Source: https://docs.mezmo.com/docs/log-parsing Learn how Mezmo automatically parses log lines into string, source, application, JSON, and metadata components for search and analysis. As Mezmo [ingests your logs](/docs/ingestion), it automatically parses information from your log lines, including string components, source information, application information, JSON objects, and user-specified metadata. You can then use [Mezmo search features](/docs/search-and-filter) to analyze data in your logs. This topic describes the various types of information that Mezmo parses, along with notes on how it is parsed. [Parse Logs with Custom Templates](/docs/parse-logs-with-custom-templates) contains additional information about using custom parsing templates. You can identify parsed lines in Views by selecting a line and viewing the data. Image If your parsed fields contain inconsistent value types, field parsing may fail, but the line will be preserved if possible. For example, if a line is passed with a `meta` object, such as `meta.myfield` of type `String`, any subsequent lines with `meta.myfield` must have `String` as the value type. This applies to all parsed fields, including JSON. ## Log Line String Components Most log line strings contain three components: **Message**, **Timestamp**, and **Log Level**. ### Message Message is a string that represents the core descriptive component of a log line. It is usually preceded by timestamp and log level. A message typically contains a mixture of static and variable substrings, and is human-readable. For example, `User myemail@email.com requested /API/accountdetails/` ### Timestamp Timestamp is required for all ingested log lines. For Mezmo log ingestion to correctly parse a timestamp, it should follow [the ISO 8601 format](https://www.iso.org/iso-8601-date-and-time-format.html). ### Log Level Log level typically follows timestamp and is automatically parsed. Mezmo log ingestion parses common log level formats, such as a timestamp followed by a separator followed by the log level. Common log levels include: * `CRITICAL` * `DEBUG` * `EMERGENCY` * `ERROR` * `FATAL` * `INFO` * `SEVERE` * `TRACE` * `WARN` * `ALERT` * `IP address` * `MAC address` ## Source Information Metadata Mezmo also parses source information metadata from log lines, which is listed in the **All Sources** menu in the web app. The only required parameter is **hostname**. ### Hostname A hostname is the name of the log line source, and is automatically parsed by the [Mezmo Logging Agent](/docs/introducing-the-agent), as well as [Syslog based ingestion](/docs/ingestion-integrations/syslog). However, when you are sending log lines for ingestion with the REST API or a [code library](/docs/code-libraries), you must specify the host name. ### Tags You can use a tag to group lines, and more than one tag can be applied to a single line. Tags are listed in the **All Tags** menu in the web app. Tagging is supported by both the [Mezmo Logging Agent](/docs/introducing-the-agent) as well as custom-template supported [Syslog based ingestion](/docs/ingestion-integrations/syslog) such as rsyslog or syslog-ng. ### Other information Other optional source information includes: * IP address * MAC address The [Mezmo Logging Agent](/docs/introducing-the-agent) automatically parses this information, and you specify it for the REST API. The Mezmo Agent also parses some instance metadata, such as instance type. ## Application Information Metadata In addition to source information, Mezmo can also parse application information from log lines. The [Mezmo Logging Agent](/docs/introducing-the-agent) automatically parses the application name as the filename (for example: `error.log`) while [Syslog based ingestion](/docs/ingestion-integrations/syslog) uses the syslog-generated `APP-NAME` tag. For the REST API and [code library](/docs/code-libraries), you must specify the app name. ## Automatic and Custom Parsing for Field Search Mezmo automatically parses certain types of log lines that enable the use of field search for those lines. ## JSON Parsing Be aware that the size of sent log data can increase after the JSON string is parsed in Node.js. Measurement is based on how much data is ingested into Mezmo, after it is parsed as JSON, and not how much data is sent in a line. Messages that end in a curly brace, `}` are parsed even if the JSON doesn't contain the entire message. If you don't want your JSON to be parsed, add an additional character after the ending curly brace such as a period. If your JSON has a `message` field, it will be used for display and search in the log viewer. We also parse out, and override any existing, log levels if you include a `level` field. ### Reserved and Protected Fields In parsed JSON lines, there are reserved fields to keep track of specific types of data. They can be identified by the prepended underscore(\_). Using the reserved fields in your root JSON object will result in an underscore (\_) prepended to those fields inside the context menu, for example `status` is stored as `_status`. Common reserved fields: * `_source` * `_type` * `_tag` * `_auth` * `_bytes` * `_connect` * `_method` * `_namespace` * `_path` * `_pod` * `_request` * `_response` * `_service` * `_space` * `_status` * `_timestamp` * `_user` Protected field names cannot be used in your object, and are removed by Mezmo when encountered. The protected field names are: * `_account` * `_retention` ### Mezmo Reserved Fields Fields with the annotation \_`mezmo_` are reserved for Mezmo-specific data. `_mezmo_line_size` Indicates the number of bytes attributed to a log line. You can view a line's size by clicking on it in the Log Viewer. ## Metadata Metadata is a field reserved for custom information associated with a log line. Sending metadata is currently supported by the [Ingestion REST API](/docs/api-referencesk9h#ingest), as well as our [Node.JS ](https://github.com/logdna/logger-node/blob/main/README.md)and [Python](https://github.com/logdna/python/blob/master/README.md) code libraries. ## Parsed Log Sources Mezmo parses lines from these sources: * Akamai * Ansible * Apache * Aptible * AWS CloudWatch * AWS ELB * AWS ECS * AWS S3 * Cron * Docker Swarm * Docker Cloud/Compose * GitHub * Golang * HAProxy * Heroku * HTTPD * IIS Log * JSON * Logfmt * LogSpout * Rancher * MongoDB * Nagios * Nginx * PostgreSQL * Redis * Ruby/Rails * Syslog * Tomcat * Windows Events # Manage Access Source: https://docs.mezmo.com/docs/manage-access Manage users, permissions, discoverability, sign-in policies, two-factor authentication, and password rules for your Mezmo organization. Adding users to your Mezmo Organization allows you to share access to logs, views, alerts, and other Mezmo resources with other members of your team. This page will show you how to manage users as a Mezmo administrator, including how to add new users, assign roles, and set global sign-in policies. To see available security features, go to [**Settings > Organization > Security**](https://app.mezmo.com/manage/team-settings). ## Access Control Access Control is on by default. While it is on, non-admin users (Basic and Read-only members) without access scopes can see all logs. Turn it off to prevent those users from seeing any logs until they hold a role that grants access scopes. You can also control what logs members see by using [Role-Based Access Control](/docs/rbac). ## Discoverability Discoverability lets you control how users can find and join your organization. * **Discover** - Members on the same domain can find and ask to join the organization. Discover is enabled by default. * **Join** - Members on the same domain can join the organization without a request. Your domain is determined by the email address you used when creating your Mezmo account. If you would like to add domains, please contact Mezmo support. ## Sign-in Policy Sign-in policy determines which authentication methods members are allowed to use when signing into your Organization. These options are turned off by default. You can use: * **SAML Sign-in** - Let members sign in using an identity provider such as Okta or OneLogin. Available for enterprise customers only. Learn how to set up [SAML SSO](/docs/enterprise-saml-sso#onelogin-setup). * **Idle Logout** - Log members out after a period of inactivity. Available for all plans. * **Redirect after logout** - Redirect members to an address after they logout. Available for all plans. ## Two-Factor Authentication You can set two-factor authentication for your own profile through the **User Preferences** menu. We support two-factor authentication only for Mezmo logins. For Google, GitHub, or Heroku, please set up two-factor authentication via their platform. 1. Log in to [the Mezmo web app](https://app.mezmo.com/). 2. Go to **Profile > User Preferences**. 3. Click **Two-factor auth**. 4. Click **Enable**. ## Password Guidelines Your account password must contain: * Eight or more characters * One lowercase character * One uppercase character * One number You password must not contain: * Your first name * Your last name * Your username You also cannot use any of your prior three passwords. # Manage Enterprise Organizations Source: https://docs.mezmo.com/docs/manage-enterprise-organizations Use Enterprise Organization features to centrally manage access control, usage, and quotas across multiple Mezmo organizations from one dashboard. This feature is only available for customers on an Enterprise plan, and is restricted to **Owner** and **Admin** user roles. Please contact your Customer Support Manager or [support@mezmo.com](mailto:support@mezmo.com) for more information. The Enterprise Organization features help Enterprise customers manage access control, and view usage and quotas for multiple organizations from one central dashboard. If you have multiple accounts to manage, the Enterprise Organization features will make life easier for your account administrators. # Manage Members Source: https://docs.mezmo.com/docs/manage-members Invite, edit, and remove members, assign roles, and handle join requests for your Mezmo organization's team. Each Mezmo Organization allows for a certain number of users determined by your [plan](https://www.mezmo.com/pricing). Users who join your Mezmo Organization are called Members, and collectively they are your Team. Organization administrators can: * Add and remove Organization members * Assign roles and privileges to members * Limit each member's access to Mezmo resources * Decide how Mezmo users can discover and join your Organization * Toggle different methods of signing into your Organization ## Members The Members screen shows each Mezmo Organization member with their email and [roles](/docs/rbac). The **Roles** column shows a role-name badge for each role a member holds, and the owner shows an **Owner** badge. You can also see how many people your plan lets you have on your team. From the [Roles](/docs/rbac) page, you can open this page filtered by member or by role. ## Invite Members When you invite a member, Mezmo sends an email invitation to the address provided prompting the user to either create a Mezmo account or sign in using [SSO](/docs/enterprise-saml-sso). To add a member: 1. Go to [Members](https://app.mezmo.com/manage/team) and click **Invite Member**. 2. Enter the email address. 3. Select one or more [roles](/docs/rbac) from the roles selector. At least one role is required. The default role is **Basic**, and you can't select the **Owner** role. 4. Click **Invite Member** to send the invitation. ## Edit a Member The same modal handles inviting and editing members. Open the edit view from the inline **Edit** (pencil) action on the member row. You can edit a member at any time, including while they have an outstanding invitation. The email address is read-only when editing. Change a member's roles through the roles selector. At least one role is required to save. ## Remove a Member To remove a member from your Organization, click **Remove** on the member row. This won't delete the user's account, but it revokes their membership to your Organization. ## Manage Join Requests If your Organization is [discoverable](/docs/manage-access), users who aren't currently members of your Organization can ask to join it. When this happens, the email addresses configured to receive join request notifications will receive an email with links to either approve or deny the request. You can also use the Join Requests screen to review outstanding requests. # Manage Usage Source: https://docs.mezmo.com/docs/manage-usage Explore tools to reduce and monitor log data usage, including the Usage Dashboard, trends, restoration, email digests, and usage alerts. ## Ways to Manage Usage There are many ways to reduce and manage the amount of logging data used by your organization. * [Exclusion Rules](/docs/exclusion-rules) - Use Exclusion Rules to control what's stored. * [Index Rate Alerts](/docs/index-rate-alerts-overview) - Track sudden spikes in logging and receive alerts if logs exceed a certain threshold. * [Usage Quotas](/docs/manage-usage-quotas) - Specify the amount of log data to store. Create alerts once storage limits are reached. * [Variable Retention Rules](/docs/variable-retention) - Enterprise Only. Define how long logs are kept based on queries. These logs do not follow any other log retention rules set. * Shut Off - Stop all incoming data. * [Usage API](/api-reference/usage-v2/retrieve-account-usage-totals) - Get information about your apps aggregated usage. ## Usage Dashboard The [Usage Dashboard](https://app.mezmo.com/manage/ingestion) provides an overview of how much data your organization is ingesting. It shows: * Estimated usage for the given month. * Breakdown of data ingested by day. * Last Month in Days - Amount of data ingested in the past 30 days. * Last month in Days, Trends - Shows the top sources, apps, and tags that create the most logs. 📘 The Usage Dashboard only shows data for the 50 most popular apps, sources, and tags each day. ## Last month in Days – Trends The Last month in Days – Trends graph lets you show data in a stacked or unstacked view and compare the total usage. Image ### Stacked and Unstacked Toggling Stacked to on, will stack the data lines on top of each other and display the sum of all visible sources. The graph will also be updated to show how each app, source, or tag compares to the total. Image Image ## Log Data Restoration Restoration is a way to bring back, or restore, archived logs from cold storage. You can search for this data in the Mezmo user interface. This is useful for finding old bugs and having more context from older logs. Learn more about [Data Restoration](/docs/data-restoration). ## Email Digest You can subscribe to a weekly or monthly digest that breaks down your usage data for the period. Email digests include: * Total usage for the current billing cycle * Comparison with the previous billing cycle * Top apps, hosts, and tags by usage * Days remaining in the current billing cycle ### Variable Retention Breakdown If your account has Variable Retention rules enabled, the email digest also includes a usage breakdown for each retention tier. For each tier, you'll see the top apps, hosts, and tags that contributed to that tier's usage, along with the retention period. ## Usage Alerts You can set up email alerts to notify you when you reach a percentage of the usage limit. The Usage Limit is the amount you want to use before getting an email alert. The Usage Limit only affects email alerts, and won't cause log ingestion to stop. After defining a Usage Limit, add email recipients who will be notified when a certain usage percentage is reached. You can add as many email addresses as you need. ### How Usage Alerts Work We send Usage Alerts at the end of your billing cycle. For example, if you get billed on the 15th of every month, our counter resets on the 16th to start a new cycle. If this isn't a billing date, such as accounts on a free trial, Mezmo will default to the account creation date. If you have a contract, your custom billing date may not match our system's cycle start date. To find out if your accounts custom billing date and the system's counter dates are aligned, please navigate to the Last month in Days – Usage chart, where the red ticker identifies the date your accounts billing cycle restarts. Image # Manage Usage Quotas Source: https://docs.mezmo.com/docs/manage-usage-quotas Set daily and monthly quotas to stop log storage, reduce volume with triggered exclusion rules, or alert when usage thresholds are reached. Usage Quotas enables developers and technical teams to see and set alarms for all important logs. This allows account administrators to manage how much data their teams are storing. The goal is to store all critical logs while staying under the monthly storage budget. These configuration options allows for fine-grained control, providing a range of options such as: * [Stop the storage](/docs/manage-usage-quotas#stop-log-storage) of all incoming logs once a certain quota is met. * [Reduce the volume](/docs/manage-usage-quotas#reduce-the-volume-of-logs) of logs that are stored using filters, while still being able to stop logs once a quota is met. * [Create alerts](/docs/manage-usage-quotas#alert-only) to notify you that your usage quota threshold is met. ## Stop Log Storage When you reach your monthly or daily log limit, stop storing them. Logs that aren't stored don't add to your bill, but you can still see live tailing and get alerts. ### Setup Stop Log Storage 1. Go to [**Settings > Usage > Usage Quotas**](https://app.mezmo.com/manage/usage-quotas). 2. Click **Edit Usage Quotas**. 3. Set the **Daily Usage Quota** and **Monthly Usage Quota** using GB, TB, or MB. 4. Choose if you want to discard logs with the daily or monthly quota is met. You can also choose to discard the logs daily and monthly once the quotas are reached. 5. Click **Apply** to save changes. Image ## Reduce the Volume of Logs Triggered Exclusion Rules let you be more specific in what you filter out, so you don't reach the usage quota value as quickly. This way, you can focus on the important log data. For example, if you set a Daily Usage Quota threshold at 2GB, and set the percentage at 50%, then at 1GB the system will start applying the triggered exclusion rule and no longer retain logs from the specified sources, apps, or queries. You can use this approach to keep certain logs out of storage, but still see them and set alerts. ### Setup Log Volume 1. Go to [**Settings > Usage > Usage Quotas**](https://app.mezmo.com/manage/usage-quotas). 2. Click **Edit Usage Quotas**. 3. Set the Daily Usage Quota and Monthly Usage Quota using GB, TB, or MB. 4. Don't check either of the discard logs boxes. 5. Click **Apply**. 6. Go to **Triggered Exclusion Rules** and select Daily or Monthly. You can also configure both. 7. On the **Apply this rule at:** percentage selector, use your cursor to select and drag the marker to the percentage point you want. This percentage indicates at what percentage point of your pre-defined Daily or Monthly Usage Quota you want to start applying the triggered exclusion rule. 8. Set the exclusion rules based on Sources, Apps, and Query. You can use one or more of these filters. One filter is required. 9. Select Discard matching log lines if you want to stop retaining logs as soon as the specified percentage is met. 10. Select Preserve for Live Tail and Alerting, If you want to stop retaining all logs as soon as the specified percentage is met, but you do want to still have ingested logs appear in live tail and be able to set alerts on them. 11. Click **Apply** to save changes. 📘 Rule limit: Each account can only have five daily and monthly Triggered Exclusion Rules. Image 📘 Archived Logs: Logs not excluded by rules or quotas won't be archived. ## Alert Only You can choose to only send alerts when the quota is met. This option stores all logs and sends alerts when the quota is met or edited. 1. Go to [**Settings > Usage > Usage Quotas**](https://app.mezmo.com/manage/usage-quotas). 2. Click **Edit Usage Quotas**. 3. Set the Daily Usage Quota and Monthly Usage Quota using GB, TB, or MB. 4. Don't check either of the discard logs boxes. 5. Click Apply. 6. On the Setup page, in the Alert recipients area, specify the notification channels and the recipients. 7. Don't add a Discard Rule. 8. You can also create a triggered exclusion rule, and receive an alert specifically based on the defined rules in addition to the alert for when you reach the overall usage quota. Image ## Remove Usage Rules To remove rules, make sure to set all daily and monthly limits to 0. Then toggle the Usage Quotas off. ## Feature Notes * If a quota value is set lower than what has already been consumed and sent to storage, all incoming logs after the rule has been set will be discarded. * If you change your Daily or Monthly Usage Quotas, it will automatically turn off all the discard rules associated with that quota. For example, if you have a Monthly quota of 100GB and three Daily discard rules based on that quota, changing the Monthly quota to anything other than 100GB will turn off those rules. We do this to prevent changes that could result in logs not being retained. So, after you make the update, check that the discard rules are still what you want. # Mezmo Agent Configuration for Kubernetes Source: https://docs.mezmo.com/docs/mezmo-agent-configuration-for-kubernetes Configure the Mezmo Logging Agent in a Kubernetes cluster by editing the DaemonSet environment variables and applying a custom config file. When using the Mezmo Logging Agent within a Kubernetes cluster, the configuration is a bit different than the standard v2 agent configuration. The only editable file is the `environmental variables `section of the `daemonset` configurations. ## To configure the agent Find the effective config at beginning of agent pod log. It should look like this: ```json theme={null} http: host: logs.logdna.com endpoint: /logs/agent use_ssl: true timeout: 10000 use_compression: true gzip_level: 2 params: hostname: main-mac-ubuntu mac: ~ ip: ~ tags: ~ body_size: 2097152 log: dirs: - /home/dmitri/SOURCE/TMP/root/subdir_missing include: glob: - "*.log" regex: [] exclude: glob: [] regex: [] lookback: start log_metric_server_stats: ~ journald: systemd_journal_tailer: ~ startup: {} ``` Copy this config file and put it in a new text file, use vim `` for example (depending on what text editor you use) Then to this file, you'll add the below params. ```json theme={null} http: params: retrybase delay_ms: 100000 retry_step_delay_ms: 10000 ``` Apply this config file to your Kubernetes cluster. Then m[odify the envs section of the DaemonSet](https://github.com/logdna/logdna-agent-v2#configuring-the-environment) you deployed with the env. variable `LOGDNA`*`CONFIG`*`FILE = ` # Mezmo Agent Supported Platforms Source: https://docs.mezmo.com/docs/mezmo-agent-support-matrix Review the operating systems, architectures, and platform versions supported by the Mezmo Agent, including End of Life (EOL) policies. This article provides information about the supported platforms for our [Mezmo Agents](/docs/introducing-the-agent). ## What is End of Life (EOL)? When a platform reaches its End of Life (EOL), Mezmo will no longer be providing additional fixes or changes specific to that particular OS or platform version. Mezmo adheres to the EOL dates established by the OS/platform maintainers. ## Supported Platforms * Ubuntu ([Ubuntu EOL Lifecycle](https://ubuntu.com/about/release-cycle)) * Debian ([Debian EOL Lifecycle](https://endoflife.software/operating-systems/linux/debian)) * CentOS ([CentOS EOL Lifecycle](https://endoflife.date/centos)) * Red Hat Enterprise Linux/RHEL ([RHEL EOL Lifecycle](https://access.redhat.com/support/policy/updates/errata)) * Kubernetes ([Kubernetes EOL Lifecycle](https://endoflife.date/kubernetes)) * Windows ([Windows EOL Lifecycle](https://learn.microsoft.com/en-us/lifecycle/products/windows-10-enterprise-and-education)) * OpenShift ([OpenShift EOL Lifecycle](https://access.redhat.com/support/policy/updates/openshift)) * Linux architectures available: AMD64 and ARM64 Installation instructions for the specified platform/OS are available on our [Github](https://github.com/logdna/logdna-agent-v2/tree/master?tab=readme-ov-file#managing-deployments). ## Additional Notes * Sending logs via Websockets is no longer supported (EOL November 1, 2020). * ppc64 architecture is not supported. # Mezmo MCP Source: https://docs.mezmo.com/docs/mezmo-mcp Connect AI assistants and IDE chat agents to Mezmo's remote MCP server for root-cause analysis, log queries, and pipeline management. ## Mezmo MCP Server (Model Context Protocol) Mezmo MCP is a **remote Model Context Protocol (MCP) server** that lets AI assistants and IDE chat agents interact with the Mezmo observability platform via the [Model Context Protocol](https://modelcontextprotocol.info/). Use it for streamlined observability, log analysis, and root-cause analysis in your favorite tools. Add Mezmo MCP and you can: * 🕵️ Run advanced **Root-cause analysis** over recent logs * 📦 List, describe, **create, and edit Pipelines** * 🔌 Tap **live pipeline traffic** for inspection * 📤 Export and filter **Logs** with powerful query syntax * 🔎 Review **AI Investigations** for your account ## Overview Mezmo MCP is a remote MCP server that connects AI assistants and IDE chat to Mezmo so you can run advanced root-cause analysis, discover and build pipelines, and export logs without hosting anything yourself. It’s built for observability use cases and works across many popular MCP clients, including [AURA](/aura/quickstart), Mezmo's own agentic SRE harness. ## Available Tools & Examples ### 1. Log Analysis & Root Cause Detection #### `analyze_logs_for_root_cause_relative_time` Fetches, processes (deduplicates, clusters), and analyzes logs to determine potential root causes for incidents within a relative time range. **Use cases:** * Investigating recent incidents * Finding root causes of system failures * Analyzing error patterns **Example:** "Analyze the logs from the last 30 minutes to find the root cause of API service returning 500 errors" #### `analyze_logs_for_root_cause_time_range` Similar to the relative time version but uses absolute time ranges. **Example:** "What caused the database connection pool to be exhausted between 10 AM and 11 AM today in the backend app?" ### 2. Log Deduplication #### `deduplicate_logs_relative_time` Removes duplicate log entries within a relative time range. **Example:** "Deduplicate the error logs from the payment service in the last 15 minutes" #### `deduplicate_logs_time_range` Deduplicates logs within an absolute time range. **Example:** "Remove duplicate exception logs from the web frontend between 2 PM and 3 PM yesterday" ### 3. Log Grouping & Aggregation #### `group_logs_by_field` Groups logs by a specific field and returns distribution of values with optional metric aggregation. **Use cases:** * "Which apps generate the most errors?" * "What is the p95 latency per host?" * "Which pods are producing the most logs?" **Examples:** "Which apps are generating the most errors? Show me the top 10" "What's the p95 latency for each host between 9 AM and 10 AM today?" "Show me the average memory usage by pod in the production namespace" ### 4. Log Visualization #### `get_log_histogram` Returns a time-series histogram of log volume over time. **Use cases:** * Visualizing log activity patterns * Identifying spikes or anomalies * Understanding when incidents occurred **Examples:** "Show me a histogram of error logs over the last hour" "Create a histogram of API 500 errors from 8 AM to 12 PM today with 5-minute buckets" "Visualize warning logs from the production namespace over the last hour with 30-second granularity" ### 5. Field Discovery #### `list_log_fields` Lists available structured fields that can be used in log queries. **Use cases:** * Discovering available fields for queries * Finding field names like "app", "host", "level", "namespace", "pod", "node" **Examples:** "What log fields are available?" "Show me all fields that contain 'kubernetes'" "List all fields that start with 'http'" "Find all fields ending with `_ms` (timing metrics)" "Is there an 'app' field in the logs?" "Show me fields that start with 'request' and end with 'time'" ### 6. Pipeline Management #### `list_pipelines` Lists available Mezmo v3 pipelines. **Example:** "Show me all the Mezmo pipelines" "What pipelines are configured?" #### `get_pipeline` Gets details for a specific Mezmo v3 pipeline, including root-level DAG/configuration fields such as `sources`, `transforms`, and `sinks` when returned by the Pipeline API. **Parameters:** * `pipeline_id` (required): UUID of the pipeline **Example:** "Show me the details for pipeline 123e4567-e89b-12d3-a456-426614174000" "What's the configuration of the production-logs pipeline?" #### `create_pipeline` Creates an empty draft Mezmo pipeline. Sources, transforms, and destinations are added separately, and this tool does not publish the pipeline. **Parameters:** * `title` (required): Pipeline title * `deploy_type` (optional): Pipeline deploy type. Defaults to `saas` * `deployment_groups` (optional): Deployment groups for the pipeline **Example:** "Create a new pipeline called 'checkout-service-logs'" #### Pipeline node discovery Use these tools to discover and inspect the source, transform/processor, and destination/sink schemas available from the Pipeline API before building a pipeline: * `get_node_catalog`: Returns the raw `GET /pipeline/node` catalog. * `search_pipeline_node_schemas`: Searches source, transform/processor, and destination/sink node schemas by `kind` and/or a free-text `query`, without surfacing nested JSON Schema fragments as selectable nodes. * `get_pipeline_node_schema`: Inspects a single node schema (`node_type`, optional `kind`) before constructing a Pipeline API body. **Example:** "What source node types are available for HTTP ingestion?" "Show me the schema for the datadog-agent destination" #### Building and editing pipelines * `add_pipeline_source`: Adds a source node to an existing pipeline from catalog/schema-backed payload fields (`pipeline_id`, `node_type`, optional `title`, `description`, `user_config`, `extra_fields`). * `add_pipeline_transform`: Adds a transform/processor node, wired to upstream nodes via raw UUIDs or aliases (`pipeline_id`, `node_type`, `inputs`, optional `title`, `description`, `user_config`, `extra_fields`). * `add_pipeline_destination`: Adds a destination node, mapped to the Pipeline API sink endpoint (same parameters as `add_pipeline_transform`). * `create_pipeline_access_key`: Creates a source gateway access key (`gateway_route_id`, `title`, optional `key_type`, `extra_fields`). Generated secret values are redacted from the MCP response. * `update_pipeline_component`: Updates an existing source, transform/processor, or destination/sink with a raw `PUT` payload (`pipeline_id`, `component_id`, `kind`, `payload`). Inspect `get_pipeline` and the relevant node schema first. * `delete_pipeline_component`: Deletes an existing source, transform/processor, or destination/sink (`pipeline_id`, `component_id`, `kind`). Use explicit confirmation before calling this, since it's destructive. **Example:** "Add an HTTP source to pipeline 123e4567-e89b-12d3-a456-426614174000" "Delete the old-syslog-source component from that pipeline" #### `create_pipeline_from_manifest` Creates a pipeline end to end from a manifest: creates the pipeline, adds sources, optionally creates generated access keys, adds transforms/processors, adds destinations/sinks, and publishes only when `publish` is explicitly `true`. **Parameters:** * `title` (required): Pipeline title * `deploy_type`, `deployment_groups`, `config` (optional): Same as `create_pipeline` * `sources`, `transforms`, `destinations` (optional): Arrays of node specs, each with an `alias`, `type`, optional `title`, `description`, `user_config`, `inputs`, and `access_key` * `publish` (optional): Publish the pipeline once created. Defaults to `false` Node `inputs` can reference other nodes by alias (for example `alias.Default` or `alias.outputs[0]`) or by raw UUID, letting the agent wire the whole DAG in one call. **Example:** "Create a pipeline named 'payments-pipeline' with an HTTP source feeding a JSON parser transform and a Datadog destination, and publish it" #### `tap_pipeline_component` Taps live events from a deployed pipeline component via `/pipeline/{pipeline_id}/tap/{component_id}`. **Parameters:** * `pipeline_id` (required): UUID of the pipeline * `component_id` (required): UUID of the source or processor component to tap * `timeout` (optional): Seconds to wait for events. Defaults to `2` * `limit` (optional): Maximum events to return, capped at `100` to keep agent context bounded. Defaults to `100` Use `get_pipeline` first to identify a source or processor component ID, and treat returned event data as potentially sensitive. **Example:** "Tap the http-source component on pipeline 123e4567-e89b-12d3-a456-426614174000 and show me a sample of live events" #### `pause_pipeline` / `unpause_pipeline` Sets a pipeline's `processing_status` to `disabled` or `enabled` respectively. Neither tool publishes or edits the pipeline DAG. **Parameters:** * `pipeline_id` (required): UUID of the pipeline **Example:** "Pause pipeline 123e4567-e89b-12d3-a456-426614174000" "Unpause the checkout-service-logs pipeline" #### `publish_pipeline` Publishes a pipeline's current draft configuration. **Parameters:** * `pipeline_id` (required): UUID of the pipeline `publish_pipeline` is never called automatically by `create_pipeline_from_manifest` unless `publish` is explicitly set to `true`, so draft pipelines stay unpublished until you confirm. ### 7. Time Utilities #### `get_current_time` Returns the current server time in various formats. **Example:** "What time is it right now?" "Get the current timestamp" **Use cases:** * Determining time ranges for log analysis * Converting between time formats #### `relative_time_to_time_range` Converts relative time expressions to absolute time ranges in milliseconds. **Examples:** "Convert 'last 5 minutes' to an absolute time range" "What's the timestamp range for '2 hours ago'?" ### 8. Trace Analysis These tools analyze OpenTelemetry trace data across your entire account, so you don't need to specify a source or component. Trace, span, and failure data lags behind real time by roughly a couple of minutes. The degraded-edge and latency-summary tools (`find_degraded_service_edges` and `list_service_latency_summaries`) draw on latency summaries that can lag further, sometimes by hours, and fill in windows retroactively as backlogs drain, so very recent activity may not appear immediately. These trace tools appear in the tool list, and are callable, only when Mezmo has analyzed trace data from your account within roughly the last 24 hours (the default look-back window). If your account has no recent analyzed trace data, the tools are hidden and do not appear. Mezmo evaluates availability automatically, and there is no feature flag to turn on. If an agent calls one of these tools while it is unavailable, the server responds with a `tool not found` error, the same as for an unknown tool. * `get_service_graph`: Gets the service graph showing service-to-service calls by operation. Takes no arguments. The graph is built from completed traces, so a missing edge does not prove that the call never happens. * `list_failed_traces`: Lists failed traces within a time range, optionally filtered by service. Supports an optional `limit` (1 to 500, default `100`). Only traces with at least one failed span are indexed, so an empty result means no failures, not no traffic. * `get_trace_spans`: Gets the spans for a given `trace_id`. Obtain the `trace_id` from `list_failed_traces` or `get_service_failures` (both described in this section). Supports optional `limit` (1 to 500, default `250`), `failed_only` (default `false`, returns only failing spans), and `include_attributes` (default `false`, includes span attributes). * `find_degraded_service_edges`: Detects service-call edges with degraded latency by comparing a current period to a baseline period, and surfaces newly appearing edges. Returns edges where the selected percentile metric increased beyond the threshold you set. Use it to spot service-to-service calls that are changing behavior. **Parameters:** * `current_from_time` (required): Start of the current window, as an RFC3339 timestamp (for example, `2023-01-01T00:00:00Z`). * `current_to_time` (required): End of the current window, as an RFC3339 timestamp. * `baseline_from_time` (required): Start of the baseline window, as an RFC3339 timestamp. * `baseline_to_time` (required): End of the baseline window, as an RFC3339 timestamp. * `metric` (optional): Percentile to compare. One of `p50`, `p90`, `p95`, or `p99`. Defaults to `p99`. * `threshold` (optional): Minimum ratio of current to baseline latency that counts as degraded, so `2.0` means current latency is at least twice the baseline. Defaults to `2.0`. * `limit` (optional): Maximum number of edges to return, from 1 to 100. Defaults to `20`. * `min_sample_count` (optional): Edges with fewer samples are treated as low confidence and suppressed. Defaults to `30`. * `include_low_confidence` (optional): Include low-confidence rows. Defaults to `false`. * `list_service_latency_summaries`: Fetches bucketed latency summaries for operations between a service and its target. Each result is a timestamped bucket with latency metrics like `p50`, `p95`, and `p99`, plus a mean. * `get_service_failure_rates`: Gets service failure rates as a histogram over the window you provide, optionally grouped by service, edge, or operation. * `get_service_failures`: Fetches individual service failure records, optionally filtered by service, edge, or operation. **Examples:** "Show me the service graph for my account" "Which services have degraded latency compared to last week?" "List failed traces for the checkout-service in the last hour" ### 9. Correlated Timeline #### `get_correlated_timeline_relative_time` Builds a correlated timeline of logs across multiple sources within a relative time range. Groups logs by a source field (like app, host, or any custom field) to help with root cause investigation. **Use cases:** * Investigating incidents across multiple services or hosts * Understanding the sequence of events leading to a failure * Correlating errors across apps in a distributed system **Example:** "Show me a correlated timeline of error logs from all apps in the last 30 minutes" #### `get_correlated_timeline_time_range` Same as above, but uses absolute time ranges. **Example:** "Build a timeline of all warnings and errors between 10 AM and 11 AM today, grouped by host" #### Key options * **Grouping field**: Logs are grouped by `_app` by default. You can also group by `_host`, `level`, or any custom field * **Deduplication mode**: Use `template` mode (default) to show one example per unique log pattern, or `none` for raw chronological order * **Limits**: Control how many logs per source and total timeline events to return ### 10. AI Investigations An AI investigation is an AI-generated root-cause and analysis record for your account. Mezmo triggers investigations internally, for example from a Pipeline Node Alert. These two tools read and list existing investigations; they do not create them. Your account needs the appropriate investigation permission to use these tools; this is managed through your account's [roles and permissions](/docs/rbac). For the full feature guide, see [AI Investigations](/docs/ai-investigations). #### `list_ai_investigations` Lists AI-generated investigations for your account. **Parameters:** * `state` (optional): filter by investigation state; one of `triggered`, `investigating`, or `completed` * `last_id` (optional): pagination cursor; the id of the last investigation from the previous page * `updated_since` (optional): an RFC3339 date-time; returns only investigations updated since then * `limit` (optional): maximum number of results, from 1 to 1000. Defaults to `1000` To fetch the next page, pass the last returned investigation's id as `last_id`. **Example:** "List my completed AI investigations from the last day" "Show me AI investigations that are still investigating" #### `get_ai_investigation` Gets details for a specific AI investigation by id. A returned investigation includes its state and the AI's root-cause findings, such as a confidence score and suggested resolution. **Parameters:** * `investigation_id` (required): the id of the investigation to retrieve **Example:** "Show me the details for investigation 66f1a2b3c4d5e6f7a8b9c0d1" *** ## Query Syntax Many MCP tools accept a `query` parameter that filters logs. You can combine text search with fielded queries to find specific logs. ### Text Search For basic searches, provide keywords. The query matches any logs containing those terms (case-insensitive, with automatic prefix-matching): ```none theme={null} connection timeout ``` Prefix-matching is automatic. For example, `timeout` matches `timeout`, `timeouts`, and `timeout_error` without needing a wildcard. ### Fielded Search Use `field:value` syntax to search specific structured fields: ```none theme={null} app:payment-service level:error host:prod-api-01 ``` #### Common Fields These fields are available in most log data: | Field | Description | | ----------- | -------------------------------------- | | `app` | Application or service name | | `host` | Hostname or server identifier | | `level` | Log level (error, warn, info, debug) | | `env` | Environment (production, staging, dev) | | `source` | Log source identifier | | `tag` | Tags applied to the log | | `namespace` | Kubernetes namespace | | `pod` | Kubernetes pod name | | `node` | Kubernetes node name | #### Nested Fields For nested or JSON fields, use dotted paths: ```none theme={null} user.id:12345 request.body.status:failed kubernetes.labels.app:frontend ``` ### Numeric Comparisons Query numeric fields using comparison operators: ```none theme={null} status:>=400 latency_ms:<100 response_time:>500 ``` ### Field Existence Use `field:*` to match logs where a specific field is present: ```none theme={null} error_code:* user.id:* ``` This is useful for finding logs that have a particular field, regardless of its value. The `*` character is only valid for field existence checks (`field:*`). Do not use wildcards in query terms or field values (for example, `pod:vector-*` or `host:prod-*`). Prefix-matching is automatic, so `pod:vector` already matches `vector-gen14-abc`, `vector-prod-01`, and similar values. ### Combining Queries Combine multiple conditions to narrow your search: ```none theme={null} level:error app:checkout-service status:>=500 ``` *** ## Query Protection for Large Log Volumes Mezmo MCP protects against queries that would process too much data. This keeps performance fast and ensures queries finish quickly. ### How It Works When you run a tool that processes logs, Mezmo MCP checks how many entries match your query: * **Confirmation prompt**: If your query matches a large number of logs, the server asks you to confirm before proceeding. You can narrow your query or confirm to continue. * **Query rejection**: If your query matches an extremely large number of logs, the request is rejected. Try narrowing your time range or adding more specific filters. ### Affected Tools These tools have query protection: * `analyze_logs_for_root_cause_relative_time` / `analyze_logs_for_root_cause_time_range` * `deduplicate_logs_relative_time` / `deduplicate_logs_time_range` * `get_correlated_timeline_relative_time` / `get_correlated_timeline_time_range` * `get_log_histogram` * `group_logs_by_field` ### Tips for Working with Large Datasets If you see a confirmation prompt or rejection: 1. **Narrow your time range.** Try "last 1 hour" instead of "last 24 hours". 2. **Add filters.** Use fielded search to target specific logs (for example, `app`, `host`, `level`, `namespace`, or `pod`). See [Query Syntax](#query-syntax) for the full list. 3. **Confirm when appropriate.** If you've reviewed the log count and want to proceed, confirm the prompt. ### Retention Window Limits Queries must fall within your account's retention window. If you query logs outside this window, you'll receive an error showing the valid time range. For example, querying logs from 6 months ago on a 7-day retention account will fail. Use relative time ranges like `last 15 minutes` or `last 1 hour` to ensure your queries stay within retention. *** ## Best Practices * Start broad for root-cause analysis * If the query is too narrow, our RCA can't do its thing. Cast a wide net, and if needed, specify app/service/level in subsequent queries. * Prefer `analyze_logs_for_root_cause` or `deduplicate_log` tools for insights * These tools deduplicate and groups similar logs for better summaries, allowing the results to fit into finite LLM context windows. * Use relative time ranges * Prefer values like `last_15_minutes`, `last_hour` when supported. * Keep prompts simple; add filters gradually * Add filters step by step. See [Query Syntax](#query-syntax) for available fields. * Use `export_logs` for raw data only * For dashboards or offline analysis, use `export_logs`; otherwise prefer RCA. *** ## Installation ### Requirements * A Mezmo **Service Key** (generate one in your Mezmo dashboard under Settings > API Keys; see [Mezmo docs](/docs/ingestion-key#ingestion-and-service-keys) for details) * Node.js ≥ 18 (only needed for clients that use the `mcp-remote` bridge) * One of the supported MCP clients below For every client we follow a simple rule: 1. **Supports remote URL? →** configure it with a `url` that points to `https://mcp.mezmo.com/mcp` and include the `Authorization` header. 2. **StdIO-only client? →** use the [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) bridge: ```bash theme={null} AUTH_HEADER="Bearer " npx mcp-remote https://mcp.mezmo.com/mcp \ --header "Authorization:${AUTH_HEADER}" ``` *** ## Client Configurations #### AURA [AURA](https://github.com/mezmo/aura) is Mezmo's open-source agentic SRE harness. It's MCP-native, so you can point an AURA agent at the Mezmo MCP server and its tools become available at runtime for root-cause analysis, log queries, and pipeline management. Add a `[mcp.servers.mezmo]` block to your AURA config (`config.toml` or a file in your `configs/` directory): ```toml theme={null} [mcp.servers.mezmo] transport = "http_streamable" url = "https://mcp.mezmo.com/mcp" description = "Mezmo root-cause analysis, log analysis, export, and pipeline management" [mcp.servers.mezmo.headers] Authorization = "Bearer " ``` AURA renders header values as Liquid templates, so you can keep your Service Key out of the config file and read it from the environment instead: `Authorization = "Bearer {{ env.MEZMO_SERVICE_KEY | default: '' }}"`. See the [AURA quickstart](/aura/quickstart) and the bundled [`incident-response-mezmo.toml`](https://github.com/mezmo/aura/blob/main/examples/complete/incident-response-mezmo.toml) example for a full agent that pairs Mezmo with other MCP servers. #### Cursor Cursor **natively supports remote MCP servers**, so you only need a remote configuration. [Install Mezmo MCP Server in Cursor](https://cursor.com/en-US/install-mcp?name=mezmo\&config=eyJ1cmwiOiJodHRwczovL21jcC5tZXptby5jb20vbWNwIiwiaGVhZGVycyI6eyJBdXRob3JpemF0aW9uIjoiQmVhcmVyIDxTRVJWSUNFIEtFWT4ifX0=) Clicking the **Install MCP Server** badge opens Cursor and automatically adds the `mezmo` entry to your `~/.cursor/mcp.json` with a placeholder for the Service Key. After it’s created, edit the file and replace `` with your actual Mezmo service key. Restart Cursor for changes to take effect. The final configuration should look like the example below. ```json theme={null} { "mcpServers": { "mezmo": { "url": "https://mcp.mezmo.com/mcp", "headers": { "Authorization": "Bearer " } } } } ``` #### Windsurf Windsurf also supports remote servers via the `serverUrl` field. ```json theme={null} { "mcpServers": { "mezmo": { "serverUrl": "https://mcp.mezmo.com/mcp", "headers": { "Authorization": "Bearer " } } } } ``` #### Trae ```json theme={null} { "mcpServers": { "mezmo": { "url": "https://mcp.mezmo.com/mcp", "headers": { "Authorization": "Bearer " } } } } ``` #### VS Code’s Copilot Chat or Visual Studio 2022 ```json theme={null} "mcp": { "servers": { "mezmo": { "type": "http", "url": "https://mcp.mezmo.com/mcp", "headers": { "Authorization": "Bearer " } } } } ``` #### Zed ```json theme={null} { "context_servers": { "mezmo": { "url": "https://mcp.mezmo.com/mcp", "headers": { "Authorization": "Bearer " } } } } ``` #### Gemini CLI ```json theme={null} { "mcpServers": { "mezmo": { "command": "npx", "args": [ "mcp-remote", "https://mcp.mezmo.com/mcp", "--header", "Authorization:${AUTH_HEADER}" ], "env": { "AUTH_HEADER": "Bearer " } } } } ``` #### Claude Code ```bash theme={null} claude mcp add --transport http mezmo https://mcp.mezmo.com/mcp \ --header "Authorization: Bearer " ``` #### Claude Desktop ```json theme={null} { "mcpServers": { "Mezmo": { "command": "npx", "args": [ "mcp-remote", "https://mcp.mezmo.com/mcp", "--header", "Authorization:${AUTH_HEADER}" ], "env": { "AUTH_HEADER": "Bearer " } } } } ``` #### BoltAI ```json theme={null} { "mcpServers": { "mezmo": { "command": "npx", "args": [ "mcp-remote", "https://mcp.mezmo.com/mcp", "--header", "Authorization:${AUTH_HEADER}" ], "env": { "AUTH_HEADER": "Bearer " } } } } ``` #### Windows (CMD) ```json theme={null} { "mcpServers": { "mezmo": { "command": "cmd", "args": [ "/c", "npx", "mcp-remote", "https://mcp.mezmo.com/mcp", "--header", "Authorization:${AUTH_HEADER}" ], "env": { "AUTH_HEADER": "Bearer " } } } } ``` #### Augment Code Add a new MCP and enter: ```bash theme={null} AUTH_HEADER="Bearer " npx mcp-remote https://mcp.mezmo.com/mcp \ --header "Authorization:${AUTH_HEADER}" ``` #### Roo Code Roo Code supports remote URLs: ```json theme={null} { "mcpServers": { "mezmo": { "type": "streamable-http", "url": "https://mcp.mezmo.com/mcp", "headers": { "Authorization": "Bearer " } } } } ``` #### ZenCoder ```json theme={null} { "command": "npx", "args": [ "mcp-remote", "https://mcp.mezmo.com/mcp", "--header", "Authorization:${AUTH_HEADER}" ], "env": { "AUTH_HEADER": "Bearer " } } ``` #### Amazon Q Developer CLI ```json theme={null} { "mcpServers": { "mezmo": { "command": "npx", "args": [ "mcp-remote", "https://mcp.mezmo.com/mcp", "--header", "Authorization:${AUTH_HEADER}" ], "env": { "AUTH_HEADER": "Bearer " } } } } ``` #### Qodo Gen ```json theme={null} { "mcpServers": { "mezmo": { "url": "https://mcp.mezmo.com/mcp", "headers": { "Authorization": "Bearer " } } } } ``` #### JetBrains AI Assistant ```json theme={null} { "mcpServers": { "mezmo": { "command": "npx", "args": [ "mcp-remote", "https://mcp.mezmo.com/mcp", "--header", "Authorization:${AUTH_HEADER}" ], "env": { "AUTH_HEADER": "Bearer " } } } } ``` #### Warp ```json theme={null} { "Mezmo": { "command": "npx", "args": [ "mcp-remote", "https://mcp.mezmo.com/mcp", "--header", "Authorization:${AUTH_HEADER}" ], "env": { "AUTH_HEADER": "Bearer " }, "working_directory": null, "start_on_launch": true } } ``` #### OpenCode ```json theme={null} "mcp": { "mezmo": { "type": "remote", "url": "https://mcp.mezmo.com/mcp", "headers": { "Authorization": "Bearer " }, "enabled": true } } ``` #### Codex CLI Add to `~/.codex/config.toml`: ```none theme={null} [mcp_servers.mezmo] command = "npx" args = [ "mcp-remote", "https://mcp.mezmo.com/mcp", "--header", "Authorization:${AUTH_HEADER}", ] env = { AUTH_HEADER = "Bearer " } ``` Codex [defaults to being sandboxed](https://github.com/openai/codex/blob/main/docs/sandbox.md), so you'll also need to give it network access. If you want to run in a sandbox mode of `workspace-write` for example, add this: ```none theme={null} [sandbox_workspace_write] network_access = true ``` You can either start with a flag like `codex --sandbox workspace-write` or set a default mode with: ```none theme={null} sandbox_mode = "workspace-write" ``` *** ## Troubleshooting ### 1. `npx` argument-escaping bug Some clients pass command-line arguments to `npx` without quoting spaces. This can split the `Authorization` header (e.g. `Bearer` and the token become separate arguments) and cause authentication failures. **Work-around:** store the header in an environment variable and pass it without spaces: ```json theme={null} { "command": "npx", "args": [ "mcp-remote", "https://mcp.mezmo.com/mcp", "--header", "Authorization:${AUTH_HEADER}" ], "env": { "AUTH_HEADER": "Bearer " } } ``` ### 2. Lost connection to the Mezmo MCP server If your client shows an error such as “server disconnected” or stops responding to MCP commands: 1. Disable or remove the **Mezmo** MCP entry in your client settings. 2. Re-enable (or re-add) the same entry, or simply restart the client. This forces the client to establish a fresh connection to the Mezmo MCP backend. ### 3. 401/403 authentication errors * Verify the `Authorization` header is present and formatted as `Bearer `. * If using `npx mcp-remote`, prefer the environment variable approach to avoid splitting the header. * Regenerate your Service Key in Mezmo and try again if issues persist. *** ## Next Steps Once your client is configured you can immediately run natural-language commands such as * `analyze my logs from the last 30 minutes and determine root cause for any issues that you find` * `list all my pipelines` * `show me details for pipeline ` * `deduplicate error logs from the last 30 minutes for app "my-app-frontend"` * `create a pipeline that ingests from HTTP, drops unnecessary data, and stores the rest in Log Analysis` * `tap the source component on and show me a sample of live events` Enjoy streamlined observability with Mezmo + AI! 🎉 # Welcome to Mezmo Source: https://docs.mezmo.com/docs/mezmo-quick-start-guide Get started with Mezmo by creating an organization, adding log sources, parsing and searching logs, building views, alerts, boards, and archiving. ## Welcome! Welcome to Mezmo! Mezmo’s log management platform lets you collect, monitor, parse, live tail, graph, and analyze logs with clear visualizations and smart alerting all within minutes. In this quick start guide, we will walk you through the steps to get you managing and analyzing logs in minutes! ## Set up Your Account and Create an Organization 1. Click **Sign Up** on the [Mezmo website](https://mezmo.com/sign-up/). You will be automatically enrolled into a 30 day free trial. 2. Your organization is an independent workspace where you can access and configure your logs, add members, change billing plan, and manage other aspects of your account. Once you create an organization, you will be given an auto-generated ingestion key, which you can use to send in logs. You can find more detail about setting up an organization in [the Mezmo Organization Management guide](/docs/organization-management-overview) and in the topic Mezmo Organization Basics. ## Add A Log Source 1. Next, choose the ingestion logging source. Mezmo offers a variety of [Mezmo Log Ingestion Methods](/docs/ingestion) including the [Mezmo Logging Agent](/docs/introducing-the-agent) and the [auto\$](/docs/api-referencesk9h). You can install the Mezmo Agent on Kubernetes, Openshift, Linux, Windows, and MacOS, and you can take a look at our source code on [GitHub](https://github.com/logdna/logdna-agent-v2). 2. Each organization can have multiple ingestion sources. To add an ingestion source, [log in to the Mezmo Web App](https://app.mezmo.com). 3. In the top navigation bar of the Web App, click the **Add Log Sources** icon. 4. Select the log source you want to use, and follow the setup instructions. Note that your ingestion key is automatically added to the code examples. You can find more information about ingestion sources in the topic [Mezmo Log Ingestion Integrations](/docs/ingestion-integrations) ## Parse Logs As your logs are ingested into Mezmo, they are automatically [parsed into components](/docs/log-parsing) that are instantly viewable and actionable. You can also create [custom parsing templates](/docs/parse-logs-with-custom-templates). ## Search Your Logs Mezmo Log Search provides [advanced search operators and syntax](/docs/search-and-filter) to find critical data in your logs, as well as the ability to [view log data by timeline.](/docs/view-log-data-by-timeline) ## Create Views and Alerts Once you've searched your logs, you can [save it as a view.](/docs/create-and-edit-views). A view is like a bookmark for your log lines, so you can easily save them and review them as many times as you want. You can also [add alerts to views](/docs/add-alerts-to-views) to notify you and your team when certain conditions are met. Mezmo has for many popular platforms, including [DataDog](/docs/datadog-alert-integration), [Slack](/docs/slack-alert-integration), and [PagerDuty](/docs/pagerduty-alert-integration). ## Set Up Exclusion Rules and Spike Protection Mezmo provides several cost control features to protect against unexpected spikes in log data volume, including: [Exclusion Rules](/docs/excluding-log-lines) that help you filter out log data that you don’t need to store, which can help you manage storage costs and focus your analysis on data that contains meaningful information. [Usage Quotas](/docs/manage-usage-quotas) allow you to set hard and soft limits on ingestion, and set additional Exclusion Rules to ensure that you aren’t going over budget while still getting the log data you need. [Index Rate Alerts ](/docs/index-rate-alerts-overview)use historical trends to notify you when there are unexpected volume-based spikes in log data, and provides insights into where they are happening. ## Set Up Archiving Mezmo stores your log data for a certain period of time, based on which plan you are on (the free trial plan stores logs for 14 days). Enabling [Archiving](/docs/archiving) ensures that once the retention period has concluded, your log data will be [exported to external storage](/docs/export-logs-to-external-storage) (for example, S3) for continued access. Additionally, enabling Archiving allows you to easily bring log data back into the Mezmo UI via our [Data Restoration](/docs/data-restoration) feature. ## Create Boards and Graphs [Boards](/docs/create-a-graph) are similar to Views, but instead of containing log line data, contain graphs that present metrics about your log data. Graphs are especially useful for application monitoring and data visualization, and provide their own controls for filtering and refining log data. ## Change Plan Once your 30 day trial is over, you will need to select a paid plan to continue using Mezmo. If you would like to retain your logs for 7, 14, or 30+ days, you can go to `Settings` > `Billing` > `Overview` to select one of our paid plans. Compare all our pay-per-usage [pricing](https://mezmo.com/pricing/) model here. If you'd like to discuss options or pricing, contact our Sales team at [outreach@mezmo.com](mailto:outreach@mezmo.com). Now that you've followed these steps to set up Mezmo and start working with your logs, you're ready to start mining your logs for information and insights to accelerate your operational success! # Mezmo Service Endpoints Source: https://docs.mezmo.com/docs/mezmo-service-endpoints Reference the inbound and outbound IP addresses for Mezmo services to configure IP-based firewall rules and allowlists. This page lists IP addresses for Mezmo services. Use these addresses to configure IP-based firewall rules. ## Inbound IP addresses Mezmo ingestion services use the following IP addresses. Traffic you send to Mezmo arrives at these endpoints. Mezmo services use domain name system (DNS) routing, so you do not need to update your agent configuration. You only need these IP addresses if you have IP-based filtering in your firewalls. **These IP addresses may change over time as our infrastructure evolves. Mezmo cannot guarantee these addresses will remain unchanged indefinitely. The ideal configurations will rely on DNS to reach these services and not the IP addresses directly. Mezmo will keep this document updated on a best-effort basis.** **Latest Update**: April 6, 2026 at 3:30PM UTC **api.mezmo.com:** 174.129.121.149 54.198.14.31 54.227.203.3 **logs.mezmo.com:** 174.129.121.149 54.198.14.31 54.227.203.3 **assets.logdna.com:** 54.224.242.53 3.230.116.140 52.4.27.175 **syslog-a.mezmo.com and syslog-u.mezmo.com:** These endpoints are for legacy direct Syslog ingestion. Mezmo's [Syslog](/docs/ingestion-integrations/syslog), [Rsyslog](/docs/ingestion-integrations/rsyslog), [Syslog-ng](/docs/ingestion-integrations/syslog-ng), and [NXLog](/docs/ingestion-integrations/nxlog-for-windows) ingestion integrations now recommend forwarding through an OpenTelemetry Collector to the Mezmo OTEL Source instead, which uses the **logs.mezmo.com** addresses listed above. 107.21.191.159 18.210.196.184 18.213.75.161 3.215.157.103 3.217.96.45 3.88.79.246 44.194.176.49 44.198.142.174 52.2.231.144 52.72.2.63 52.86.146.53 54.159.155.79 54.159.171.103 54.162.98.246 54.167.185.230 ## Outbound IP addresses When Mezmo connects to external services like third-party alerting integrations or external APIs, traffic comes from these IP addresses: 34.228.164.79 34.236.93.215 3.229.58.59 If your security policies require IP allowlisting for inbound connections from Mezmo, add these addresses to your allowlist. # NGINX Ingress Controller Template Source: https://docs.mezmo.com/docs/nginx-ingress-controller-template Use Mezmo's pre-built Views, Boards, and Screens plus a recommended log format to gain observability into NGINX Ingress Controller logs. The Mezmo NGINX Ingress Controller Template provides you better observability into your infrastructure with NGINX Ingress Controller ([nginxinc/kubernetes-ingress](https://github.com/nginxinc/kubernetes-ingress)) logs. The Template includes pre-configured Views, Boards, Screens, and a [logging format](#nginx-ingress-controller-recommended-log-format-configuration) to get the most out of your logs. After configuring the template, you can set up alerts on excessive HTTP 500's, graph response codes broken down by upstream service name, and more. ### Views Views are saved shortcuts to a specific set of filters and search queries. You can also add Alerts to views to notify you when specific conditions are met. Check out the topic [Add Alerts to Views](/docs/add-alerts-to-views) for more information. * HTTP 2XX * HTTP 5XX * HTTP 404 Errors * HTTP Forbidden/Unauthorized (401, 403) * HTTP Server Errors - includes 500's and NGINX error logs, if those error logs are using the default error log format. ### Boards Boards are collections of graphs. Using boards, you can track trends with response codes and understand how they fluctuate over time at a glance. Drill down using subplots to see which host or path is generating the most errors. Check out the topic [Visualize Log Data with Graphs](/docs/visualize-log-data-with-graphs) for more information. * HTTP Response Codes and Errors * Breakdown by App, Host, Request (Path), Client IP, and Upstream Kubernetes Service * 95th Percentile Response Times for Upstream and Total Time * Traffic Volume with Total Bytes and Requests ### Screens Screens are collections of customized dashboards that can display data in various forms. See the topic [Use Screens and Widgets to Monitor Log Data](/docs/use-screens-and-widgets-to-monitor-log-data) for more information. * Web Analytics, such as traffic trends, most popular pages, and referrers * Server Health, such as number of 500s by grouped by upstream service * Web Server Security, such as top 401 and 403 errors by IP Address ## NGINX Ingress Controller Log Format Configuration To take full advantage of the NGINX Ingress Controller Template with [nginxinc/kubernetes-ingress](https://github.com/nginxinc/kubernetes-ingress), you should add our recommended custom log format. The custom log format exposes additional NGINX variables such as `upstream service name` and `response times` to further aid in debugging your infrastructure. Without this custom log format, parts of the template will not be populated with data. [The NGINX documentation](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/) contains more details on configuring NGINX via a ConfigMap resource. Log formats that deviate from this format or the default Apache Common or Combined Log Format will lead to incorrect data being populated in this template. ### Recommended Log Format Configuration ```yaml theme={null} log-format: 'verb="$request_method" request="$uri" response=$status clientip="$remote_addr" scheme="$scheme" bytes=$bytes_sent agent="$http_user_agent" referrer="$http_referer" request_time=$request_time upstream_response=$upstream_status upstream_response_time=$upstream_response_time upstream_service="$service" resource_name="$resource_name" resource_namespace="$resource_namespace" resource_type="$resource_type" http_host="$http_host" request_id="$request_id" time_date="$time_iso8601"' log-format-escaping: 'json' ``` ### Example ConfigMap You may need to modify this depending on your NGINX Ingress Controller setup. ```yaml theme={null} kind: ConfigMap apiVersion: v1 metadata: name: nginx-config namespace: default data: log-format: 'verb="$request_method" request="$uri" response=$status clientip="$remote_addr" scheme="$scheme" bytes=$bytes_sent agent="$http_user_agent" referrer="$http_referer" request_time=$request_time upstream_response=$upstream_status upstream_response_time=$upstream_response_time upstream_service="$service" resource_name="$resource_name" resource_namespace="$resource_namespace" resource_type="$resource_type" http_host="$http_host" request_id="$request_id" time_date="$time_iso8601"' log-format-escaping: 'json' ``` ## Configuration for Community Version If you're using the ingress version from `kubernetes/ingress-nginx,`the default template will still be compatible, but the extra Kubernetes metadata will not be automatically parsed. This means that template elements that require data like upstream service/response time will not be available. To expose additional Kubernetes metadata to Mezmo, you can configure a custom [ConfigMap](https://kubernetes.github.io/ingress-nginx/examples/customization/custom-configuration/) and [log format](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/log-format/) that is similar to the default log format shown in the previous section. # Okta SAML Setup Source: https://docs.mezmo.com/docs/okta-saml-setup Configure Okta as a SAML identity provider for Mezmo single sign-on, including JIT provisioning and SSO enforcement behavior. This feature is only available for customers on an Enterprise plan, and is restricted to **Owner** and **Admin** user roles. Please contact your Customer Support Manager or [support@mezmo.com](mailto:support@mezmo.com) for more information. ## Step 1: Get Your Mezmo Single Sign On URL 1. In your Mezmo app go to **Settings > Organization > Access Management**. 2. Go to **SAML Configuration** and copy the URL under **Single Sign On URL**. 3. Keep this URL available since it will be used in **Step 2: Configure Okta**. ## Step 2: Configure Okta 1. In Okta go to **Applications > Applications**. 2. Click **Create App Integration**. 3. In the window, choose **SAML 2.0**. 4. Fill out General Settings. 5. In the next window, enter the Mezmo single sign on URL from **Step 1: Get Your Mezmo Single Sign On URL** 6. Make sure **Recipient URL and Destination URL** are checked. 7. Set the **Audience URI (SP Entity ID)** to your account ID. At the end of your URL. For example, `https://app.mezmo.com/auth/saml-consume/`. You can leave the rest of the options as is. 8. Fill out step 3 Feedback. ## Step 3: SAML Configuration Image 1. In Okta, on the settings page, click the button; **View SAML setup instructions**. 2. If not already done, in Mezmo, select **configure manually**. 3. Copy the **Identity Provider Single Sign-On URL** in Okta to **Identity provider sign-in URL** in Mezmo. 4. Download the **X.509 Certificate** from Okta and upload it to Mezmo. 5. Save your config in Mezmo. 6. You can also copy and save the XML data on Okta under **Optional** and upload it to Mezmo. ## Frequently Asked Questions **Does the Mezmo integration with Okta have Just In Time (JIT) or other provisioning features? For example, assigning users to the Okta app, or provisioning accounts automatically?** Yes, it has JIT provisioning and accounts are created automatically. **Does the Mezmo Integration also automatically expand our licensing count?** It does not add licensing, it only creates the account. **If I remove the application assignment from a user, does it deprovision/deactivate their Mezmo account?** It does not deprovision/deactivate their Mezmo Account. They would need to login with their email address, but they would not have a password. They would need to click **Forgot Password** to create a new one. **Is Okta SSO enforced, or can people also log in via their Google login?** When Okta SSO is enforced, it will force normal users to use Okta to login. The one exception is the owner of the account. The account owner may need an alternative access method should there be problems with the SSO setup. # OneLogin SAML Setup Source: https://docs.mezmo.com/docs/onelogin-saml-setup Configure OneLogin as a SAML identity provider for Mezmo single sign-on using a custom connector and downloaded SAML metadata. This feature is only available for customers on an Enterprise plan, and is restricted to **Owner** and **Admin** user roles. Please contact your Customer Support Manager or [support@mezmo.com](mailto:support@mezmo.com) for more information. ## Step 1: Get Your Mezmo Single Sign On URL 1. In your Mezmo app go to **Settings > Organization > Access Management**. 2. Go to **SAML Configuration** and copy the URL under **URL for the Single Sign On Service to Consume**. 3. Keep this URL available since it will be used in **Step 2: Configure OneLogin**. ## Step 2: Configure OneLogin 1. Log in to your instance of OneLogin. 2. Go to **Applications > Applications**. Then click **Add App**. 3. In the search box, search for `SAML Custom Connector`. Select `SAML Customer Connector (Advanced)`. 4. Give the connector a name and then Save. 5. Select configuration from the menu in OneLogin. 6. Enter the Mezmo Single Sign On Service to Consume URL from **Step 1: Get Your Mezmo Single Sign On URL**, into `ACS (Consumer) URL Validator` and `ACS (Consumer) URL`. 7. Select **SSO** from the left menu. 8. Make sure the **SAML Signature Algorithm** is SHA-256 in the dropdown. ## Step 3: SAML Configuration Image Image Image 1. In OneLogin, go to the **SSO** tab. 2. In the **More Actions** dropdown, select **SAML Metadata**. An XML file will download. 3. In your Mezmo app, go to **Settings > Organization > Security**. 4. Toggle **SAML Sign-in** to on. 5. In **SAML Configuration** upload the XML file downloaded. 6. Save and your OneLogin SSO is ready to go. # OpsGenie Alert Integration Source: https://docs.mezmo.com/docs/opsgenie-alert-integration Create OpsGenie alerts from Mezmo log data by configuring test, presence, and absence alerts with an OpsGenie API integration key. The OpsGenie alert integration interacts with the [OpsGenie Alert API ](https://docs.opsgenie.com/docs/alert-api)and enables users to create alerts in OpsGenie based off of log data in Mezmo. After clicking on OpsGenie- a window pops up with options for test, presence and absence alerts. Test alerts allow the user to test with a third party integration, presence alerts send alerts to the third party integration after a criteria is met (i.e. 5 or more matches within 30 seconds) and [absence alerts](https://www.mezmo.com/blog/logdna-absence-alerting) send an alert when there are fewer log lines than anticipated. To retrieve the API key from OpsGenie- go to Teams > Integrations > Add Integration > API Integration. Input the API key into the API key section of the Mezmo OpsGenie alert integration (shown above). After saving the alert or hitting test from your Mezmo window- the alert will now appear in OpsGenie under Alerts! The first alert is an example Mezmo test alert integrated with OpsGenie. **NOTE**: The US OpsGenie endpoint is used by default. If using the EU OpsGenie endpoint is required please reach out to [support@mezmo.com](mailto:support@mezmo.com). # Organization Management Overview Source: https://docs.mezmo.com/docs/organization-management-overview Create, delete, join, switch, and manage Mezmo organizations, including ownership, default organization, and API key settings. When you create your Mezmo account, Mezmo automatically creates and assigns you to a new Organization. This Organization comes with a 30-day trial and an auto-generated [ingestion key](/docs/ingestion-key) which you can use to send logs to Mezmo for storage and analysis. You can be a member of many Organizations, but can only view one organization at a time. Your current active organization appears in the organization switcher in the top navigation bar. Click the organization name to open the **Manage Organizations** menu. ## Create an Organization 1. Click the organization name. Then **All Organizations** You'll see a list of all the organizations you're a member of. 2. Then click **Create a new organization**. ## Delete an Organization Deleting an Organization will purge the Organization from Mezmo. This includes the Organization’s logs, billing data, and customizations. Only the Organization **Owner** can deactivate the Organization. Enterprise customers can't deactivate their own Organization. Please contact your Customer Support Manager or [support@mezmo.com](mailto:support@mezmo.com) for more information. This action can't be reversed. Make sure you want to delete your Organization before continuing. 1. Switch to the Organization in the Mezmo web app. 2. Navigate to **Settings > Organization > General**, then scroll down to **Deactivate Account**. 3. Click **Deactivate This Account**. Mezmo will prompt you to reenter your account ID for confirmation. ## Organization Ownership Creating an Organization automatically makes you the owner of the Organization. Organization owners can invite users to join or make the Organization discoverable to certain users. Learn more about [Managing Roles](/docs/rbac). ## Join an Existing Organization Mezmo allows you to join multiple Organizations. You can join or leave an organization by selecting the organization name and selecting Join or Leave. ## Manage an Organization If you are an owner you have the ability to modify the Organization. Click **Settings > Organization > General** in the web app to get started. ### Change the Organization Name Changing the Organization name updates the display name of the Organization in Mezmo for all members. ### Change the Organization Owner Changing the Organization owner passes ownership of the Organization to a different member. Once you do this, you will no longer have the ability to make changes such as changing the Organization name or deleting the Organization. ### Switching Between Organizations If you are a member of multiple Organizations, you can switch between them in the Mezmo web app. Only one Organization can be viewable at a time. The selected organization appears in the organization switcher in the top navigation bar. Click the organization name to open the **Manage Organizations** menu, then click the name of the organization you want to switch to. The web app will refresh and bring you to that organization’s log view. ### Set a Default Organization For users with access to multiple organizations, you can specify the organization to open when you log in to Mezmo. 1. To set your default organization, in the top navigation bar click the caret beside your current organization’s name in the organization switcher. The list of all organizations you have access to appears. 2. Find the name of the organization that you want as your default in the list, and hover your cursor over the name. 3. Click the heart icon that appears beside the organization name to set it as your default. ## API Keys Image API keys (Ingestion Keys and Service Accounts) are used to connect Mezmo to third-party applications and services, for ingesting log files and for API calls. Learn more about [Using Service and Ingestion Keys](/docs/ingestion-key). # Organize Visualizations by Category Source: https://docs.mezmo.com/docs/organize-visualizations-by-category Group Boards, Views, and Screens into Categories to organize your Mezmo visualizations and manage them under Settings. You can organize Boards, Views, and Screens by Categories. There are two ways to create Categories. * When you create the Board, Screen, or View and give it a name and category. * By going to the Categories screen and adding a new Category. Image ## Manage Categories * Manage all of your categories under Settings > Categories. * New categories will be empty. Drag Views, Boards, or Screens to Categories to organize information. ## Feature Notes * A View, Board, or Screen can be part of more than one Category. * Deleting a Category, does not delete the View, Board, or Screen. # PagerDuty Alert Integration Source: https://docs.mezmo.com/docs/pagerduty-alert-integration Connect Mezmo alerts to PagerDuty so log-based alerts trigger incidents, including how to test, delete, and the 400 KB payload limit. ## About PagerDuty Alert Integration Mezmo’s alert integration with PagerDuty interfaces with [PagerDuty Event API](https://developer.pagerduty.com/docs/events-api-v2/overview/) and allows alerts sent from Mezmo to trigger events in PagerDuty. ## Integrating with PagerDuty On the alerts creation page, select the PagerDuty logo. You will be directed to PagerDuty to authorize the integration, using your email address and password for your PagerDuty account. After authorization, you are prompted to select the PagerDuty service **logging alerts** and then click **Connect**. After completing the connection, remember to save your Mezmo alert so that your alerts will trigger in PagerDuty and new alerts display in your PagerDuty incidents dashboard. ## Testing Your PagerDuty Integration You can choose trigger a test alert with test data, even before saving your PagerDuty alert. As long as you have selected a connected source from PagerDuty, you will be able to create a test alert. ## Deleting Your PagerDuty Integration 1. Log in to the Mezmo Web App. 2. Go to **Settings > Integrations**. 3. Select your PagerDuty integration to open its **Settings** page. 4. Click the **X** next to the Service you want to delete. 5. Click \*\_Yes, delete \*\_in the confirmation dialog to remove the service. ## Limitations on PagerDuty data payload Because of [PagerDuty integration limitations](https://developer.pagerduty.com/docs/events-api-v2/overview/), we trim all requests to be at most 400 kilobytes. # PagerDuty Auto-resolve Settings Source: https://docs.mezmo.com/docs/pagerduty-auto-resolve Configure Mezmo to automatically resolve PagerDuty incidents based on log line volume within a specified time frame for an alert. ## About auto-resolving PagerDuty incidents With auto-resolution, you can configure settings to automatically resolve PagerDuty incidents based on the number of log lines that show up in the associated view during the specified time frame, rather than manually resolving incidents in the PagerDuty application. When a PagerDuty alert channel has auto-resolution settings configured and the conditions are met to auto-resolve the incidents, all incidents that were triggered from the associated PagerDuty alert channel will be resolved. For example, say you have a View that you use to look for specific incoming log lines. This might be a View that filters for a certain error type of log, so if you get more than 10, you consider that a problem that should be looked into. You attach an alert to the View for when 10 or more log lines are ingested and configure the alert to notify the PagerDuty app to create an incident when it happens. With PagerDuty auto-resolve, you can further specify that if fewer than 10 log lines come in within a certain time frame, you want PagerDuty to auto-resolve any previous incidents that were triggered (within the specified timeframe) by the alert, and move them to the “Resolved” area on the PagerDuty application UI. The auto-resolve settings are found at the bottom of the Alert modal for creating PagerDuty alerts. ## How to configure auto-resolution for PagerDuty incidents 1. On the [Alert modal](/docs/add-alerts-to-views#how-to-attach-an-alert-to-an-existing-view), click the **Auto-resolve incidents** option to toggle it "on." 2. Specify the two conditions for the trigger to auto-resolve: * the number of log lines to appear * the timeframe within which that number of log lines appears 3. Click **Save Alert**. * The auto-resolve option recognizes if the alert is a Presence alert (e.g. alert me when this specific type of log line appeared) or an Absence alert (e.g. alert me if this specific type of log line did NOT appear). Subsequently, the auto-resolve feature will look for "fewer than" the specified numbers of lines for a Presence alert, while for an Absence alert the auto-resolve functionality uses a "more than" value to trigger auto-resolution. * A recommendation for auto-resolution is to use the same number and time interval that was defined for the alert itself. Though each use case could be different; some users might want to auto-resolve incidents very quickly, while others may be more conservative and wait until a larger number of log lines come in. * There may be a delay of several minutes before the alert takes effect. # Parse Logs with Custom Templates Source: https://docs.mezmo.com/docs/parse-logs-with-custom-templates Create custom parsing rules and templates to parse log formats Mezmo doesn't auto-parse, and manage template order and status. You must be an [admin or owner](/docs/feature-access-matrix) to use this feature. Mezmo supports [common log types](/docs/ingestion#supported-types) and parses the lines automatically for you. However, if you have a log format that does not fit one of the supported log types, you can create your own parsing rules using [Create a Parsing Template](/docs/create-a-parsing-template) or do a one time log extraction using [Extract Fields](/docs/extract-fields). ## Feature Notes ### Multiple Templates You can have multiple custom parsing templates. Custom Parsing templates are applied to your logs based on the order of active templates. For example if you have two active templates that target the same log line, then the templates are applied in the order on the Manage Parsing page. * Template One * Template Two First, Template One will be applied to incoming logs, if there is a matching log line in Template Two, it will then be applied. You can change the order of active templates by dragging them on the Manage Parsing page. ### Template Status * **On** - Valid parsing templates that will be applied to your logs. * **Off** - Valid parsing templates that won't be applied to your logs. * **Draft** - Invalid parsing templates. These are either incomplete, contains errors or have not been validated. ## Tips and Troubleshooting For tips and troubleshooting information, check out [the Mezmo Log Analysis Support Knowledge Base](https://supportkb.mezmo.com/). # Parsing Operators Source: https://docs.mezmo.com/docs/parsing-operators Reference for the parsing operators used to extract, transform, and capture values when building custom parsing rules in Mezmo. Parsing operators are used to extract and manipulate data. ## Extract Values by Delimiters Extracts values between a single delimiter. It creates tokens that can be selected and used for the next operator in the rule path. One delimiter is accepted as an input and one or more matching tokens are returned as an output (One to Many). | Operator Fields | Description | | ----------------------------------------- | --------------------------------------------------------------- | | Delimiter | The delimiter that the logline or string should be split on. | | Preserve Delimiters Between (Start & End) | Preserve any encountered delimiters between start and end keys. | ## Extract Values Between Delimiters Extracts values between start and end delimiters. It creates tokens that can be selected and used for the next operator in the rule path. One token is accepted as an input, one or more matching tokens are returned as an output (One to Many). | Operator Fields | Description | | --------------- | ------------------------------------------------------------ | | Start Key | The delimiter that the logline or string should be split on. | | End Key | String delimiter for the end of an extraction. | ## Parse JSON Parse stringified json objects into objects. ## Parse CSV Turn CSV strings into a list. ## Concatenate Values by Delimiter Combines all of the selected values into a single token separated by a delimiter. At least two or more tokens are accepted as an input. One token is returned as an output (Many to One). | Operator Fields | Description | | --------------- | ------------------------------------------- | | Delimiter | The key to add between each token selected. | ## Trim Value Trims the selected value by specifying start and end indices. Indices are numbered from 0 to n. 0 represents the first character, n represents the last character. One token is accepted as an input and one token returned as an output (One to One). | Operator Fields | Description | | --------------- | ---------------------------------- | | Start | Start index for partial selection. | | End | End index for partial selection | ## Convert to Number Converts a string value to a number. One token is accepted as an input and one 1 token returned as an output (One to One). ## Capture in Field Captures the value in a field. This operator ends the rule path. The field name is required to successfully end the rule path. Field name has to be unique within the parsing template. 1 or more token is accepted as an input, 1 token returned as an output (One to One, or Many to One). ## Feature Notes After using Concatenate Values by Delimiter, Trim Value, Convert to Number, you cannot use Extract Values by Delimiters and Extract Values Between Delimiters operators in the same rule path. # PCI Compliance Source: https://docs.mezmo.com/docs/pci-compliance Understand how PCI DSS Requirements 3, 10, and 12 apply to your logging strategy and how Mezmo helps meet them as a PCI Level 1 Service Provider. The Payment Card Industry Data Security Standard (PCI DSS) establishes protections for credit cardholder data, including how it’s stored and accessed. This means that a significant portion of the standard—specifically, Requirement 10—governs auditing and logging your applications and systems. Complying with these regulations may seem daunting, but we’re here to help. As a PCI Level 1 Service Provider, we take compliance very seriously. We developed this guide to summarize your requirements for creating and managing log data. Note that this is not a legal document, but a guide to help you understand your potential obligations. Always check with your legal team before making any changes to your logging procedures or infrastructure. ## What PCI Means for Logging Logs are one of your primary tools in monitoring and auditing systems containing sensitive cardholder data. While logs may not directly contain cardholder data, they do contain critical information about the operation, state, and contents of systems that handle cardholder data. Implementing a PCI compliant logging strategy therefore involves: * Generating logs that can be used to audit systems * Logging data that allows you to trace user activity on these systems * Storing these logs securely and restricting access to authorized individuals Requirement 10 addresses these directly, but Requirements 3 and 12 also impact log data. We’ll explain each of these requirements and their sub-requirements in more detail. ## Requirement 3: Protect stored cardholder data Requirement 3 explains how cardholder data should be secured using techniques such as encryption, masking, and truncation. While logs most likely won’t contain cardholder data, you shouldn’t rule out the possibility. If a credit card number was accidentally leaked in a log file, it could result in significant penalties. To avoid this, any sensitive data that gets logged should be hashed or masked to avoid leaks. A common method involves using regular expressions to scan each log event and replace matches with a masked value. For example, if you use Fluentd, the [record transformer plugin ](https://docs.fluentd.org/filter/record_transformer)will replace a specific field with a different value that you specify. ### Requirement 10: Monitor access to network resources and cardholder data Requirement 10 outlines the necessary practices for auditing, monitoring, and alerting on user access to sensitive systems and data. It oversees the collection of log data, its contents, how it should be stored, how long it should be retained, and more. The goal is to ensure that any action can be traced back to a specific user or process. Let’s look at each requirement in more detail: **Requirement 10.1** requires that all access logs be linked to an individual user. Your logs should create an audit trail that can be traced from a particular event back to the user who initiated it. This means storing contextual user data in your logs, such as usernames or user IDs. **Requirements 10.2** and **10.3** list the types of events and associated data to collect in order to fully audit your systems. You will need to log any valid or invalid attempts to access cardholder data, any actions performed by a privileged account, access or changes to your logs and logging mechanisms, changes to identification or authentication mechanisms, and changes to system-level objects (such as databases or stored procedures). To create a complete audit trail, each log event should include a timestamp, the type of event and whether it succeeded, user identification, where the event occurred, and what component(s) it affected. Mezmo includes much of this data when logging from the agent or a code library, but more contextual data (like user IDs, event types, and successes or failures) may need to be added to your logging calls. **Requirement 10.5** enforces secure, unalterable audit trails. Mezmo prevents any modifications to your log data after ingestion, centralizes and encrypts your logs, and provides comprehensive authentication measures and role-based access control (RBAC) to prevent access from unauthorized users. **Requirements 10.6 & 10.8** involve regularly reviewing logs for suspicious activities and component failures (Requirement 10.8 only applies to service providers). PCI DSS allows for automated systems to be used in place of manual reviews, and with Mezmo alerts you can monitor your logs 24/7. If a security alert fires, make sure to follow up with your team to make sure the issue is investigated and addressed according to your organization’s security policies. **Requirement 10.7** specifies that at least the past 3 months of log data is immediately available for analysis. To increase your Mezmo retention period to 3 months (90 days) or more, please contact us. (Standard retention times are 7, 14, and 30 days.) The 10.7 requirement also specifies one year or more of cold log data for auditing purposes. To meet both the requirements for the past 3 months and the cold storage of 1 year, you can configure Mezmo to automatically generate nightly log archives and send them to your encrypted cloud storage bucket. ## Requirement 12: Maintain an information security policy PCI DSS requires organizations to main a security policy explaining how employees, contractors, and consultants is expected to treat cardholder data. While this requirement isn’t directly related to logging, logging can help with compliance. **Requirement 12.5.2** requires that security alerts are not only monitored, but distributed to the appropriate parties so that they can respond to the alert. For example, if an alert indicates a systems failure, this might involve exporting logs to the Ops team so that they can troubleshoot the affected system. These alerts and the individual(s) responsible for responding to them should be included in your incident response plan, according to Requirement 12.10.5 \*\_Requirement 12.8 \*\_defines your relationship with any service providers who might affect the security of cardholder data. This includes Mezmo. While we are committed to complying with PCI DSS, it’s your responsibility to perform due diligence (Requirement 12.8.3) and ensure that we are PCI DSS compliant at least once per year (Requirement 12.8.5). ## How Mezmo Supports PCI DSS Compliance Mezmo is committed to protecting your data. We have been audited by an independent PCI Qualified Security Assessor (QSA) and certified as a PCI Level 1 Service Provider. If you have any questions about compliance, please[ contact us](https://www.mezmo.com/contact-us). # Role-Based Access Control Source: https://docs.mezmo.com/docs/rbac Learn how Mezmo roles grant member privileges and how access scopes restrict which log data members can see. Members are users who have signed up for your Organization. You assign each member one or more [roles](/docs/manage-members) that grant privileges. A role can also carry access scopes that determine which logs the member can see. ## Roles Roles have default privileges. Mezmo supports four roles: * **Owner** (`owner`) - Each Mezmo Organization has one owner. The owner can't be restricted, has the most access, and manages both admins and members. * **Admin** (`admin`) - An Organization can have more than one admin. Admins have the second-highest level of access, can view all logs, and can't have their access restricted. * **Basic** (`basic`) - Standard members of your Organization (previously called "Member"). Their log access is governed by the access scopes on the roles they hold. * **Read-only** (`readonly`) - Read-only members can view logs, run searches, view screens, and export lines. They have no permission to make changes. A member can hold multiple roles. Review the [Feature Access Matrix](/docs/feature-access-matrix) for a breakdown of each role's privileges. ## Access Scopes Access scopes are [Log Analysis](/docs/about-mezmo-log-analysis) query strings attached to a role that limit which logs a member holding that role can see. A role can carry up to 10 access-scope queries. A role with no access scopes grants access to everything. For example, you can add an access-scope query that matches only logs from Node.js applications and attach it to a role. Members who hold that role see only Node.js logs. You manage roles and their access scopes at [**Settings > Organization > Roles**](https://app.mezmo.com/manage/roles). You can manage access scopes from the Role form and the Roles list: * In the Role form, **Permissions** and **Access Scopes** appear on separate tabs. * A role must grant at least one permission or at least one access scope. * In the Roles list, an **Access Scopes** badge marks any role that has access scopes. * Use the **Preview** link to test a query against live tail (the real-time streaming view of incoming logs) before you save. To learn more about query syntax, see [search log contents](/docs/searching-log-contents). ## Create or Edit a Role 1. Go to [**Settings > Organization > Roles**](https://app.mezmo.com/manage/roles). 2. Click **Create Role** to add a role, or click **Edit** next to an existing role. 3. Enter a name for the role. 4. On the **Permissions** tab, select the privileges the role grants. 5. On the **Access Scopes** tab, add up to 10 Log Analysis query strings to limit which logs the role can see. Leave this empty to grant access to everything. 6. Click **Save**. Use **Preview** to test a query against live tail before you save. ## View Members Assigned to a Role In the Roles list, the **Users** column shows the number of members assigned to each role. * Click the chevron on a role row to expand it and see the members who hold that role, listed by email address in alphabetical order. * If you expand a role with no members, it shows the message No users have this role. * Click a member's email in the expanded list to open the [Members](/docs/manage-members) page filtered to that member. * Click **Open in Members page** to open the [Members](/docs/manage-members) page filtered to that role. # Read Archived Logs Source: https://docs.mezmo.com/docs/read-archived-logs Re-ingest and parse archived JSON log files using Mezmo Data Restoration, Amazon Athena, Google BigQuery, IBM SQL Query, or jq. Mezmo log files are exported and stored in zipped JSON Log files format. There are several tools you can use to re-ingest and parse the historical data in your archived logs, including the Mezmo Data Restoration feature, Amazon Athena, Google BigQuery, IBM SQL Query, and jq. ## Mezmo Log Data Restoration Log Data Restoration let you re-ingest, or restore, archived logs from cold storage so you can search the log data in the Mezmo user interface. Restoration is useful for troubleshooting older bug tickets, as well as bringing up additional context from older logs beyond your retention period. You can find more detailed information in the [Restore Log Data](/docs/data-restoration) topic. ## Amazon Athena Amazon Athena is a serverless interactive query service that can analyze large datasets residing in S3 buckets. You can use Amazon Athena to define a schema and query results using SQL. More information about Amazon Athena is available in the[ AWS Athena documentation](https://aws.amazon.com/athena/). ## Google BigQuery Google BigQuery is a serverless enterprise data warehouse that can analyze large datasets. You can find more information about Google Big Query in [the official documentation](https://cloud.google.com/bigquery/). ## IBM SQL Query [IBM SQL Query](https://www.ibm.com/cloud/sql-query) is a serverless data processing and analytics service for large volumes of data stored on IBM Cloud Object Storage. This [blog article](https://www.ibm.com/blog/analyze-logdna-log-data-on-ibm-cloud-object-storage-using-ibm-cloud-sql-query/) provides detailed information for using IBM SQL Query with Mezmo data in IBM Cloud. ## jq jq is a handy command-line tool used to parse JSON data. Once your archive has been uncompressed, you can use jq to parse your archive log files. You can find more information about jq on their [Github page](https://stedolan.github.io/jq/). # Search and Filter Log Data Source: https://docs.mezmo.com/docs/search-and-filter Learn how search terms and operators work in the Log Viewer, plus tips for resolving empty results, unexpected matches, and query warnings. Learn how to search and filter log data using the Log Viewer. ## How Search Works A **search** term is a string consisting of a single word, or a phrase surrounded by quotes. Terms are searched against the entire log, but can be searched on a particular field if one is specified. By default, terms are also case insensitive and treated as prefixes. For example, search `lin` will return logs containing `lin`, `line`, and `Linux`. **Operators** change how search terms are interpreted. Using operators gives you greater control over the search process in order to provide more relevant results. ## How To Search * Learn about filters and operators in [Search Log Contents](/docs/searching-log-contents). * Learn how to search JSON (nested fields) in [Search Log Fields](/docs/search-json-fields). ## Search Tips ### No Search Results Found Searches that return no results are usually caused by searches that are too vague. For example, searching for `the` or another common word would result in an excessive number of results, and not all of them will be displayed. For this reason, Mezmo doesn’t support single-character searches. To avoid this problem, try using the exact match operator such as equal (`:==`) instead of the prefix match operator (`:`). You should also search on a specific field, instead of the entire log. Lastly, if you know exactly how your search term appears in your logs, you can enable case sensitivity by using the `:===` operator. Learn more in [Search Tips](/docs/search-tips). ### Unexpected Results Mezmo searches visible fields as well as internal fields, such as metadata and the raw log message. Your query can match on these "invisible" fields, resulting in the log appearing in the results. You can override this by specifying the field to search. For example, if your logs contain a `level` field and you want to find error-level logs, enter `level:error` as your query instead of just `error`. ### Search Query Warnings The warning `Unsupported visualizations query: symbols` is displayed whenever any symbols are used in a query, for example, `/` , `+`, etc. There is no issue with your query and search results. The warning shows up to let you know that the query was performed without the symbols to populate the Graphs or Timeline Graphs. To search for symbols they must be wrapped in quotes. ### Camel Cased Words And Case Sensitivity When logs lines are stored and indexed, compound words with internal capitalization like `SyntaxError` get split up into separate words, `Syntax` and `Error`. This means that performing a search (defaults to case insensitive) for `syntaxerror` may not return lines containing that term because it was not stored that way. You will need to query for 'SyntaxError' to match the original form in this case. ### Search Feedback When searching, the Mezmo app will provide feedback if a search is correct. **Field Search Correct Syntax** The search will highlight a yellow color. Image **Field Search Incorrect Syntax** The search will have a red underline and offer tips by clicking on the red triangle. Image **Simple Search Feedback** Simple text searches do not show if a search is correct or not. ## Troubleshooting and Tips For tips and troubleshooting information, check out [the Mezmo Log Analysis Support Knowledge Base](https://supportkb.mezmo.com). # Search Log Fields Source: https://docs.mezmo.com/docs/search-json-fields Search parsed and nested log fields using value, comparison, existence, term-match, and case-sensitive operators, plus metadata and line-size searches. Mezmo provides several capabilities for searching fields in your logs. In this topic you'll find information on nested field searches, searching by field comparison, searching for the existence of fields, and general and exact term field searches. ## Access Search 1. Log in to [app.mezmo.com](https://app.mezmo.com). 2. In the **Search** box at the bottom of the log viewer, enter your search terms. 3. Select the **Timeframe** that you want to search. 4. Select if you want to search **Live** log data, or historical. 5. In the **Viewer Tools** menu, enter any text you want highlighted in the search results. Image ## JSON Field Search To search for a field with a particular value, use a colon to separate the field and value. This example will return all parsed log lines with the field `response` with a value of `404`. ```bash theme={null} response:404 ``` ### Nested Field Search To search for a nested field, use periods to separate each nested field. This example will return all log lines containing the key:value structure `{ "user": { "id": 12345 }}.` ```bash theme={null} user.id:12345 ``` ## Filters Using the same field search syntax, you can also set filters directly in the search bar. This example will return all log lines that originate from the source `myawesomehost` and not from the app `mycoolapp`. ```bash theme={null} host:myawesomehost -app:mycoolapp ``` ## Metadata With the REST API or Node.js library, you can upload a metadata object as part of a log line's context. To search for field values contained in the metadata object, use the `meta` prefix. This example will return all log lines containing the context object with the key:value structure `{ "status_code": 404}.` ```bash theme={null} meta.status_code:404 ``` ## Field Comparison Operators For parsed fields with a numeric value, we support the following operators: ```bash theme={null} * = * < * > * <= * >= ``` To search for parsed fields matching comparison operators, use a colon followed by the comparison operator. ```bash theme={null} response:>=400 ``` ## Compound Field Comparison Search To form a compound field search query using comparison operators, use a colon followed by parentheses. This example will return all log lines with the field `response` with values greater than or equal to `400`, less than `500`, and not `404`. ```bash theme={null} response:(>=400 <500 -404) ``` ## Case-Sensitive Field Search To search for a case-sensitive parsed field, use a colon followed by an equal sign `=`. This example will return all log lines with the field `name` with the case-sensitive string value `camelCasedName`. ```bash theme={null} name:=camelCasedName ``` ## Existence Field Search To search for the existence of a parsed field, use a colon followed by the asterisk `*.` This example will return all log lines that have a value for the `user` field. ```bash theme={null} user:* ``` ## Term Match Field Search To search for a term match for a field value, use `==`. This example will return all lines with the exact name field value of `bob`, and will not match `bobby`. ```bash theme={null} name:==bob ``` ## Term Match Case-Sensitive Field Search Prefix search is set by default for all string fields. To search for an **exact** match for a field value, use `===`. This example will return all lines with the exact name field value of `Bob`, and will not match `bob` or `Bobby`. ```bash theme={null} name:===Bob ``` ## Line Size Search You can search for log lines by size by using the `mezmo_line_size` field annotation. This example will return lines with a line-size greater than 4000 bytes. ```bash theme={null} _mezmo_line_size:>4000 ``` You can create a search based on the size of specific log line by clicking the **Line size** value in the Log Viewer. You can also create graphs based on `mezmo_line_size` as described in [Create a Graph](/docs/create-a-graph). ## Colons Since the colon is a reserved character for field search, quotes are required when searching for strings with colons in them. This example will return all log lines with the string `response:` in them. ```bash theme={null} "response:" ``` ## Combining Operators You can combine operators to make your search more specific. This searches first for logs scanned from`/var/log/syslog` or that contain the word `ERROR`, then limits the results to `node1.` ```bash theme={null} source:node1 AND (file:/var/log/syslog OR ERROR) ``` This searches logs where the value stored in the status field is greater than 100 and less than 503. ```bash theme={null} status:(>100 AND <= 503) ``` ## Lists Any whitespace between search terms is automatically interpreted as AND. For example, searching warning error returns logs containing both warning and error. The only exception is when using lists, which treat whitespace as a part of the search term. For example, searching `message:[file, exists]` will only search for instances of exists that are preceded by a space. Some other examples: * `level:[warning,error]` will return as normal. * `level:[warning, error]`with a space between warning and error, will search for entries of the field level which have "warning" or " error" with space included. * `level:[warning,(error)]` will search for entries of the field level which have "warning" or "(error)" with parenthesis included. # Search Tips Source: https://docs.mezmo.com/docs/search-tips Troubleshoot vague queries, unexpected matches, symbol warnings, and case-sensitivity issues, and interpret Mezmo's search syntax feedback. Searches that return no results are usually caused by searches that are too vague. For example, searching for `the` or another common word would result in an excessive number of results, and not all of them will be displayed. For this reason, Mezmo doesn’t support single-character searches. To avoid this problem, try using the exact match operator such as equal (`:==`) instead of the prefix match operator (`:`). You should also search on a specific field, instead of the entire log. Lastly, if you know exactly how your search term appears in your logs, you can enable case sensitivity by using the `:===` operator. ### Unexpected Results Mezmo searches visible fields as well as internal fields, such as metadata and the raw log message. Your query can match on these "invisible" fields, resulting in the log appearing in the results. You can override this by specifying the field to search. For example, if your logs contain a `level` field and you want to find error-level logs, enter `level:error` as your query instead of just `error`. ### Search Query Warnings The warning `Unsupported visualizations query: symbols` is displayed whenever any symbols are used in a query, for example, `/` , `+`, etc. There is no issue with your query and search results. The warning shows up to let you know that the query was performed without the symbols to populate the Graphs or Timeline Graphs. To search for symbols in they must be wrapped in quotes. ### Camel Cased Words And Case Sensitivity When logs lines are stored and indexed, compound words with internal capitalization like `SyntaxError` get split up into separate words, `Syntax` and `Error`. This means that performing a search (defaults to case insensitive) for `syntaxerror` may not return lines containing that term because it was not stored that way. You will need to query for 'SyntaxError' to match the original form in this case. ### Search Feedback When searching, the Mezmo app will provide feedback if a search is correct. **Field Search Correct Syntax** The search will highlight a yellow color. Image **Field Search Incorrect Syntax** The search will have a red underline and offer tips by clicking on the red triangle. Image **Simple Search Feedback** Simple text searches do not show if a search is correct or not. # Search Log Contents Source: https://docs.mezmo.com/docs/searching-log-contents Use terms, phrases, and AND/OR/NOT operators with grouping and precedence to search log contents, plus Time Search and special-character handling. Mezmo provides advanced capabilities for searching the contents of your logs. In this topic you'll find detailed information about search operators with examples of search queries, using Time Search, and how to work with special characters in your search queries. ## Access Search 1. Log in to [app.mezmo.com](https://app.mezmo.com). 2. In the **Search** box at the bottom of the log viewer, enter your query. 3. Select the **Timeframe** that you want to search. 4. Select if you want to search **Live** log data, or historical. 5. In the **Viewer Tools** menu, enter any text you want highlighted in the search results. Image ## Search ### Introduction A **query** is composed of **terms**, **phrases**, and **operators**. In this example, ```none theme={null} ClassCastException OR "Null Pointer Exception" ``` the **query** can be broken up into the following: * `ClassCastException` is a **term**. * `"Null Pointer Exception"` is a **phrase**. * `OR` is an **operator**. * This **query** will filter for any lines that contains the text `ClassCastException`, `Null Pointer Exception`, or both. ### Terms A term will match any line that contains that term anywhere within the line. By default every term match is case insensitive and a prefix match. This behavior can be changed with the case sensitive operator and exact match operators respectively. For example all of these queries, ```none theme={null} BAR barrier ===barrier ``` will match the following line: ```none theme={null} We really like the barrier reef ``` ### AND Operator **Implicit AND** Any whitespace between terms is implicitly interpreted as `AND`. For example, the query `file does not exist` will return any log containing all four words, regardless of their order. To search for the exact phrase, surround it in double quotes, `"file does not exists"` The only exception is when using lists, which treat whitespace as a part of the search term. **Explicit AND** You can also add AND as an operator to your search query to explicitly combine terms. ```bash theme={null} healthcheck AND -successful ``` This example query will look for the word `healthcheck` AND log lines that **do not** contain the word `successful`. #### Explicit AND You can also add AND as an operator to your search query to explicitly combine terms. ```bash theme={null} healthcheck AND -successful ``` This example query will look for the word `healthcheck` AND log lines that **do not** contain the word `successful`. ### OR Operator Specifying the **OR** operator returns results with either term. ```bash theme={null} healthcheck OR ping ``` This example query will return all log lines that contain the word `healthcheck` or contain the word `ping.` ### Chained Operators You can chain operators together for specific searches. See the section Order of Operators for more information on the order of operator execution. ```bash theme={null} healthcheck -successful OR ping ``` This query example will return lines with `healthcheck`, then lines that **do not** have the word `successful` or `ping`. ### Grouping Use parentheses to explicitly specify operator precedence for your search terms, Queries in parenthesis are executed first. ```bash theme={null} app:memo -("success" OR "waiting") ``` This will search in the Mezmo app and exclude log line containing success or waiting. ### Order of Operations Search query operations are executed in this order: 1. `( )` Grouping 2. `-` Not 3. `OR` 4. `AND` or implicit white space between terms The query `me myself OR I` is equivalent to `me AND (myself OR I)` since **OR** operators have higher precedence than **AND** operators. The query `(me myself) OR I` is equivalent to `(me AND myself) OR I` since grouping operators have higher precedence than OR operators. All remaining operators are at the same precedence as the grouping operators. ## Time Search You can search for logs that were generated during specific or broad time frames in the **Jump to Timeframe** area in the log **View**. You can use both text and numeric values, For example: * `last friday` * `yesterday 1pm` * `5:4am` * `2/24 2:30pm` ## Special Characters ### Escaping Special Characters Put words in parenthesis to escape special characters such as white space. Unless escaped, terms are combined by whitespace and use AND to search. This query example will look for `403 Forbidden` and `login:` `"403 Forbidden" login` ### Symbols By default, all symbols in queries are matched exactly. In this example, the query will return all logs that contain the term `%VARIABLE%` : `%VARIABLE%` ## Filter Logs Filters can be found on the [Views page](https://app.logdna.com/logs/view) in your Mezmo app. * **Tags -** A tag can be used to group lines and more than one tag can be applied to a given line. * **Source -** Contains a list of your logging sources. A source can be a host, computer, virtual machine, or Heroku app. Source metadata such as IP address or OS may be displayed with the source. * **App** - Contains a list of logging buckets. An app can represent a log file, program or container. Typically, log lines within an app are inherently similar. * **Level** - Contains a list of parsed log levels, such as `INFO`, `DEBUG`, or `ERROR`. Log levels are automatically parsed from log lines using standard log level detection and common patterns. ### Use Filters To use filters, click on the desired filter drop-down menu and tick the checkboxes of the entries you are interested in. Selecting more than one checkbox within a filter will include log lines that belong to any of the selected entries. Selecting checkboxes in more than one filter, such as selecting 1 app and 1 source, will return log lines that match both that app and that source. * Selected entries within such as selecting two sources, use OR * Selected entries across filters such as selecting a source and an app, use AND ```bash theme={null} (host1 AND website) OR (host1 AND react_app) OR (host2 AND website) OR (host2 AND react_app). ``` If you select two sources, `host1` and `host2` , then select two apps, `website` and `react_app` . Log lines matching these conditions will be returned. ```bash theme={null} (host1 AND website) OR (host1 AND react_app) OR (host2 AND website) OR (host2 AND react_app). ``` # Server Analytics Screen Source: https://docs.mezmo.com/docs/server-analytics-screen Build a Screen with Time Shifted Graph and Table widgets to monitor bytes sent to the server and the top response codes. You'll create a Screen that monitors bytes sent to the server and top response codes. ## Bytes Sent 1. Create a new Screen. 2. Add a **Time Shifted Graph** widget. 3. Set the Data field to bytes. 4. Under Field Value: 1. Set it to Any 2. Set Advanced Filtering to `request:*` 5. Change Operation to Cumulative. 6. Change Duration to Today vs Yesterday. 7. Change Appearance label to `Bytes Sent Daily`. ## Top Response Codes 1. Create a **Table** widget. 2. Under Data: 1. Group by Response 2. Set Field to All Lines and add `response:* request:*` under Advanced Filtering. 3. Leave Operation as Counts. 3. Set Duration to Last 1 Day. 4. Under Data Formatting: 1. Set the Number of Rows to 10. 2. Set Sorting Descending. 5. Under Appearance: 1. Change the label to Top 10 Response Codes 2. Change Left Column to Response 3. Change Right Column to Count 6. Uncheck Display as Bar Chart. Save your Screen. Give it the name **Server Health** and add the category, **Web App Monitoring**. # Server Security Screen Source: https://docs.mezmo.com/docs/server-security-screen Build a Screen with widgets to track 401/403 errors and identify the top IP addresses generating forbidden requests. You'll monitor the number of 401 and 429 errors, maintain a list of IP address causing those errors. ## Daily 400 Errors 1. Create a **Time Shifted Graph** widget. 2. Under Data: 1. Set the field to All Lines. 2. Add the query `response:401 OR response:403 request:*` 3. Leave Operation as Counts. 3. Set Duration to `Today vs Yesterday`. 4. Set the Appearance label to`Access Forbidden/Permission Denied Requests (Daily)`. ## Top IP Address 1. Create a **Table** Widget. 2. Under Data: 1. Group by `clientip`. 2. Set the field to `All Lines` and add the query `response:401 OR response:403 request:*`. 3. Leave Operation as Counts. 3. Set Duration to `Last 1 Day`. 4. Data format can left at the defaults. 5. Under Appearance: 1. Set the Table label to `Top Offending 401/403 IPs`. 2. Left Column Label to `Client IP`. 3. Right Column Label to `Request Counts`. 4. Turn on Display as a bar chart. Save your Screen. Give it the name **Server Security** and add the category, **Web App Monitoring**. # Slack Alert Integration Source: https://docs.mezmo.com/docs/slack-alert-integration Connect Mezmo log alerts to your Slack workspace, configure presence and absence alerting, and remove the integration. The Slack alert integration enables users to easily integrate Slack alerts on log data with their Slack workspace. After attaching an alert and selecting view-specific alerts- Mezmo provides you with several options for third-party integrations and one of these is Slack: After clicking on Slack- a window pops up with options for test alerts, presence and absence alerting. Test alerts allow the user to check whether the third party integration is working, presence alerts send alerts to the third party integration after a criteria is met (i.e. 5 or more matches within 30 seconds) and [absence alerts](https://www.mezmo.com/blog/logdna-absence-alerting) sends an alert when there are fewer log lines than anticipated. The user can select the Slack channel for sending alerts, the color of the message, the type of alerting, and the number of matches and seconds after which to send the alert. An example of setting up presence match alerting with Slack- containing the log source, the number of matched lines as well as the log-lines: ## Delete Your Integration 1. Log in to the Mezmo Web App. 2. Go to **Settings > Integrations**. 3. Select your Slack integration to open its **Settings** page. 4. Click the **X** next to the channel you want to delete. 5. Click **Yes, delete** in the confirmation dialog to remove the channel. # SOC 2 Compliance Source: https://docs.mezmo.com/docs/soc-2-compliance Understand the SOC 2 Trust Services Criteria and how Mezmo logging helps you meet and maintain SOC 2 compliance. More and more industries are turning towards cloud computing services as a way to process more data at lower costs. However, organizations that handle sensitive data need assurance from service providers that their data will be strongly protected. While regulations like [HIPAA](/docs/hipaa-compliance) and [the GDPR](/docs/gdpr-compliance) create strict rules for specific industries and regions, there is no single federal law enforcing data protection in the U.S. To fill this gap, many service providers including Mezmo offer Service Organization Controls (SOC) 2 compliance, which describes the policies and practices that a provider has in place for protecting customer data. In this guide, we’ll explain how you can use Mezmo to become and remain compliant with SOC 2. This document does not provide legal advice, but is a general purpose guide to help you identify and understand your potential obligations. Always check with your legal team before making any changes to your operations. ## Background Information on SOC and SSAE SOC is neither a regulation or a standard, but a report that describes an organization’s internal controls over data managed on behalf of their users. It stems from the Statement on Standards for Attestation Engagements (SSAE), which is an auditing standard for service providers (called service organizations) and maintained by the American Institute of Certified Public Accounts (AICPA). SSAE requires service organizations to describe the systems, controls, and processes that they have in place for protecting and maintaining the integrity of data belonging to their customers (called user entities). To comply with SSAE, a service organization is audited by a third party CPA. The CPA reviews the organization’s controls over data and documents their findings in a SOC report. SOC 1 reports describe these controls and how they could affect a user entity’s financial reporting capabilities. SOC 2 reports, on the other hand, describe how the service organization handles all types of data, not just data related to its user’s finances. SOC 1 and SOC 2 reports come in two types: * **Type I** reports describe the service organization’s controls at a particular point in time. This shows that the controls in place are properly designed. * **Type II** reports describe the effectiveness of the organization’s controls over a period of time. These are generally preferred, since they indicate that the controls are most likely in place and working as intended every day. ### Trust Services Criteria SOC 2 reports are based on a set of criteria called the Trust Services Criteria (TSC). These criteria outline how the organization was evaluated and reported on. These criteria fall into five categories: security, availability, processing integrity, confidentiality, and privacy. We’ll explain each of these categories in greater detail and show how logs can help you meet the criteria in each category. ## How Logs Factor into SOC 2 Compliance The purpose of a SOC 2 Type II report is to show that your systems and processes operated securely over a period of time. This means having the ability to monitor your infrastructure, identify unusual events or security incidents, and troubleshoot problems. Logs play a vital role in this process, since they store highly detailed records of infrastructure operations and events over a period of time. This makes them ideal for reviewing and auditing both current and past operations. Note that the quoted passages in this section are from the official [Trust Services Criteria Publication](https://www.aicpa.org/content/dam/aicpa/interestareas/frc/assuranceadvisoryservices/downloadabledocuments/trust-services-criteria.pdf) from the AICPA. ### Security Security involves protecting information and systems “against unauthorized access, unauthorized disclosure of information, and damage to systems that could compromise the availability, integrity, confidentiality, and privacy of information or systems and affect the entity’s ability to meet its objectives.” Strong security controls protect data—and the systems that handle this data—from being accessed or modified by an unauthorized entity. Security is the only category required for SOC 2 compliance. A vital use of logs is monitoring for security events. This can include user logins, software modifications, changes to system settings and processes, and changes to your organization’s network. Many critical security events are already logged by the operating system; for example, the “auth.log” file found in most Linux distributions records all login attempts and any administrator-initiated actions, making it essential for monitoring system-level access. Security events must be monitored for signs of suspicious activity. With Mezmo, you can use [Alerts](/docs/add-alerts-to-views) to continuously scan incoming logs and send a notification if an anomaly is detected. For instance, if a user logs into a server containing sensitive data as an administrator, you may want to notify your organization’s security team to investigate further. The team can then audit the event, find out what the user did, and in case of a breach, determine the severity and scope of the incident. ### Availability Availability is the assurance that “information and systems are available for operation and use to meet the entity’s objectives.” Limited availability of even a single component can have numerous effects on your total operations, including: * Limiting customers’ access to their data * Reducing the availability of other components * Reducing trust in your services The problem may be as simple as a network connection failure, or as critical as a component failure, but without a way of monitoring availability, your team may spend hours diagnosing the cause of the problem. Monitoring for availability is straightforward with logs. Nearly all systems, devices, and applications generate log data, and a lack of log data is a fairly reliable indicator of a communication or operational problem. [Absence alerts](https://mezmo.com/blog/mezmo-absence-alerting/) are a useful method of monitoring availability, since they will raise a notification when log volume drops below a certain level over a period of time. Using logs to monitor availability won’t just alert you to a problem, but it will also give you the means to troubleshoot and identify the cause of the problem. ### Processing Integrity Processing integrity ensures that systems “perform their intended functions in an unimpaired manner, free from error, delay, omission, and unauthorized or inadvertent manipulation.” If a system fails to process data correctly, or if it does so slowly, it can rapidly erode customer trust in your service. The challenge to monitoring processor integrity is knowing when an error occurs. If you tried to monitor the timing and accuracy of every single processing activity in your system, your logs would quickly grow unwieldy. Instead, you might consider logging processing activities for key systems, especially systems containing or operating on sensitive customer data. In addition, you can use trace logs to monitor the flow of data throughout the application and ensure each step is working as intended. In our blog post on [Challenges Logging Services Applications](https://www.mezmo.com/blog/challenges-with-logging-serverless-applications) we showed how logging contextual data makes it easy to index logs by request or other custom fields. You can then use the [Mezmo web app](https://app.Mezmo.com/) to quickly filter on these fields in order to audit a specific request without having to comb through your other logs. ### Confidentiality and Privacy Confidentiality and privacy are two different criteria with a similar focus: protecting sensitive information from unauthorized disclosure. The main difference is that “privacy applies only to personal information, whereas confidentiality applies to various types of sensitive information.” The privacy criteria also sets the requirements for collecting, using, retaining, disclosing, and disposing of data, similar to how [the GDPR](https://www.mezmo.com/gdpr) sets restrictions on the use of personal data belonging to EU citizens. Privacy is one of the most important categories to monitor since it involves a number of scenarios regarding the use of customer data. For example, criteria 2 (choice and consent) requires you to communicate the choices that your data subjects have over the use of their personal information. If a data subject grants or revokes their consent, logging this action creates a historical record in case of a dispute or any uncertainty in how a subject’s data should be handled. The importance of monitoring these types of activities is also outlined in criteria 8 (monitoring and enforcement). Logging events affecting user data is not only a good practice, but it can also help protect you in case of a dispute. ### Other Considerations There are several additional points to keep in mind when logging for SOC 2 compliance. Depending on how long your audit period is, you may need to retain logs for six months or longer. Mezmo offers 30 days of retention, but you can use the [archive feature](/docs/archiving) to export logs to a cloud storage service indefinitely. Although archives can’t be searched in the Mezmo web app, you can use [one of many tools](https://www.mezmo.com/blog/how-to-search-through-mezmo-archives) to parse and search them outside of Mezmo. In addition, we recommend using views to control and filter your log stream. Auditing an entire infrastructure is no small task, and SOC 2 specifies many different categories of events to analyze. [Using views effectively](https://www.mezmo.com/blog/guide-mezmo-views) will help simplify and streamline the log analysis process, both for your organization and for auditors. ## How Mezmo Supports SOC 2 Compliance Whether you’re following all five criteria categories or just the common criteria, logging should be a key part of your SOC 2 compliance strategy. Mezmo provides a comprehensive platform for collecting, monitoring, and analyzing logs across your entire infrastructure. We became [SOC 2 compliant](https://www.mezmo.com/compliance) back in 2017, and we want to make sure that our platform can help you reach your SOC 2 compliance targets. If you have any questions, including questions about Mezmo’s SOC 2 compliance, please feel free to [contact us](https://www.mezmo.com/contact). # Enable Raw Line Storage Source: https://docs.mezmo.com/docs/store-and-show-raw-lines Turn on storing and viewing raw, unparsed log lines for your account and understand the potential storage cost impact. Mezmo [automatically parses log lines](/docs/log-parsing) received from [various sources](/docs/ingestion-integrations) to surface the content of log lines, so your account isn’t charged for retaining the entirety of bulky log line formats. However, there may be situations where you want to receive the raw line in its entirety, including the original format. In those situations, you can enable saving and viewing raw lines for your account. The option to store and show raw log lines is only available to Owners and Admins of the organization. The toggle to turn this selection on and off will not be visible to other roles. Enabling this feature may result in the retention of more data in your account, and your storage costs may increase accordingly. Most accounts will only see a minor increase in storage costs, but certain log formats will increase costs more than others. Github and Akamai as log sources tend to be more verbose in the raw form, and are anticipated to lead to greater cost increases. Other log sources that have large raw payloads may also result in increased costs. 1. In the left-hand navigation in the [Web App](https://app.mezmo.com), go to **Settings > Organizations > General**. 2. Under **Store and Show Raw Line**, move the toggle switch to **On**. Within five minutes of toggling this feature on, your raw log lines will be sent to storage. # Sysdig Alert Integration Source: https://docs.mezmo.com/docs/sysdig-alert-integration Configure Mezmo presence alerts that trigger events in your Sysdig Monitor instance using an API key and instance URL. ## About Sysdig Alert Integration Mezmo integrates with Sysdig and allows you to configure alerts, based on log lines ingested by Mezmo, that trigger new events in your Sysdig instance. This integration provides access to custom log metrics data that is useful in debugging and monitoring the health of a system. For example, you can monitor deployments with a Mezmo view that queries for specific errors during deployment. Then attach to the view an integrated alert that is configured to trigger when more than the expected number of logs with that specific error appear. The integrated alert then sends that event to the Sysdig Events feed. ### Considerations * Alerts are supported only for Sysdig Monitor events. * Only presence alerts, not absence alerts, are supported. * The Sysdig event timestamp will reflect when the alert is received by Sysdig. This will be displayed in the timezone as configured in the Sysdig UI (either local, or UTC) * The event "description" field will contain a formatted version of the first log line that triggered the alert. Note that the timestamp of this line will differ from the Sysdig event timestamp depending on the configuration of the Mezmo alert. ## Integrating with Sysdig 1. In either the View-specific or preset Alert modal, select the Sysdig logo to configure a Sysdig alert channel. 2. On the Alert modal, define the alert properties. 3. Enter the Sysdig **API key**. You can find this value by clicking on your user icon in the lower left corner of the Sysdig Monitor UI, and then selecting **Settings** to display your User Profile page with the "Sysdig Monitor API Token." 4. Enter the **Sysdig instance URL**. This URL is the base URL for your Sysdig Monitor instance. You can choose from the dropdown list of available URLs, or enter your own (for on-premise or other Sysdig installations) 5. Select the **Severity** level for the alert, as you want it to appear in the Sysdig application UI. For example, if the query upon which your view is based searches for multiple failed log-in attempts, you could select **High** as the severity label. By default, the **Severity** level is set to `info`. 6. Click **Save Alert**. When the parameters of the alert's conditions are met, the Sysdig UI displays the alerts in the Events feed. # Set Up System Cross-Domain Identity Management (SCIM) Source: https://docs.mezmo.com/docs/system-cross-domain-identity-management Automate user and group provisioning in Mezmo from identity providers like Okta and Microsoft Entra ID using SCIM 2.0. This feature is only available for customers on an Enterprise plan, and is restricted to **Owner** and **Admin** user roles. Please contact your Customer Support Manager or [support@mezmo.com](mailto:support@mezmo.com) for more information. ## What is SCIM **System for Cross-domain Identity Management** (SCIM) is a set of application-level protocols that use JSON, REST, and several different authentication methods to automate the task of account provisioning. Using SCIM you can provision/de-provision user accounts in Mezmo via your identity providers, such as Okta and Azure. ## How SCIM works with Enterprise Accounts and Child Orgs Mezmo's implementation of SCIM allows you to map Groups in your IDP with Child Organizations in Mezmo. It also maps users in your IDP to users in Mezmo. Assignments of these users to groups in your IDP will be reflected by Mezmo assigning these users to the matching Org. The Groups on this page are identity-provider groups used to provision users into child organizations. They are unrelated to the RBAC access-control groups that have been removed from Mezmo. ### Automatic Role Assignment Users provisioned via SCIM are automatically assigned the Basic role in their child organizations. This role grants standard log and feature access. For more information about roles and permissions, see [Role-based Access Control](/docs/rbac). ### Role Preservation During User Lifecycle When users are deactivated through your identity provider, Mezmo preserves their role assignments and permissions. If the user is later reactivated: * Previous roles are automatically restored * Custom permissions are reinstated * Access to the same child organizations is restored ### Managing User Roles You can manage user roles through: * **Identity Provider Groups** - Users assigned to different groups in your IDP will have access to corresponding child organizations in Mezmo Owner and Admin roles cannot be assigned through SCIM provisioning and must be set manually in the Mezmo Web App. ### Account Owner Protection If a user owns child organizations, you cannot: * Deactivate that user * Remove that user from a child organization they own If you try to do either of these through your identity provider, the SCIM API returns a 409 CONFLICT error with details about which organizations are affected. To deactivate or remove a user who owns child organizations, you'll need to transfer ownership to another user in the Mezmo Web App first. Once that's done, you can retry the SCIM operation through your identity provider. For more information about transferring ownership, contact your Customer Support Manager. ## General Setup To utilize SCIM, you will need to obtain an Enterprise Service Key. You can find this in the **Access Management** section of the Mezmo Web App in your **Enterprise Dashboard**. You will need this key, as well as our SCIM endpoint: `https://api.mezmo.com/v1/enterprise/scim` ## Support While our SCIM endpoint was built to SCIM 2.0 specifications, we have only tested it with the following IDPs. You may use other IDPs, but we cannot guarantee support for untested providers. Please note that while we support group sync, we do not support creation and deletion of groups. ## Okta 1. Log into to your Okta Admin Console 2. Create an App Integration (skip this step if you have already done this):Follow the steps in our [Okta SAML instructions](/docs/okta-saml-setup) ### Enable SCIM Provisioning 1. Go into your Mezmo App Integration. 2. Go to **General** and click **Edit** in **App Settings.** 3. Under **Provisioning**, choose **SCIM** and click **Save**, ### Configure Provisioning Settings 1. In the Mezmo App Integration, go to **Provisioning**, and under **SCIM connection**, click **Edit**. 2. For **SCIM connector base URL** enter `https://api.mezmo.com/v1/enterprise/scim` 3. For **Unique identifier field for user** enter `userName.` 4. Under **Supported provisioning actions,** select all options. 5. For **Authentication Mode,** choose **HTTP Header.** 6. In the field labeled **Bearer,** enter your enterprise token. 7. Click **Test Connector Configuration**. 8. Assuming everything succeeds, click **Save.** ### Configure **To App** settings 1. Click **Edit** next to **Provisioning to App.** 2. Select all options except **sync password.** 3. Click **Save**. ### Set Up Initial Connection of Users/Groups 1. In the **Import** tab, click **Import Now.** 2. If required, choose a method to reconcile users. ### Set Up Links between Okta and Mezmo Groups 1. Create a group and assign **App Mezmo** to this group (you may have already done this if you are using Mezmo SAML). 2. In the Mezmo Application, click**Push Groups**, then click the gear configuration icon. 3. Clear the option **Rename app groups to match group name in Okta.** 4. Choose **Push Groups**, then **By Name**. 5. Find the group you created. 6. Clear the option **Push group memberships immediately.** 7. Choose **Link group** and choose the Mezmo account you want to sync. 8. Click **Save**. 9. Assign people to groups as necessary. ### Activate Groups * In the Mezmo App, go to **Push Groups**. * Under **Push Status**, choose **Activate group push.** Provisioning in Mezmo should start shortly. You can find errors and status under **Reports, System Log.** For additional information please consult [Okta Documentation](https://help.okta.com/en-us/content/topics/apps/apps_app_integration_wizard_scim.htm). ## Microsoft Azure Entra ID (formerly Active Directory) In Azure Portal, go to your **Entra ID** instance. ### Set Up SCIM as a New Enterprise Application 1. Click **Enterprise applications** and then **New application**. 2. Click **Create your own application**. 3. Enter **Mezmo SCIM**, then click **Create**. 4. Once this application is created, click **Provisioning**. 5. Click **Get Started**. 6. Click **Connect Your Application** under **Create Configuration** section 7. Under section **Admin Credentials** enter: * For **Tenant URL**, enter: `https://api.mezmo.com/v1/enterprise/scim?aadOptscim062020` * For **Secret token** enter your enterprise token. *Please ensure you enter this token every time you edit this area* * Click **Test Connection** and then **Create**. ### Configure Group Mappings 1. Go to **Provisioning > Attribute Mapping (Preview)**, and click **Provision Microsoft Entra ID Groups.** 2. Make sure there is **Matching precedence** set to **1** for **displayName** attribute to allow Entra matches its groups with Mezmo groups (Child orgs) 3. Under **Target Object Actions,** clear the **Create** and **Delete** options and click **Save.** In order to make groups provisioning work properly, Microsoft Entra group names must match Mezmo account (Child orgs) names. For example, if your Mezmo child org name is "Department", then your Microsoft Entra group name should be "Department". Any users assigned to this group on the Entra side will be synced with the corresponding account (Child org) on the Mezmo side. ### Configure User Mappings 1. Go to **Provisioning > Attribute Mapping (Preview)**, and click **Provision Microsoft Entra ID Users.** 2. Under section **Attribute Mappings** remove all attributes except: * userName * active * displayName * emails\[type eq "work"].value * name.givenName * name.familyName * name.formatted 3. Edit attribute **userName** by replacing **Source attribute** from **userPrincipalName** to **mail**. Mezmo assumes **userName** is a unique user ID which must be the user's email address. 4. Click **Save.** For users to sync successfully, please ensure they have the following fields filled out: * Display Name * First Name * Last Name * User Principal name * Email Otherwise they will be skipped during the sync ### Choose Groups to Sync 1. Under **Mezmo SCIM**, click **Users and groups**. 2. Click **Add user/group.** 3. Choose the group(s) you want to sync and click **select.** 4. Click **Assign** ### Start Provisioning 1. Under **Mezmo SCIM,** click **Provisioning** then again **Provisioning**. 2. Set **Provisioning Status** to **On.** 3. After this setting SCIM provisioning should start automatically with a default 40 minutes interval. Provisioning in Mezmo should start shortly. You can find errors and status on the **Mezmo SCIM Provisioning Overview** page and **Monitor** section in **Mezmo SCIM -> Provisioning**. For additional information check out the [Azure Documentation](https://learn.microsoft.com/en-us/entra/architecture/sync-scim). # Terraform Provider for Mezmo Source: https://docs.mezmo.com/docs/terraform-provider Manage Mezmo Views and Alerts as code with the Terraform provider, including setup, use cases, and an example HCL configuration. ## About the Terraform Provider for Mezmo [Mezmo](https://mezmo.com) is a centralized log management platform. The Terraform provider from Mezmo allows organizations to manage Views and Alerts programmatically via Terraform commands. With the Terraform provider, which utilizes our [Configuration AP](/docs/api-referencesk9h#tag-configuration)[I](/docs/api-referencesk9h#tag-configuration), automate deployments of Views and Alerts to your Mezmo environment. Read the full documentation on the official [Terraform site](https://registry.terraform.io/providers/logdna/logdna/latest). If you are interested in contributing to this project, the source is in [GitHub](https://github.com/logdna/terraform-provider-logdna). Your service key can be generated or retrieved from the Mezmo web application. Navigate to **Settings > Organization > API Keys**. ## Scenarios for Using the Terraform Provider One use case for the Terraform provider is if you need to quickly spin up multiple Alerts on Views in several Kubernetes clusters and you need them to all be exactly the same. You can use Terraform to create and deploy the Alerts. Automation like this helps reduce the risk of error and speeds the process. Other use cases include: * As a developer, I want to replicate a set of Views/Alerts with minor tweaks from a base template in the same or different account. * I want to manage my infrastructure and infrastructure SaaS apps (ex. PagerDuty, Sysdig, etc.) including Mezmo from the same toolset. * As a SRE manager, I want all changes to a critical set of views/alerts to be code reviewed so that my workflow and alerting doesn’t get disrupted by accident, and mistakes can be quickly undone. ## Example This example code for creating a new View and Alert is in [HCL (Hashicorp Configuration Language)](https://www.terraform.io/docs/configuration/syntax.html). ```none theme={null} terraform { required_providers { logdna = { source = "logdna/logdna" version = "1.0.0" } } } # Configure the LogDNA Provider provider "logdna" { servicekey = "" } resource "logdna_view" "my_view" { apps = ["app1", "app2"] categories = ["Demo1", "Demo2"] hosts = ["host1", "host2"] levels = ["fatal", "critical"] name = "Email PagerDuty and Webhook View-specific Alerts" query = "test" tags = ["tag1", "tag2"] email_channel { emails = ["test@mezmo.com"] immediate = "false" operator = "absence" terminal = "true" timezone = "Pacific/Samoa" triggerinterval = "15m" triggerlimit = 15 } pagerduty_channel { immediate = "false" key = "" terminal = "true" triggerinterval = "15m" triggerlimit = 15 } webhook_channel { bodytemplate = jsonencode({ hello = "test1" test = "test2" }) headers = { hello = "test3" test = "test2" } immediate = "false" method = "post" terminal = "true" triggerinterval = "15m" triggerlimit = 15 url = "https://yourwebhook/endpoint" } } ``` Note that if you create a new View using Terraform, but then delete the View by using the Mezmo Dashboard UI and *not* Terraform, then Terraform will not be aware of the deletion and displays an `Error: Resource Not Found` message. For more information about handling "deletion drift" refer to our full documentation on the official \Terraform site\. Be aware that running `terraform plan` and/or `terraform apply` may not display the full delta of possible changes, if TF is not aware of all existing Mezmo resources. # The Enterprise Dashboard Source: https://docs.mezmo.com/docs/the-usage-dashboard Review your organization's data usage with retained, restored, and ingested vs. retained graphs plus top app, source, and tag trends. The Enterprise Dashboard provides an overview of how much data your organization is ingesting. It shows: * Retention of log lines per day in a stacked graph with categories for retention periods * Restored data per day * Ingested data v. retained data per day * Trends in usage for the top 50 apps, sources, and tags ## Retained Data The **Retained Data** graph displays the volume and count of retained log lines per day for the selected time period. For each day, the graph also displays the volume and count of log lines retained for each type of retention period. Image ## Restored Data The Restored Data graph displays the volume of data restored per day for the selected time period ## Ingested v. Retained Data The Ingested v. Retained Data graph displays the volume of ingested data v. retained data per day for the selected time period. Image ## Stacked and Unstacked Toggling Stacked to on, will stack the data lines on top of each other and display the sum of all visible sources. The graph will also be updated to show how each app, source, or tag compares to the total. ## Trends The Trends graph displays the volume of data for the selected data type for the top Apps, Sources, and Tags. In the Stacked view, the graph will display the data lines on top of each other and display the sum of all visible sources. The graph will also be updated to show how each app, source, or tag compares to the total. Image # Create Usage Alerts Source: https://docs.mezmo.com/docs/usage-alerts Set a usage limit and notify organization members when data usage exceeds your threshold before the end of the usage cycle. You can set usage alerts to notify you and other members of your organization when your usage exceeds a set threshold before the end of your usage cycle. If you have a Professional or Enterprise account, the usage cycle is the same as your billing cycle. For Enterprise accounts, you can check when your usage cycle ends and begins in the Enterprise Dashboard. Each chart includes a marker that indicates the end and beginning of the cycle, as shown in this example. Image ## Create a Usage Alert 1. In the Mezmo Web App, navigate to **Usage > Dashboard**. 2. Scroll to the Usage Alert section. 3. Set the **Usage Limit**. When your usage reaches this volume of data, in GB, it will trigger an alert. 4. Select the recipients to receive the alert. The list of available recipients is based on the members of your organization. # Use the Mezmo Assistant Source: https://docs.mezmo.com/docs/use-ai-assistant Analyze logs and telemetry in natural language with the Mezmo Assistant using prompts, time expressions, context mentions, and slash commands. The Mezmo Assistant helps you analyze logs, investigate issues, and work with your telemetry data using natural language. Ask questions and get insights without writing complex queries. It's powered by [AURA](/aura), Mezmo's open-source harness for production-ready agents. ## Access the Mezmo Assistant You can access the Mezmo Assistant from two locations: * **From the homepage**: Click the Mezmo Assistant icon or prompt area to start a conversation * **From the AI Agent page**: Navigate to the dedicated AI Agent interface for an expanded view with chat history The assistant is available throughout your Mezmo workspace and provides context-aware help based on where you access it. ### Open the AI Assistant with a Prompt Link You can open the homepage AI Assistant and automatically send a prompt by adding a `q` query parameter to the homepage URL. This is the same AI Assistant you open under From the homepage above. Use the following URL form, where the prompt value must be URL-encoded: `https:////homepage?q=` For example, the following URL encodes the prompt "I have a question about this page": `https:////homepage?q=I%20have%20a%20question%20about%20this%20page` On load, the AI Assistant places a non-empty `q` prompt in its input and sends it automatically. The prompt is submitted for you, not just pre-filled. An empty or whitespace-only `q` value is ignored, and nothing is sent. After the page loads, the `q` parameter is removed from the address bar, so refreshing the page does not resend the prompt. The URL shown after landing is no longer a shareable prompt link. To share a prompt link, use the original URL you constructed, not the one shown after the page loads. ## Ask Questions Type your questions or requests in plain language. The Mezmo Assistant can help with: * Analyzing log patterns and trends * Performing root cause analysis on issues * Exploring your log data * Getting information about your pipelines * Searching the Mezmo knowledge base **Example prompts:** * "Show me error logs from the last hour" * "Analyze logs for root causes of the recent spike" * "What fields are available in my logs?" * "Count logs matching 'authentication failed'" ## Using Time Expressions You can use natural language time expressions when querying log data with the Mezmo Assistant. ### Time Formats You can specify time ranges using any of these patterns: * "last 30 seconds" or "30 seconds ago" or "30 seconds" * "last 5 minutes" or "5 minutes ago" or "5 minutes" * "last 1 hour" or "1 hour ago" or "1 hour" * "last 2 days" or "2 days ago" or "2 days" * "last 3 weeks" or "3 weeks ago" or "3 weeks" ### Time Units Supported time units (singular or plural): * second or seconds * minute or minutes * hour or hours * day or days * week or weeks ### Example Queries * "Show me error logs from the last hour" * "Analyze logs from 30 minutes ago" * "Count authentication failures in the last 5 minutes" * "Deduplicate logs from 2 days" * "What happened in the last 24 hours?" ## Add Context with @ Mentions Use the `@` symbol to attach context to your prompts for more relevant responses. **To add context:** 1. Type `@` in the prompt input. 2. Select from available context types: pipelines, pipeline nodes (when viewing a pipeline), tap data (when available), or other context relevant to your current view. 3. Continue typing your prompt. You can attach multiple pieces of context to a single prompt. The Mezmo Assistant will show suggested prompts based on your attached context. ## Use Slash Commands Slash commands give you quick access to common prompts and actions. 1. Type `/` in the prompt input. 2. Select a category from the menu, such as **Pipeline prompts** or **Log Analysis prompts**. 3. Select a prompt to add it to your message. The default prompts change based on where you open the Mezmo Assistant: * **Pipeline pages**: Pipeline-related prompts * **Log Analysis or Live Tail pages**: Log Analysis prompts When you attach context with `@` mentions, the available prompts update to match. ## View Tool Call Progress When the Mezmo Assistant performs operations like analyzing logs or querying data, you'll see progress indicators showing: * Which tool is currently running * Progress updates as the operation proceeds * Results when the operation completes You can cancel long-running requests by clicking the stop button. ## View Completed Tool Calls After the Mezmo Assistant responds, you can review the tools used during the conversation. A collapsible section below the response shows how many tools were called. ### Tool Summary Click **"X tool(s) used"** to expand or collapse the list. Use **Expand all** and **Collapse all** to control all tool details at once. ### Tool Details Each tool displays: * **Status**: Green checkmark for success, red X for error * **Tool name**: What the tool does * **Duration**: How long the tool took to run Click any tool to expand its details: * **Tool**: The specific tool that was called * **Arguments**: Parameters passed to the tool * **Result**: Output returned (successful calls) * **Error**: Error message (failed calls) ## What the Mezmo Assistant Can Do The Mezmo Assistant uses specialized tools to help you work with your data: **Log Analysis and Root Cause Analysis** * Analyze logs to identify root causes using time ranges or relative time expressions (for example, "last hour") * Investigate patterns and anomalies in your log data **Cross-Source Timeline Correlation** * Correlate logs across multiple sources (apps, hosts, or custom groupings) to find root causes * View per-source timelines in chronological order * See a combined timeline with relative timestamps to compare events across sources * Use template-based deduplication to reduce noise while keeping unique log patterns **Log Exploration and Metrics** * Count logs matching specific queries * Get context around specific log lines * List available fields in your log data * Query log histograms to understand volume over time * Group logs by specific fields **Log Deduplication and Pattern Discovery** * Deduplicate logs using time ranges or relative time expressions * Find patterns and recurring log messages **Pipeline Operations** * List your available pipelines * Get details about specific pipelines * Understand pipeline configurations **Trace Analysis** If your account has trace analysis enabled, the Mezmo Assistant can help you analyze OpenTelemetry trace data: * View service graphs showing how your services communicate * List failed traces for a time range, optionally filtered by service * Get span details for a trace to understand request flow and timing * Find latency anomalies grouped by service, edge, or operation * Sort anomalies by count, score, or latency multiplier to prioritize investigation * Compare latency between two time periods to detect degraded service edges * Get bucketed latency summaries for service-to-service calls with p50, p95, and p99 metrics * View service failure rates as a histogram, optionally grouped by service, edge, or operation * Fetch individual service failure records with optional filtering by service, edge, or operation **Knowledge Base and General Help** * Search the Mezmo knowledge base for product information * Get help with Mezmo features and best practices **Time and Utility Functions** * Get the current time for time-based queries ## Chat History Your conversations with the Mezmo Assistant are automatically saved. The homepage sidebar shows your recent chat history, so you can review previous conversations, continue earlier discussions, and access past analysis and insights. ## Responses and Formatting The Mezmo Assistant provides responses in formatted markdown, including: * **Code blocks** with syntax highlighting for technical content * **Tables** for structured data * **Lists** for organized information * **Mermaid diagrams** for visualizing workflows and relationships * **Links** to relevant documentation ## Tips for Effective Use * **Be specific**: Include relevant details like time ranges, field names, or error messages * **Use context**: Attach pipelines or other context with `@` mentions for more accurate responses * **Ask follow-up questions**: The Mezmo Assistant remembers your conversation context * **Try slash commands**: Use `/` to discover common prompts and actions ## Cancel Requests If a request is taking too long or you want to stop the current operation, click the **Stop** button while the Mezmo Assistant is processing. The partial response will remain visible, and a "Cancelled" message will indicate the request was stopped. You can then start a new prompt or refine your question. # Use Screens and Widgets to Monitor Log Data Source: https://docs.mezmo.com/docs/use-screens-and-widgets-to-monitor-log-data Learn how Screens and the four widget types (Count, Gauge, Table, Time Shifted Graph) present log metrics for monitoring and analysis. ## Screens Screens display Widgets that display metrics about logs over a period. These are useful are to analyze and present log data. In topics in this section you'll learn how to create Screens for specific use cases, while Mezmo also provides pre-defined [Templates for Views, Boards, and Screens](/docs/using-templates) for a variety of log data types. 1. [Web Analytics Screen](/docs/web-analytics-screen) 2. [Server Analytics Screen](/docs/server-analytics-screen) 3. [Server Security Screen](/docs/server-security-screen) To access Screens and Widgets in the Mezmo Web App, go to [Screen](https://app.logdna.com/screens/new-screen)s. ## Widgets Widgets are added to Screens to display log metrics. There are four different widget types: * **Count** - Displays a single aggregate number. * **Gauge** - Displays a count out of a minimum and maximum value using a gauge indicator. * **Table** - Displays a horizontal bar chart with logs grouped by a selectable field. * **Time Shifted Graph** - Displays a layered line graph comparing values over two different periods. For example, log volume today vs. log volume yesterday. To get even more ideas on how to use widgets, check out [Widget Use Cases](/docs/widget-use-cases). ## Feature Notes * Screens can only accommodate a certain number of widgets. Click and drag widgets to adjust size or create a new Screen. ## Related Topics # Templates for Views, Boards, and Screens Source: https://docs.mezmo.com/docs/using-templates Browse, import, edit, and remove Mezmo's prebuilt templates to quickly add Views, Boards, and Screens for common queries. Mezmo provides templates for commonly used queries, so that you can quickly add Views, Boards and Screens for those specific queries. Browse the [full library](https://app.logdna.com/manage/template-library) of Mezmo Templates , select which templates you want, and then click on **Get Started** to download and import the template. After importing the templates you want, they will automatically appear in your Mezmo web application, under your Views, Boards, and Screens. You can at any time go back to the Template Library and click **Edit** on any template that you have already downloaded in order to modify which Views, Boards, and Screens you want to install, or remove them. Note that if you have already modified any of the Views, Boards, Screens, parsing templates, or exclusion rules that were installed from a template, you will need to remove each of the modified items manually. ## Related Topics * # Set Variable Retention Source: https://docs.mezmo.com/docs/variable-retention Create rules that apply custom retention periods to specific log lines and track their impact in the Usage Dashboard and email digests. This feature is only available for customers on an Enterprise plan, and is restricted to **Owner** and **Admin** user roles. Please contact your Customer Support Manager or [support@mezmo.com](mailto:support@mezmo.com) for more information. ## About Variable Retention With **Variable Retention**, you can use Mezmo's search functionality to identify specific log lines, and then set custom retention policies for those lines so they are retained only for the period of time in which they are relevant. For example, if you are currently on a plan with a 30-day retention period, but have a subset of log lines that only need to be stored for three days, you can use Variable Retention to remove those lines from storage after that three day period. These log lines could originate from ephemeral environment logs, debug-level logs, or CI/CD logs where detailed results aren't needed after a few days. Variable Retention also includes a **Usage Dashboard,** where you can view the total usage for your account, along with breakdowns for each Variable Retention tier. This can help you understand how your Variable Retention rules are affecting your overall usage and costs. ## Using Variable Retention Rules with Other Mezmo Features * Variable Retention rules do not allow you to store logs for longer than the plan retention period associated with your account. For example, if your account has a Maple plan retention period of 14 days, but you set a 30 day variable retention rule, logs will not be searchable beyond 14 days. * Variable Retention rules only affect new log lines received by Mezmo, and do not affect any log lines retained before a rule has been created or changed * **Index Rate Alerts**, **Usage Quotas**, and **Usage Alerts** are triggered by the total of all GBs/log lines ingested into Mezmo, regardless of retention tier * **Exclusion Rules** apply before any Variable Retention rules. If you want to apply a Variable Retention rule to an excluded log line, be sure to [make an exception in the corresponding Exclusion Rule](/docs/exclusion-rules) to not exclude it. * New Variable Retention rules may require up to five minutes to take effect, as will changes to existing rules ## Create and Manage a Variable Retention Rule You can create a Variable Retention Rule by specifying the retention period for the log lines, and then the log line search criteria. ### Create a Variable Retention Rule 1. In the Mezmo user interface, go to **Settings** > **Usage** > **Variable Retention**. 2. Click **Add Rule**. 3. Enter a **Name** for the Variable Retention rule. 4. For **Retention Period**, select the number of days that logs matching this rule should be preserved. 5. Provide the **Host**, **App**, **Level**, or **Query** to use for the log line search. You can use any of these criteria alone or in combination with the other criteria, and if you set multiple criteria, they are joined by an AND operator. The topic [Search and Filter Log Data](/docs/search-and-filter) contains more information about using the Mezmo search syntax for creating queries. 6. Click **Save**. ### Enable or Disable a Variable Retention Rule You can disable or enable a Variable Retention rule by using the on/off switch on the right side of the rule. ### Edit a Variable Retention Rule Click the three dot icon next to the rule on/off switch and click **Edit** to make changes to the rule **Name**, **Retention Period**, or search criteria. ### Delete a Variable Retention Rule Click the three dot icon next to the rule on/off switch and click **Delete** to remove the rule. ### Manage Variable Retention Rule Priority Variable Retention rule priorities determine which rule should be applied to a log line if it matches multiple retention rules. Rules with lower priority numbers take precedence over rules with higher priority numbers. For example, if a log line matches a rule with a priority of 5, and a rule with a priority of 20, only the rule with the priority of 5 will be applied. There are two ways you can set rule priorities on the **Variable Retention Rules** page. * Click the six dot icon next to the priority number of the rule and drag it to a new priority rank * Click the three dot icon next to the rule on/off switch to edit the rule, and enter a new priority number in the **Change Priority** field ## View Variable Retention Usage ### Viewing Variable Retention Usage in the Log Viewer Once you have enabled Variable Retention rules and they take effect, log lines affected by a Variable Retention rule will have a badge added to the top right of the log line to show its retention period, as shown in this screenshot. ### Viewing Variable Retention Usage in the Usage Dashboard Additionally, once you have enabled Variable Retention rules and they become effective, the [Usage Dashboard](/docs/the-usage-dashboard) will be updated to show breakdowns based on Variable Retention tiers, as illustrated in this bar chart. Here the log lines that are retained according to the plan retention period of 7 days are shown in blue, while the log lines that are being stored following a Variable Retention rule are shown in green. The daily usage chart will be updated to show the number of GBs or lines stored in each retention tier. * Hover your mouse cursor over each bar in the chart to see the exact number of GBs or lines used for a specific retention tier on a specific day. * To view the breakdown of data usage by retention tier, hover your mouse cursor over **See Breakdown by Retention** under the **Data Usage** number on the top of the Usage Dashboard. * To view the breakdown of Variable Retention usage by **Apps**, **Sources**, and **Tags**, use the **Last month in Days - Trends** chart and select the retention tier you want to view. This way you can identify which apps, sources, or tagged logs are utilizing different retention tiers. ### Viewing Variable Retention Usage in Email Digests Weekly and monthly usage email digests include a breakdown of usage by retention tier when Variable Retention rules are active. Each tier shows the top apps, hosts, and tags that contributed to that tier's usage. See [Manage Usage](/docs/manage-usage) for more information about email digests. If you set a Variable Retention rule to have the same retention period as your plan policy, you will have one breakdown showing your plan retention, and another showing your variable retention tier. For example, if you're on a 14 day plan and you set a Variable Retention rule with a 14 day retention period, you'll see one breakdown for **14 day plan retention**, and another for "14 day variable retention". # VictorOps Alert Integration Source: https://docs.mezmo.com/docs/victorops-alert-integration Trigger VictorOps incidents from Mezmo log alerts by configuring the URL to notify and routing key for the integration. The VictorOps alert integration enables users to easily trigger incidents in VictorOps based off of log data in Mezmo. ## Integrating with VictorOps 1. On the alerts creation page, select the VictorOps logo. 2. Define the alert properties. To retrieve the URL to notify go to Integrations -> REST -> URL to notify and input the url up to the final / into the URL to notify section of the Mezmo VictorOps alert integration. The URL to notify has a few requirements: * The URL can't point to a private IP address, such as `192.168.x.x`, `10.x.x.x`, or `172.16.x.x` through `172.31.x.x` * The hostname can't resolve to a private IP address via DNS * The URL can't return a redirect response because Mezmo won't follow redirects If your URL doesn't meet these requirements, the request will fail. For the routing key, go to Settings -> Routing Key. More information on routing keys can be found [here](https://help.victorops.com/knowledge-base/routing-keys/#:~:text=%3E%3E%20Routing%20Keys.-,Creating%20Routing%20Keys%20in%20VictorOps,Escalation%20Policy%20for%20a%20team). After clicking test alert from the Mezmo alert UI, you will now see incidents in VictorOps! **Note:** When creating a new Routing Key, changes may take up to 10 minutes to propagate on the VictorOps side. If you don't see an alert show up in the VictorOps dashboard, please try again in 10 minutes. # View Log Data Source: https://docs.mezmo.com/docs/view-log-data Explore the Log Viewer to see your logs, apply Views, filter by tags, sources, apps, and levels, search, and use the timeline. The Log Viewer is where you can view all your logs, filter data, and create custom views. When opening the [Log Viewer](https://app.logdna.com/logs/view) for the first time, you'll see all your logs for any apps or hosts added. Image ## Views Any Views you create will appear here. Use Views to narrow down the data you want to see. Learn more in [Create and Edit Views](/docs/create-and-edit-views). Image ## Filter Toolbar Further narrow down logs by filtering on Tags, Sources, Apps, and Levels. Use the [Mezmo Logging Agent](/docs/introducing-the-agent) to add data. Image ## Search and Custom Templates Check out [Search and Filter Log Data](/docs/search-and-filter) for more information about using the search field, and [Parse Logs with Custom Templates](/docs/parse-logs-with-custom-templates) to learn how to create search templates based on your parameters. Image ## Timeline Use the timeline to find trends. Learn how in [View Log Data by Timeline](/docs/view-log-data-by-timeline). Image ## Related Topics # View Log Data by Timeline Source: https://docs.mezmo.com/docs/view-log-data-by-timeline Use the Mezmo Timeline to spot trends, change time scales, scope into time frames, and jump to specific points in your logs. The Mezmo Timeline is a tool for seeing patterns and trends in log lines over time. By default, it shows the most recent 60 minutes of log lines. Use the timeline to: * View the counts of logs that match a specific query to understand how often the same action is occurring in a timeframe. * Debug issues that occurred during a specific timeframe to do root cause analysis. * Determine exactly when a certain error first started occurring. Access the timeline by selecting [**Views**](https://app.logdna.com/logs/view). ## Change the Time Scale To change the time scale, click the downward arrow beside the Time scale, then select a predefined time scale. The time scale will update to show the time you selected. Image ## Scope In To zoom in or **Scope** to a block of time, place your cursor in the timeline, click to select, and then drag up or down the timeline to select a time frame. The Log Viewer and time scale update to show the time you selected. You can continue to scope until you get to the time you want to view. Each bar represents 15 minutes. You can not use the Time scale selector to zoom in further than your current time frame selection. ### Compare Time Scale Once you have scoped in on the time scale, you can investigate and compare times before and after. For example, the original time you zoomed in on was 10 minutes. Then you selected 30 minutes in the time scale drop down. You will see log lines for: * 10 minutes before the original selection * The original 10 minutes * 10 minutes after the original selection Changing the time scale will not change the log viewer. You must zoom in. ## Select a Time Select a time by clicking one of the lines in the timeline and clicking **Jump to**. The log viewer will update to show the selected time. Before selecting **Jump to**, it will show which time has been selected. Image ## Reset the Time Scale Click the circle with the arrow to reset the time scale. ## Jump to Timeframe To jump to a point in time in your logs, type the desired day and time in the **Jump to timeframe** field. You can enter queries such as yesterday at a11pm. ```bash theme={null} today at 11am ``` To set a timeframe for your logs, type two different dates and times separated by `to`. ```bash theme={null} last fri 4:30p to 11/12 1 AM ``` This will show all logs between last Friday at 4:30 PM and November 12 at 1 AM. ## Feature Notes * When the Log Viewer is in live tail mode, the timeline graph is a few minutes behind the Log Viewer display. To turn if off, toggle the **Live** button in the Log Viewer. # Visualize Log Data with Graphs Source: https://docs.mezmo.com/docs/visualize-log-data-with-graphs Use Boards and Graphs to chart log metrics over time with operations, plots, filters, and breakdowns like histograms and pie charts. Boards are collections of Graphs that enable you to visualize log data over time. Topics in this section will introduce you to basic concepts for Boards and Graphs, and show you how to [Create a Graph](/docs/create-a-graph) of HTTP response codes, [add Plots to compare data](/docs/add-a-plot-to-a-graph), and [add a Breakdown](/docs/add-a-breakdown-to-a-graph) to show which apps are sending the response codes. Boards **Boards** are a collection of Graphs. Both graphs and boards can be filtered and queried. Changes to the filters or date and time on a board, also changes the graphs. ### Filtering You can filter your board by entering an query into **Filter this board**. Filter the board using [search](/docs/search-and-filter). ### Date and Time Set the date and time on a board by selecting the calendar. You can also set a duration by selecting from preset duration time. You can also add live data to your board, by toggling **Live**. ## Graphs Graphs are used to aggregate metrics about your log lines over time. Graphs are displayed as a line graph. ### Graph Functions * **Operation** - Choose a metric used to plot the data * **Filter** - Add a query filter to a single graph. * **Stack Lines** - Place multiple lines together. on the same graph as a percentage of all other line values. * **Plots** - Add additional data points to your graph. All the plots are listed by the line color used to represent that plot, under the corresponding graph. Learn how to in [Add a Plot to a Graph](/docs/add-a-plot-to-a-graph). ### Operation A metric is a function that changes a graphed dataset. They are also known as operations. When graphing data, the metric default is Counts. You can change the metric on your graphs. Only number fields can use all metrics. Metrics only work with data that includes the selected plots and matches the dataset filter. Current available metrics or operations: * **Counts** - Counts the number of lines in each interval. This is the only metric function that can be applied to string fields. * **Min** - Returns the lowest value for each interval. * **Max** - Returns the highest value for each interval. * **Average** - Returns the average value for each interval * **Cumulative** - Returns the sum of values for each interval * **Diff** - Returns the largest difference between values for each interval * **Percentiles** - Returns the Nth percentile of a set of values for each interval. You can choose from the 75th, 85th, 95th, and 99th percentile presets. ### Breakdown Breakdowns can be added to graphs. They are represented as histograms and pie graphs. Learn how to [Add a Breakdown to a Graph](/docs/add-a-breakdown-to-a-graph). ### Histograms Can show data from available fields. They show the distribution of the plots against a chosen field. For example, you have added plots for 200 and 500 response codes. Create a histogram based on the request field to see which URL is returning certain responses. ### Pie A pie breakdown can show data from a plotted data or added fields. ## Feature Notes * For STRING fields, you can only graph count. * For NUMBER fields, you can choose field values and operations. * When creating a Graph, they use Live data. Toggle live to turn on and off. * Change the [Board](/docs/using-templates) date to change the graph date. * Choose **All Lines** to graph or plot all lines in your application. ## Related Topics # Web Analytics Screen Source: https://docs.mezmo.com/docs/web-analytics-screen Build a Screen with widgets to track week-over-week traffic and your most popular pages from web server logs. You'll create a screen that monitors: * Week over week traffic * Most popular pages ## Week Over Week Traffic 1. Create a **Time Shifted Graph** Widget. 2. Under Data: 1. Set Field to **All Lines**. 2. Set Advanced Search to: `response:>=200 response:<300 request:*`. 3. Leave Operation as Counts. 3. Set Duration to **This Week vs Last Week(Mon-Sun)**. 4. Under Appearance set the label to **Week Over Week Traffic Trends**. ## Most Popular Pages 1. Create a **Table Widget**. 2. Under Data: 1. Group by Request 2. Field to All Lines 3. Advanced Filter to `response:>=200 response:<300 request:*` 3. Leave the Operation as **Counts**. 4. Set the Duration to **Past 1 Week**. 5. Under Data Format: 1. Set the number of rows to 10. 2. Sort Descending. 6. Under Appearance: 1. Label the table: Most Popular Web Pages 2. Left Column Label: URL Path 3. Right Column Label: Count 7. Uncheck Display as Bar Chart. Save your Screen. Give it the name **Web Analytics** and add the category, **Web App Monitoring**. # Web Server Template Source: https://docs.mezmo.com/docs/web-server-template Get pre-configured Views, Boards, and Screens for HTTP web servers to monitor response codes, errors, traffic, and security. The Mezmo Web Server Template enables you to immediately receive valuable data from HTTP web servers via pre-configured Views, Boards, and Screens. Browse the[ full library of Mezmo Templates](https://app.Mezmo.com/manage/template-library). ## Views Views are saved shortcuts to a specific set of filters and search queries. Using views, the Web Server template can track server errors and alert you when they occur. To set up alerting, we recommend attaching presence alerts to the "HTTP Server Error" view and absence alerts to the "HTTP 200" view to be alerted on excessive errors, or insufficient successful traffic. Check out the topic [Add Alerts to Views](/docs/add-alerts-to-views) for more information. * HTTP Server Errors (Recommended to Presence Alert on) - includes 500's and all NGINX or Apache error logs, if those error logs are using the default error log format * HTTP Success (200) (Recommended to Absence Alert on) * HTTP 404 Errors * HTTP Forbidden/Unauthorized (401, 403) ## Boards Boards are collections of graphs. Using boards, you can track trends with response codes and understand how they fluctuate over time at a glance. Drill down using subplots to see which host or path is generating the most errors. Check out the topic [Visualize Log Data with Graphs](/docs/visualize-log-data-with-graphs) for more information. * HTTP Response Codes and Errors * Breakdown by App, Host, Request (Path), Client IP, and Referrer * Traffic Volume with Total Bytes and Requests ## Screens Screens are collections of customized dashboards that can display data in various forms. Using screens, you can see top 401 and 403 requests by IP address in the last hour, day, and week. See the topics [Use Screens and Widgets to Monitor Log Data](/docs/use-screens-and-widgets-to-monitor-log-data) and [Web Analytics Screen](/docs/web-analytics-screen) for more information. * Web Analytics, such as traffic trends, most popular pages, and referrers * Server Health, such as count of 200 and 500's on time shifted graphs * Web Server Security, such as top 401 and 403 errors by IP Address # Webhook Alert Integration Source: https://docs.mezmo.com/docs/webhook-alert-integration Send Mezmo alerts to third-party services with a custom webhook, configuring method, URL, headers, body, and tokens, using Jira as an example. The Webhook Alert integration enables customers to easily integrate Mezmo alerts with third-party services. You can configure the webhook's method, headers, and body. Using an extensive set of tokens, you can customize exactly what content you want to extract from your logs and then include it in your third-party service’s notifications. For example, if you want to automatically trigger the creation of an Issue in your Jira ticketing system, and include specific information in the Description field, you can do so via webhooks. Read further to see detailed information about webhooks, based on this example of creating a Jira ticket. ## Create an Alert Using the Webhook Integration As an example throughout this document, we’ll configure our webhook alert to create a new issue in Jira after seeing more than 10 matches for a line with a specific error within 30 seconds. The first step is to [attach an alert to an existing view](/docs/create-and-edit-views#attaching-an-alert) that is based on a query for the specific error for which you want to create a Jira ticket. (Alternatively, if you predict that you will want to use webhook integrations for multiple views, you can [create a preset alert](/docs/add-alerts-to-views#configure-a-preset-alert-template). In the image below, we attach an alert to a View named “Errors.” On the Alert modal, click the Webhook option. ## Defining the Alert First, specify the basic settings for the alert. You can configure the number of matching log lines, the period to observe, and the timing of when the alert is sent. Optionally, you can create a custom schedule for the alert. ## Configuring Your Custom Webhook Integration In the lower area of the Alert modal, define the specifics of your webhook integration, including the method, the URL for the third-party service, headers for authorization, and finally the body. We will build out the body using our tokens, which enable us to define exactly what log content we want to be written to the Jira ticket. ### Method & URL First, customize the webhook by selecting the HTTP method and adding the webhook URL. In our example, we select the POST HTTP method and add our webhook URL, https\://\.atlassian.net/rest/api/2/issue/. The exact URL and HTTP method will depend on the app for which you add the webhook. Webhook URLs have a few requirements: * The URL can't point to a private IP address, such as `192.168.x.x`, `10.x.x.x`, or `172.16.x.x` through `172.31.x.x` * The hostname can't resolve to a private IP address via DNS * The URL can't return a redirect response because Mezmo won't follow redirects If your webhook URL doesn't meet these requirements, the request will fail. ### Headers Next, set up the headers for the authentication method. The headers depend on the app you are triggering with the webhook. For our example with Jira, we use [API token authentication](https://confluence.atlassian.com/cloud/api-tokens-938839638.html), which requires us to add an Authorization header with the API token from Atlassian in it. ### Body By default, the webhook body is pre-populated with alert information. Customize the body to conform to the third-party service’s API; in this case, we customize the webhook body using one or more tokens and structure the body according to the [Jira REST API](https://developer.atlassian.com/server/jira/platform/jira-rest-api-examples/) for creating issues. So, first we configure the body to create a new issue with the view name in the Summary field of the Issue, and more alert information in the issue description field. We can validate the body to make sure it has valid JSON using the **Validate JSON** button right below the body editor. This validation catches errors and typos without actually executing the webhook. ### Body Tokens The following tokens allow you to further customize the webhook body with specific information about the view or the matched lines that triggered the alert. * \{\{ name }} View name * \{\{ matches }} Number of matched lines * \{\{ lines }} Raw output of matched lines * \{\{ level }} Severity level (info, warn, error, etc) in the first line of the log * \{\{ url }} The full URL of the View, with first matched line (presence alert only) * \{\{ query }} The query of the View to which this alert is attached * \{\{ app }} Single application included in the first line of the log * \{\{ host }} Single host included in the first line of the log * \{\{ tag }} Single tag included in the first line of the log * \{\{ line }} The exact text of the first line of the log * \{\{ line\_objects }} Array of matched line objects * \{\{ first\_line\_object }} First matched line object (the entire line) - When using the `line_objects` and `first_line_object` tokens, they must be used in isolation because they will insert an array or object into the webhook body, e.g., "lineObjectsArray": "\{\{ line\_objects }}" will turn into lineObjectsArray: \[\{ ... }]. - Access array indexes and object properties using bracket or dot notation. For example you can use “\{\{ line\_objects\[0] }}” or “\{\{ first\_line\_object.\_line }}”. - Be aware that using some of the tokens, such as the `line_objects`, might result in a large number of returned log lines. ### Testing Test the webhook with the Test link at the top of the pane. The Test link will trigger the webhook with test data. You can also use the Test link to try out what the alert would look like in a real example by filling in the Webhook JSON template. We use it in our example to attempt to create the issue in our Jira project. After the test runs, we can check that the issue has been appropriately created in our Jira project: ## Further Examples ### Microsoft Teams This webhook will send a message to your [Microsoft Team chat](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook). Note that you should add the following header to the alert: `Content-Type: application/json`. ```json theme={null} { "name": "Mezmo Alarm Topic", "sentFrom": "{{ name }}", "summary": "{{ matches }} line(s) matched in {{ name }}. Security level: {{ level }}", "text": "[Goto]({{ url }}) Mezmo Alert on {{ name }} with {{ matches }} line(s) matched for query {{ query }}\n\nFirst matched log\nApp: {{ appp }}\nHost: {{ host }}\nLevel: {{ level }}\nTag: {{ tag }}\nLine: {{ line }}\n\nLines\n{{ lines }}" } ``` ### Google Chat This webhook will send a message to a given [Google Chat Space](https://developers.google.com/chat/how-tos/webhooks). Note you that you need to add the following header to the alert's webhook: `Content-Type: application/json; charset=UTF-8`. ```json theme={null} { "cards": [ { "header": { "title": "Mezmo Alert: {{ name }}", "subtitle": "{{ matches }} matches for {{ query }}" }, "sections": [ { "widgets": [ { "textParagraph":{ "text": "Goto Mezmo Alert on {{ name }} with {{ matches }} line(s) matched for query {{ query }}\n\nFirst matched log\nApp: {{ app }}\nHost: {{ host }}\nLevel: {{ level }}\nTag: {{ tag }}\nLine: {{ line }}" } } ] } ] } ] } ``` Which will look like # Widget Use Cases Source: https://docs.mezmo.com/docs/widget-use-cases Step-by-step examples of using Count, Time Shifted Graph, and Table widgets to monitor log counts, errors, apps, and authentication attempts. You can use widgets to monitor various aspects of application health. To learn how to build a Screen to monitor your web application, see [Use Screens and Widgets to Monitor Log Data](https://logdna.slite.com/app/docs/7uw4zKMJ-P3H80). This article lists some other ways you can use Widgets. ## Count All Log Lines for the Past Day 1. Create a **Count** widget. 2. Set the duration to **Last 1 Day**. 3. Optionally set the host and field value if you want to filter the log lines. ## Monitor Error Logs There are multiple ways to measure error logs. ### **Time Shifted Graph** Compare errors over a period. 1. Create a **Time Shifted** widget. 2. Set field to `level`. 3. Select **Advanced** filtering and enter the query `level:(warn OR error)`. 4. Set the duration to **Last 1 Week**. ### **Count** See the number of errors using a filter. 1. Create a **Count** widget. 2. Select **Advanced** filtering and enter the query `level:(warn OR error)`. 3. Set the **Duration** to **Last 1 Week.** 4. To get a view of errors over time, you can create the widget again and choose a different duration. ### **Table** Use a table to see error levels. 1. Create a **Table** widget. 2. Under **Data**: 1. Set **Group by** to `level`. 2. Set **Field** to `All Lines`. 3. Set **Duration** to `Last 1 Week`. 4. Under **Appearance**: 1. Set Left Column Label to `Level`. 2. Set Right Column Label to `Counts`. ## See Log Lines by App 1. Create a **Table** widget. 2. Under **Data**: 1. **Group** by `app`. 2. Set **Field** to `host`. 3. Optionally set the field value. 3. Set the **Duration**. 4. Under **Appearance**: 1. Set Left Column Label to `App`. 2. Set Right Column Label to `Counts`. ## Authentication Attempts 1. Create a **Count** widget. 2. Under **Data**: 1. Set **Field** to `host`. 2. Set **Field** value to the host you want to monitor. 3. Click **Advanced Monitoring** and add `authentication` 3. Set the **Duration** to **Last 1 Day.** # Windows Security Template Source: https://docs.mezmo.com/docs/windows-security-template Gain visibility into Windows security events with prebuilt Views, Boards, and Screens, plus the required NXLog configuration. The Windows Security Template enables you to quickly gain insights into excessive login attempts, audits on cleared logs, or anomalous access patterns. Set up alerts to monitor when unexpected events happen. or use our dashboards to get constant visibility into the access patterns of your servers. Browse the[ full library of Mezmo Templates](https://app.mezmo.com/manage/template-library). ## NXLog Requirement and Configuration The Windows Security Template requires NXLog to be set up to collect security event logs. Uncomment the line `` in your NXLog config file and restart NXLog to apply changes. The [NXLog](/docs/ingestion-integrations/nxlog-for-windows) ingestion integration topic contains more information on setting up NXLog. The Windows Security Template will not work with other integrations such as FluentD. ## Views Views are saved shortcuts to a specific set of filters and search queries. You can also add Alerts to views to notify you when specific conditions are met. Check out the topic [Add Alerts to Views](/docs/add-alerts-to-views) for more information. * 1102 / Audit log cleared * 4616 / System time was changed * 4624 / Successful account log on * 4625 / An account failed to log on * 4634 / An account logged off * 4720 / User account created * 4725 / Disabled account * 4740 / Locked account * 4946 / Firewall exception added * 5025 / Windows Firewall stopped ### Boards Boards are collections of graphs. Using boards, you can track trends with response codes and understand how they fluctuate over time at a glance. Drill down using subplots to see which host or path is generating the most errors. Check out the topic [Visualize Log Data with Graphs](/docs/visualize-log-data-with-graphs) for more information. * Windows Server Activity * Events Count by Channel * Failed Logins * Successful Logins ### Screens Screens are collections of customized dashboards that can display data in various forms. See the topic [Use Screens and Widgets to Monitor Log Data](/docs/use-screens-and-widgets-to-monitor-log-data) for more information. * Security log events daily and weekly trends * Distribution of log events by event id * Distribution of log on events by user name * Total successful and failed authentications per week * Total log events per week # 1 - Introduction Source: https://docs.mezmo.com/guide-to-pipeline-architecture/1--introduction Workshop overview and prerequisites for optimizing OpenTelemetry logs, metrics, and traces with Mezmo Pipelines. If you run into any issues or have feedback on either the workshop or Pipeline, please reach out to us at [support@mezmo.com](mailto:support@mezmo.com). ## Prerequisites * A Mezmo account, you can [sign up for a trial here](https://www.mezmo.com/sign-up-mezmo-platform). * [Docker](https://www.docker.com/get-started/) ## Overview In this workshop, we will be exploring telemetry data produced with the [OpenTelemetry Demo](https://github.com/mezmo/opentelemetry-demo) while optimizing it for both MTTR and cost. To accomplish this we will: 1. Create a OpenTelemetry Log, Metric, and Trace [Shared Source](/telemetry-pipelines/shared-sources) in Mezmo. 2. [Configure OpenTelemetry collector](https://github.com/mezmo/opentelemetry-demo/mezmo-otel-config-extras.yml) with Mezmo Shared Source credentials. 3. Explore the OpenTelemetry Logs via [Data Profiling](/telemetry-pipelines/data-profiling) 4. Send log data to [Mezmo Log Analysis](/docs) 5. [Aggregate](/telemetry-pipelines/reduce-processor) specific log patterns 6. [Parse](/telemetry-pipelines/parse-sequentially-processor) custom Apache data 7. [Aggregate OpenTelemetry Metrics](/telemetry-pipelines/aggregate-processor) to lower fidelity 8. [Sample OpenTelemetry Traces](/telemetry-pipelines/sample-processor) 9. Configure Pipelines to be [Responsive](/telemetry-pipelines/configure-responsive-pipelines) (i.e., capture full fidelity when in an incident or deployment state) ## Final Product In the end you are going to build four Pipelines that look like these * Log Profiling Pipeline Image * Log Handler Pipeline Image * Metric Handler Pipeline Image * Trace Handler Pipeline Image These pipelines will optimize your OpenTelemetry data by aggregating, parsing, and configuring data flow responsively. By allowing for easy, granular control you can make sure the right data winds up where it belongs. The end result is a system that provides the insight needed, at the fidelity when it's needed, leading to an order of magnitude in savings. # 2 - Create Shared Sources Source: https://docs.mezmo.com/guide-to-pipeline-architecture/2---shared-sources Create OpenTelemetry log, metric, and trace shared sources in Mezmo for use in later pipeline steps. If you run into any issues or have feedback on either the workshop or Pipeline, please reach out to us at [support@mezmo.com](mailto:support@mezmo.com). ## Create an OpenTelemetry Log Shared Source This is the source you will use for the OTel Log Handler Pipeline in Step 5. 1. In the [Mezmo Web App](https://app.mezmo.com), go to **Pipelines > Shared Sources**. 2. Click **New shared source**. 3. Select **OpenTelemetry Logs**, and give the shared source a **Title**. You should save the access key for all the shared source, you will need it for future steps. ## Create an OpenTelemetry Metric Shared Source This is the source you will use for the Metric Handler pipeline in Step 6. 1. In **Pipelines > Shared sources**, click **New shared source.** 2. Select **OpenTelemetry Metrics**, and give the shared source a **Title.** ## Create an OpenTelemetry Trace Shared Source This is the source you will use for the Trace Handler pipeline in Step 7. 1. In **Pipelines > Shared sources**, click **New shared source.** 2. Select **OpenTelemetry Metrics**, and give the shared source a **Title.** # 4 - Analyze the Source Data Source: https://docs.mezmo.com/guide-to-pipeline-architecture/3---configure-and-build-the-demo Build a Log Explorer pipeline with a Script Execution and Data Profiler processor to analyze your OTel source data. If you run into any issues or have feedback on either the workshop or Pipeline, please reach out to us at [support@mezmo.com](mailto:support@mezmo.com). Mezmo's [Data Profiler](/telemetry-pipelines/data-profiler-processor) analyzes your source data and provides a [a data profile](/telemetry-pipelines/data-profiling) that helps you understand your source data, and configure the [Pipeline Processors](/telemetry-pipelines/supported-processors) to optimize it for your purposes. In this step, you'll set up a pipeline with the shared OTel sources that will include a [Script Execution Processor](/telemetry-pipelines/js-script-processor) to format the data for analysis, and a [Data Profiler Processor](/telemetry-pipelines/data-profiler-processor) to analyze it. ## Create a Log Explorer Pipeline 1. In the [Mezmo Web App](https://app.mezmo.com), go to **Pipelines** and click **New Pipeline.** 2. Select **Create a blank pipeline**. 3. For **Pipeline Name,** enter `Log Explorer`. 4. Under **Deployment Options**, select **SaaS**. 5. Under **Select a path**, select **Create a blank pipeline**. 6. Click **Continue**. ## Add the OpenTelemtry Log Source 1. In the Pipeline Map, click **Add Source**. 2. Under **Shared Sources**, select the **OTel Log Source**. 3. Click **Save**. The Source will be added to the Pipeline Map. ## Add the OTel Mapping Script This script will map OTel fields to a format for the Data Profiler to analyze. 1. In the Profile Map, click **Add Processor**. 2. Select **Script Execution**. 3. Copy and paste this script into the **Script** field. 4. Click **Save**. 5. Connect the Source to the Script Execution Processor. ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { let line = message let app = metadata.resource.attributes["container.name"] let host = metadata.resource.attributes["container.hostname"] let level = metadata.level if( app == null || app == '' ){ app = metadata.resource["service.name"] } if( app == null || app == '' ){ app = metadata.resource["service_name"] } if( app == null || app == '' ){ app = metadata.scope.name } if( app == null || app == '' ){ app = 'na' } if( host == null || host == '' ){ host = metadata.headers["x-bus-partition-key"] } if( host == null || host == '' ){ host = metadata.attributes["log.file.path"] } if( host == null || host == '' ){ host = 'na' } if( level == null || level == '' ){ level = annotations.level } let new_msg = { "line":line, "app":app, "host":host, "level":level } // Extract metadata to top level fields for( const meta of Object.entries(metadata) ){ let meta_name = 'metadataotel_' + meta[0].toString() let meta_val = meta[1] new_msg[meta_name] = meta_val } return new_msg } ``` ## Add a Data Profiler Processor 1. In the Pipeline Map, click **Add Processor**. 2. Select **Data Profiler**, and give it the name `OTel Demo Log Exploration`. 3. Click **Save**. 4. Connect the Script Execution Processor to the Data Profiler Processor. ## Deploy the Pipeline and View the Data Profile In the Pipeline Map, click **Deploy Pipeline** to activate the Pipeline. The Data Profiler will begin to run, and after a few minutes you will see a Data Profile similar to this: Two things you will immediately notice: 1. The `load-generator` service is sending a huge volume of logs simply stating a homepage is being flooded. This is standard behavior of the OpenTelemetry Demo using the [Feature Flag: loadgeneratorFloodHomepage](https://opentelemetry.io/docs/demo/feature-flags/) , but this data is noisy and costly to retain. Image 2. There are unparsed events that appear to be custom Apache logs being sent from the `frontend-proxy` service. While these are [defined in the demo code here](https://github.com/mezmo/opentelemetry-demo/blob/main/src/frontend-proxy/envoy.tmpl.yaml#L80), we can take steps to make sure this data is structured and parsed properly to be searchable in any downstream Observability system. Image In the next step, you will build out a log telemetry pipeline to address both of these potential issues. # 3 - Build the OTel Demo Source Source: https://docs.mezmo.com/guide-to-pipeline-architecture/4---profile-log-data Clone the OpenTelemetry Demo, configure it with your Mezmo shared source credentials, and build it locally. If you run into any issues or have feedback on either the workshop or Pipeline, please reach out to us at [support@mezmo.com](mailto:support@mezmo.com). This workshop uses an OpenTelemetry Demo with expanded logs that you will connect to the shared sources you created in the previous step. In this step you will clone the repo, modify the config, and then build the OTel demo. 1. Run `git clone https://github.com/mezmo/opentelemetry-demo` into a local folder. 2. In the local folder, modify the `mezmo-otel-config-extras.yml` file with the endpoints and access keys of the shared sources you created in the previous step. 3. To build the OTel demo source, run `sh run.sh`. This will build the demo and deploy it to `localhost:8080`. It can take up to 15 minutes for the demo to build and deploy, so you may want to take a break before proceeding with the rest of the tutorial. ## # 5 - Create an OTel Log Handler Pipeline Source: https://docs.mezmo.com/guide-to-pipeline-architecture/5---optimize-logs Build a responsive log pipeline that parses custom Apache logs, reduces flood events, and routes data by operational state to Log Analysis. If you run into any issues or have feedback on either the workshop or Pipeline, please reach out to us at [support@mezmo.com](mailto:support@mezmo.com). In this step you will create a responsive pipeline to handle the OpenTelemetry log data that includes processing functionality to optimize the custom Apache logs identified in the Data Profile. The Pipeline will send the processed data to the [Mezmo Log Analysis](/telemetry-pipelines/mezmo-destination) destination. ## Pipeline Architecture ## 1 - Create the Pipeline and Add the Source 1. In the Mezmo Web app, click **New Pipeline** and name it `Log Handler`. 2. In the Pipeline Map, click **Add Source**, then select the OpenTelemetry Log source you created in Step 2. ## 2 - Add State Variables A responsive pipeline changes its functioning based on detection of state changes. For this example, you will use the [Script Execution Processor](/telemetry-pipelines/js-script-processor) to add variables to the data that indicate the operational state of the pipeline. 1. Click the `...`menu in the upper-right corner of the OpenTelemetry Log source. 2. Select **Add Node > Add Processor > Script Execution.** 3. **Connect the processor to the OTel Log source.** 4. Copy and paste this script into the **Script** field in the processor configuration panel, then click **Save**. In addition to adding state variables to the data, this script also normalizes the data so it is more compatible with Mezmo Log Analysis. ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { metadata.resource.attributes["pipeline.path"] = "with_mezmo" const state = getPipelineStateVariable("operational_state") let line = message let app = metadata.resource.attributes["container.name"] let host = metadata.resource.attributes["container.hostname"] let level = metadata.level if( app == null || app == '' ){ app = metadata.resource["service.name"] } if( app == null || app == '' ){ app = metadata.resource["service_name"] } if( app == null || app == '' ){ app = metadata.scope.name } if( app == null || app == '' ){ app = 'na' } if( host == null || host == '' ){ host = metadata.headers["x-bus-partition-key"] } if( host == null || host == '' ){ host = metadata.attributes["log.file.path"] } if( host == null || host == '' ){ host = 'na' } if( level == null || level == '' ){ level = annotations.level } metadata.headers = null let new_msg = { "line":line, "app":app, "host":host, "level": level, "op_state":state, "meta":metadata, '_cnt': 1 } if( message == null ){ return null } return new_msg } ``` ## 3 - Create the Custom Apache App Logs Parsing Processor Chain As you saw in the Data Profile from step 4, the OTel data being sent by`frontend-proxy` is an unparsed, custom [format defined by the OpenTelemetry demo](https://github.com/mezmo/opentelemetry-demo/blob/main/src/frontend-proxy/envoy.tmpl.yaml#L80). This data needs to be send through a specialized processor chain that contains a [Route Processor](/telemetry-pipelines/route-processor), a [Parse Sequentially Processor](/telemetry-pipelines/parse-sequentially-processor) that includes a Grok parser, and a [Script Execution Processor](/telemetry-pipelines/js-script-processor) to structure it and make it more easily searchable. ### Route Custom Apache App Data 1. In the Pipeline Map, click **Add Processor**, and select **Route.** 2. Connect the Route processor to the Script Execution processor. 3. Enter these configuration options for the processor, then click **Save**. | Configuration Options | Setting | | ------------------------- | ----------------------------------------- | | **Title** | `App Router` | | **Route** | `Frontend Proxy` | | **Conditional Statement** | `if (message.app equal 'frontend-proxy')` | ### Parse the App Data 1. In the Pipeline Map, click **Add Processor,** and select **Parse Sequentially**. 2. Enter these configuration options for the processor, then click **Save**. 3. Connect the Parse Sequentially processor to the `Frontend Proxy` output of the Route processor. | Configuration Option | Setting | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Field | `message.line` | | Target Field | `message.line_parsed` | | Custom Parser Title | `Custom Apache` | | Customer Parser | `Grok Pattern` | | Custom Parser Pattern | `%{SQUARE_BRACKET`}%\{TIMESTAMP\_ISO8601:dt`}%{SQUARE_BRACKET} %{DOUBLE_QUOTE}%{DATA:method} %{DATA:path} %{DATA:http_protocol}%{DOUBLE_QUOTE} %{DATA:rsp_code} %{DATA:rsp_flags} %{DATA:rsp_code_details} %{DATA:conn_term_details} %{DOUBLE_QUOTE}%{DATA:upstream_transport_failure_reason}%{DOUBLE_QUOTE} %{DATA:bytes_received} %{DATA:bytes_sent} %{DATA:duration} %{DATA:rsp_upstream_service_time} %{DOUBLE_QUOTE}%{DATA:req_forward_for}%{DOUBLE_QUOTE} %{DOUBLE_QUOTE}%{DATA:req_user_agent}%{DOUBLE_QUOTE} %{DOUBLE_QUOTE}%{DATA:req_id}%{DOUBLE_QUOTE} %{DOUBLE_QUOTE}%{DATA:req_authority}%{DOUBLE_QUOTE} %{DOUBLE_QUOTE}%{DATA:upstream_host}%{DOUBLE_QUOTE} %{DATA:upstream_cluster} %{DATA:upstream_local_addr} %{DATA:downstream_local_addr} %{DATA:downstream_remote_addr} %{DATA:requested_server_name} %{GREEDYDATA:route_name}` | ### Merge Custom App Data Lines In this case, the Script Execution processor is used to preserve the original line for the custom app data. 1. In the Pipeline Map, click **Add Processor** and select **Script Execution**. 2. Copy and paste this script into the **Script** field. 3. Click **Save**, then connect the Script Execution processor the to the `Custom Apache` output of the Parse Sequentially processor. ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { let old_line = message.line message.line = message.line_parsed message.line.message = old_line message.line_parsed = null return message } ``` ## 4 - Route Data Based on State The next Route processor in the chain will route data based on the operational state. For the Incident and Deploy states, the data is sent directly to Mezmo Log Analysis, while Normal or unmatched data is sent through a final processing chain to aggregate and reduce the volume of the custom app log events. 1. In the Pipeline map, click **Add Processor,** and select **Route**. 2. Enter these configuration options for the processor, then click **Save**. | Configuration Option | Setting | | ----------------------------- | --------------------------------------------- | | Route 1 Title | `Normal` | | Route 1 Conditional Statement | `if (message.op_state` `contains` `normal)` | | Route 2 Title | `Incident` | | Route 2 Conditional Statement | `if (message.op_state` `contains` `incident)` | | Route 3 Title | `Deploy` | | Route 3 Conditional Statement | `if (message.op_state` `contains` `deploy)` | ## 5 - Create the "Flooding Homepage" Reduction Processing Chain During `normal` conditions, the log data contains a high volume of "homepage flooding" logs that convey little information in their unprocessed state. One technique to deal with noisy data like this is to [convert events to metrics. ](/practioner-guide-data-optimization/pipeline-example--convert-200-events-to-metrics) In this case, the data is routed to a [Reduce Processor](/telemetry-pipelines/reduce-processor), which will provide a count of the events over five minutes to Mezmo Log Analysis, and then sent to a Script Execution process to generate a summary message. ### Route the Data 1. In the Pipeline Map, click **Add Processor,** then select **Route**. 2. Enter these configuration options for the processor, then click **Save**. 3. Connect the `Normal` and `Unmatched` routes of the State Router to the input of this Route processor. | Configuration Options | Setting | | --------------------------------- | ------------------------------------------------------------------------------------------------ | | **Route 1 Title** | `Flooding homepage` | | **Route 1 Conditional Statement** | `if (message.app equal load-generator) AND (message.line contains Flooding homepage, iteration)` | ### Reduce the Data * In the Pipeline Map, click **Add Processor,** then select **Reduce**. * Enter these configuration options for the processor, then click **Save**. | Configuration Option | Setting | | ------------------------ | ------------------- | | Title | `5min Flood count` | | Duration | `5 minutes` | | Group by Field Path | `message.host` | | Merge Strategy per Field | | | Field Path | `message.cnt sum` | ### Add a Summary Message Finally, we will convert the output into a summary message using the following configuration ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { message.line = { 'message':'Flooded homepage ' + message._cnt.toString() + ' times', 'count': message._cnt } return message } ``` ## 6 - Sample Normal State Logs For the `unmatched` logs that pass through the Router, you only need to sample a small proportion of them while the pipeline is in the `normal` operational state. For this example, you will add a Sample processor that will sample every 1 in 10 of the unmatched logs. 1. In the Pipeline Map, click **Add Processor,** then select **Sample**. 2. Enter these configuration options for the processor, then click **Save**. | Configuration Options | Setting | | --------------------- | ------- | | **Rate** | `1/10` | ## 7 - Connect to Mezmo Log Analysis Finally, we will send all of this data into Mezmo Log Analysis. Because of our earlier work normalizing data in Step 3, we can simply add a final Destination to all nodes (including the `Incident` and `Deploy` paths). 1. In the Pipeline Map, click Add Destination. 2. Select Mezmo Log Analysis, enter these settings, then click **Save**. 3. After saving the configuration, connect the Log Analysis destination to the outputs of the other processors as shown in the architecture schematic. | Configuration Options | Setting | | ---------------------------- | -------------------------------------------- | | **Ingestion Key** | Generate a new one or select an existing one | | **Host Name** | `{{message.host}}` | | **Tags** | `otel-demo` | | **Log Construction Scheme** | `message pass-through` | ## 7 - Deploy the Pipeline To activate the pipeline, click **Deploy** in the upper-right corner of the pipeline map. ## 8 - Initiate State and Grab State ID Our final step is to initiate and grab the state ID for the pipeline in Normal operation. Click the **State** menu in the upper-left corner of the active pipeline, change the state to `Incident`, then change it back to `Normal.` Now that has been initiated, you will need get the `Log Handler` pipeline's ID (found in the URL at `app.mezmo.com/ACCOUNT_ID/pipelines/PIPELINE_ID`) along with a Pipeline API Key here. Then, modify the following script with both that `PIPELINE_ID` and Pipeline API Key ```bash theme={null} curl --request GET \ --url 'https://api.mezmo.com/v3/pipeline/state-variable?pipeline_id=PIPELINE_ID' \ --header 'Authorization: Token PIPELINE_API_KEY' ``` Take the response and save the `STATE_ID` for later. You will find it in the `operational_state`'s data packet, which should look something like this: ```bash theme={null} { "meta": { "pk": "id", "type": "pipeline-state-variable", "links": { "self": { "create": { "uri": "/v3/pipeline/{pipeline_id}/state-variable", "method": "post" }, "list": { "uri": "/v3/pipeline/{pipeline_id}/state-variable", "method": "get" }, "replace": { "uri": "/v3/pipeline/{pipeline_id}/state-variable/{id}", "method": "put" }, "update": { "uri": "/v3/pipeline/{pipeline_id}/state-variable", "method": "patch" }, "detail": null }, "related": { "pipeline": { "list": "/v3/pipeline", "detail": "/v3/pipeline/{pipeline_id}" } } }, "page": { "next": null, "previous": null } }, "data": [ { "id": "STATE_ID", "account_id": "ACCOUNT_ID", "pipeline_id": "PIPELINE_ID", "state": { "operational_state": "normal" }, "created_at": "UTC Timestamp", "updated_at": "UTC Timestamp" } ] } ``` ## 9 - View In Mezmo Log Analysis Navigate to Log Analysis and view the incoming data. In particular, if you used the `tag` above you can simply search for `tag:otel-demo`. First, look for the aggregated data by searching for `tag:otel-demo "flooded homepage"`. Notice that instead of raw lines like we saw in the Profile, we now have an aggregated message to watch saving tens of thousands of log lines. Image Second, check out the newly parsed data by searching for `tag:otel-demo app:frontend-proxy`. While logs are displayed nicely in the Log Viewer, you can expand a line and see all the nested structure that is easily searchable. For instance, to see all 2xx responses enter the query `tag:otel-demo app:frontend-proxy resp_code:(>=200 AND <300)` Image If you want to learn more about Log Analysis and creating things like saved Views, Alerts and more check out our [docs here](/docs) or reaching out to [support@mezmo.com](mailto:support@mezmo.com) # 6 - Create an OpenTelemetry Metric Handler Pipeline Source: https://docs.mezmo.com/guide-to-pipeline-architecture/6---optimize-metrics Build a responsive metric pipeline that aggregates metrics and limits tag cardinality, routing data by operational state. If you run into any issues or have feedback on either the workshop or Pipeline, please reach out to us at [support@mezmo.com](mailto:support@mezmo.com). In this step you will create a responsive pipeline to handle the OpenTelemetry metrics data. ## Pipeline Architecture This schematic shows the architecture of the Pipeline you will create in this stop. The numbers in the schematic correspond to the step in the build process. ## 1 - Create the Pipeline and Add the Source 1. In the Mezmo Web app, click **New Pipeline** and name it `Metric Handler`. 2. In the Pipeline Map, click **Add Source**, then select the OpenTelemetry Metric source you created in Step 2. ## 2 - Add State Variables A responsive pipeline changes its functioning based on detection of state changes. For this example, you will use the [Script Execution Processor](/telemetry-pipelines/js-script-processor) to add variables to the data that indicate the operational state of the pipeline. 1. Click the `...`menu in the upper-right corner of the OpenTelemetry Metric source. 2. Select **Add Node > Add Processor > Script Execution.** 3. Copy and paste this script into the **Script** field in the processor configuration panel, then click **Save**. ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { const state = getPipelineStateVariable("operational_state") message.op_state = state message.name = message.name.toString() message.tags.op_state = state metadata.resource.attributes["pipeline.path"] = "with_mezmo" if( message == null ){ return null } return message } ``` ## 3 - Route Data Based on State You can now set the Pipeline to route data based `on operational_state` with the [Route Processor](/telemetry-pipelines/route-processor). 1. **In the Pipeline Map, click Add Processor.** 2. Select **Route Processor,** and for **Title**, enter `State Router`. 3. You will create three routes, one for the `Normal` state, one for the `Incident` state, and one for the `Deploy` state. After you configure the options for the `Normal` state, click **Add route** to configure the `Incident` and `Deploy` routes. ### Normal Route Configuration | Configuration Option | Setting | | ------------------------- | ------------------------------------- | | **Title** | `Normal` | | **Conditional Statement** | `if message.op_state contains normal` | ### Incident State Configuration | Configuration Option | Setting | | ------------------------- | --------------------------------------- | | **Title** | `Incident` | | **Conditional Statement** | `if message.op_state contains incident` | ### Deploy State Configuration | Configuration Option | Setting | | ------------------------- | ------------------------------------- | | **Title** | `Deploy` | | **Conditional Statement** | `if message.op_state contains deploy` | ## 4 - Create the Metric Data Optimization Processor Chain In normal functioning, a common approach to reducing metric volume is to aggregate metrics and reduce tag cardinality. In this step, you will connect a Script Execution Processor, a [Tag Cardinality Limit Processor](/telemetry-pipelines/metrics-tag-cardinality-limit-processor), and an [Aggregate Processor](/telemetry-pipelines/aggregate-processor), in that order, to the normal and unmatched routes of the Route Processor. ### Script Execution Processor Configuration 1. In the Pipeline Map, add a Script Execution Processor to the Pipeline, and connect it to both the unmatched and normal routes. 2. Copy and paste this script into the processor configuration. ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { let service_name = message.tags.service_name let host_id = message.tags.host_id if( service_name == null ){ service_name = metadata.resource.attributes['service.name'] } if( service_name == null ){ service_name = 'NA' } if( host_id == null ){ host_id = metadata.resource.attributes['host.id'] } if( host_id == null ){ host_id = 'NA' } message.tags = { 'service_name': service_name, 'host_id': host_id } return message } ``` ### Tag Cardinality Limit Processor Configuration This configuration will limit the number of tags for the `host_id field` to 10. 1. In the Pipeline Map, add a Tag Cardinality Limit Processor and connect it to the Script Execution Processor. 2. Enter these configuration options for the processor. | Configuration Option | Setting | | -------------------- | ---------------------- | | **Tags** | `message.tags.host_id` | | **Action** | `drop_tag` | | **Value Limit** | `10` | | **Mode** | `Probablistic` | ### Aggregate Processor Configuration This configuration will aggregate metrics based on a five minute interval. 1. In the Pipeline Map, add an Aggregate Processor and connect it to the Tag Cardinality LImit Processor. 2. Keep the default configuration settings, but set **Interval (seconds)** to `300`. ## 5 - Add the Blackhole Destination You can send your optimized data to any of Mezmo's [Supported Telemetry Data Destinations](/telemetry-pipelines/supported-telemetry-data-destinations), but in this case, the route will terminate in a [Blackhole](/telemetry-pipelines/blackhole-destination) destination that drops all data sent to it. This is useful for testing the data processing of your Pipeline before sending it to production destination. 1. In the Pipeline Map, click **Add Destination**. 2. Select **Blackhole**, and connect it to the to outgoing routes of the Route Processor as shown in the pipeline architecture schematic. ## Deploy the Pipeline To activate the Pipeline, click **Deploy**. ## Initialize State and Get State ID As you did with the Log Handler Pipeline, you need to initialize the state of the Pipeline and get the `State ID` of the pipeline for reference in Step 8. 1. In the Pipeline Map, click on the **State** setting in the upper-left corner, and change it to `Incident`. 2. Change the **State** setting back to `Normal`. This will initialize the Normal state, and generate the`Pipeline_ID` for the pipeline `state-variable`. 3. In a terminal, run this command with the First, flip the State in the UX from Normal to Incident and back to Normal to initialize. Then, in your terminal run the following command with the metric `pipeline's ID` and grab that `State ID`. ```bash theme={null} curl --request GET \ --url 'https://api.mezmo.com/v3/pipeline/state-variable?pipeline_id=PIPELINE_ID' \ --header 'Authorization: Token PIPELINE_API_KEY' ``` # 7 - Create an OTel Trace Handle Pipeline Source: https://docs.mezmo.com/guide-to-pipeline-architecture/7---optimize-traces Build a pipeline that samples OpenTelemetry traces during normal operation and routes data by operational state. If you run into any issues or have feedback on either the workshop or Pipeline, please reach out to us at [support@mezmo.com](mailto:support@mezmo.com). In this step you will create a pipeline to handle OpenTelemetry Traces. ## Pipeline Architecture ## 1 - Create the Pipeline and Add the Source 1. In the Mezmo Web app, click **New Pipeline** and name it `Trace Handler`. 2. In the Pipeline Map, click **Add Source**, then select the OpenTelemetry Trace source you created in Step 2. ## 2 - Add State Variables A responsive pipeline changes its functioning based on detection of state changes. For this example, you will use the [Script Execution Processor](/telemetry-pipelines/js-script-processor) to add variables to the data that indicate the operational state of the pipeline. 1. Click the `...`menu in the upper-right corner of the OpenTelemetry Metric source. 2. Select **Add Node > Add Processor > Script Execution.** 3. Copy and paste this script into the **Script** field in the processor configuration panel, then click **Save**. ```javascript theme={null} function processEvent(message, metadata, timestamp, annotations) { const state = getPipelineStateVariable("operational_state") message.op_state = state message.name = message.name.toString() message.tags.op_state = state metadata.resource.attributes["pipeline.path"] = "with_mezmo" if( message == null ){ return null } return message } ``` ## Route Based on State You can now set the Pipeline to route data based `on operational_state` with the [Route Processor](/telemetry-pipelines/route-processor). 1. **In the Pipeline Map, click Add Processor.** 2. Select **Route Processor,** and for **Title**, enter `State Router`. 3. You will create three routes, one for the `Normal` state, one for the `Incident` state, and one for the `Deploy` state. After you configure the options for the `Normal` state, click **Add route** to configure the `Incident` and `Deploy` routes. | Configuration Options | Setting | | --------------------------------- | ---------------------------------------- | | **Route 1 Title** | `Normal` | | **Route 1 Conditional Statement** | `message.op_state` `contains` `normal` | | **Route 2 Title** | `Incident` | | **Route 2 Conditional Statement** | `message.op_state` `contains` `incident` | | **Route 3 Title** | `Deploy` | | **Route 3 Conditional Statement** | `message.op_state` `contains` `deploy` | ## 5 - Sample Traces in Normal State For the `unmatched` and `normal` data that pass through the Router, you only need to sample a small proportion of them while the pipeline is in the `normal` operational state. For this example, you will add a Sample processor that will sample every 1 in 10 of the unmatched logs. 1. In the Pipeline Map, click **Add Processor,** then select **Sample**. 2. Enter these configuration options for the processor, then click **Save**. | Configuration Options | Setting | | --------------------- | ------- | | **Rate** | 10 | 6 - Add the Blackhole Destination You can send your optimized data to any of Mezmo's [Supported Telemetry Data Destinations](/telemetry-pipelines/supported-telemetry-data-destinations), but in this case, the route will terminate in a [Blackhole](/telemetry-pipelines/blackhole-destination) destination that drops all data sent to it. This is useful for testing the data processing of your Pipeline before sending it to production destination. 1. In the Pipeline Map, click **Add Destination**. 2. Select **Blackhole**, and connect it to the to outgoing routes of the Route Processor as shown in the pipeline architecture schematic. ## Deploy the Pipeline To activate the Pipeline, click **Deploy**. ## Initiate State and Grab State ID As you did with the Log Handler Pipeline, you need to initialize the state of the Pipeline and get the `State ID` of the pipeline for reference in Step 8. 1. In the Pipeline Map, click on the **State** setting in the upper-left corner, and change it to `Incident`. 2. Change the **State** setting back to `Normal`. This will initialize the Normal state, and generate the`Pipeline_ID` for the pipeline `state-variable`. 3. In a terminal, run this command with the trace `pipeline's ID` and grab that `State ID`. ```bash theme={null} curl --request GET \ --url 'https://api.mezmo.com/v3/pipeline/state-variable?pipeline_id=PIPELINE_ID' \ --header 'Authorization: Token PIPELINE_API_KEY' ``` # 8 - Test the Responsive Pipelines Source: https://docs.mezmo.com/guide-to-pipeline-architecture/8---optimize-responsibly Use the switch-state script to toggle pipeline states and observe how responsive pipelines adjust fidelity during incidents. If you run into any issues or have feedback on either the workshop or Pipeline, please reach out to us at [support@mezmo.com](mailto:support@mezmo.com). ## Why It Matters Telemetry data is both fundamental and costly for any business. While this data is invaluable for troubleshooting, monitoring and various other concerns, it is not always valuable at the same time. Mezmo introduced [Responsive Pipelines](/telemetry-pipelines/configure-responsive-pipelines) specifically to address the dynamic nature of telemetry data. By allowing for extreme configurability, telemetry flows can be tuned for changing circumstances from incidents to deployments. ## 1 - Modify Responsive Test script Using the State ID's for your Log, Metric and Trace pipelines modify [`switch_state.sh`](https://github.com/mezmo/opentelemetry-demo/blob/main/switch_state.sh) with the proper credentials. ## 2 - Run Responsive Test Script Now let's run the script with your desired state to initiate Mezmo Pipeline flow changes. For instance, to flip to Incident mode you would run: ```bash theme={null} sh switch_state.sh incident ``` ## 3 - Evaluate Impact Notice that when in `Normal` mode, data in your pipelines are sampled and rolled up ensuring the needed signals are captured while remaining cost conscious. However, when in `Incident` or `Deployment` modes, data is grabbed at full fidelity. You will also see this represented in the Mezmo Web App. ### Normal Mode Image ### Incident Mode Image Due to the flexibility of [Mezmo's API](/docs/api-referencesk9h), any pipeline can be integrated with just about any Incident Management or Deployment method, from PagerDuty to Github to Shell scripts. To learn more, reach out to Mezmo at [support@mezmo.com](mailto:support@mezmo.com). # Basic Data Optimization Pipeline Source: https://docs.mezmo.com/guide-to-pipeline-architecture/basic-data-optimization-pipeline Explore the Demo Pipeline's Route, Filter, and Encrypt processors for handling transaction errors and encrypting credit card data. ## The Scenario The Demo Pipeline is designed to illustrate a typical scenario, where there are multiple pipeline sources, a Route Processor that filters data based on conditional criteria, and additional processors for the filtered data that prepares it for routing to one destination for analysis, and another for archival storage. The specific scenario is a typical processing of JSON and Financial data to identify errors related to credit card transactions, which also requires encryption of credit card data before it can be sent to an analytical tool. In this tutorial, you'll see how to use the Route, Filter, and Encrypt Processors to accomplish this. ## Architecture Overview To explore the Demo Pipeline in the Mezmo Web App: 1. Log in to [the Mezmo Web App](https://app.mezmo.com). 2. Click **Pipelines**. 3. Under **Cloud**, select **Demo Pipeline**. If necessary, you can also click **Re-start Pipeline** to send the demo source data through the Pipeline. Image ### 1 - Sources The Demo Pipeline has two Sources, which are both versions of the [Demo Logs](/telemetry-pipelines/demo-logs-source). You can use this Source to build your Pipelines using sample data before connecting them to live Production Sources, to make sure that your Processors are producing the results you want. #### Financial Data and JSON Use the [Pipeline Tap feature](/telemetry-pipelines/view-pipeline-data) to view the sample Financial and JSON data. You can also download the sample data to view the full JSON, and build your own sample data. Image #### JSON Data Image ### 2 - Route Processor The [Route Processor](/telemetry-pipelines/route-processor) uses conditional statements to send data to other processors or destinations. In this case, there are four statements: | Route Name | Purpose | Conditional Statement | | Routed To | | --------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------- | - | ------------------------------ | | Purchase Transactions | Selects transaction events | `if (exists(.event) AND .event contains 'transaction')` | | Allow "Card Denied" Filter | | Login/Logout Events | Selects login and logout events | `if (exists(.event) AND .event contains 'log')` | | Drop Login/Logout Event Filter | | HTTP non-200s | Selects HTTP events that are not 200s (Success) | `if (exists(.status) AND .status greater 200)` | | Long Term Analysis Destination | | Unmatched | Bucket for any data that is not selected by the other conditional statements | None | | Archival Destination | You can test your Route Processor by using a [PIpeline Tap](/telemetry-pipelines/view-pipeline-data) to view the data flowing into it from the Sources, and inserting a tap for each route to make sure that data is passing through as expected. ### 3 and 4 - Filter Processors The Route Processor sends matched data to two [Filter Processors](/telemetry-pipelines/filter-processor). | Filter Processor | Purpose | Conditional Statement | Routed To | | ------------------------ | ------------------------------------------------------------------------------------- | ---------------------------------------------------------- | -------------------------------------------- | | Allow "Card Denied" | Filters the Purchase Transactions data to select those with a result of "Card Denied" | `if (.transaction.result_reason __contains 'card_denied')` | Encrypt Card Details Encrypt Field Processor | | Drop Login/Logout Events | Drops the Login/Logout Events matched by the Route Processor | `if (.access.action contains 'log')` | Long Term Analysis Destination | ### 5 - Encrypt Field Processor For security compliance, credit card information should be encrypted before reaching the Long Term Analysis destination. With the [Encrypt Field Processor](/telemetry-pipelines/encrypt-fields-processor), you can set encryption for a specific field, along with the encryption algorithm and key, and the Initialization Vector (IV) field. | **Encrypted Field** | `.transaction.cc.cc_number` | | ------------------------------- | ---------------------------------------------------- | | **Encryption Algorithm** | `AES-256-CFB (key = 32 characters, iv=16 characters` | | **Encryption Key** | `keyenrcypt123456789keyenrcypt123` | | **Initialization Vector Field** | `.IVFIELD` | You can use the [Decrypt Field Processor](/telemetry-pipelines/decrypt-fields-processor) with the same settings if you need to later decrypt the data. ### 6 - Destinations The routed and filtered data is sent to two versions of the [Blackhole](/telemetry-pipelines/blackhole-destination) destination, one representing Long Term Analysis, the other Archival Storage. As with the Demo Logs Pipeline Source, the Black Hole destination is useful for making sure your log data is processed as expected before connecting it to a Production Destination. # Basic Data Rehydration Pipeline Source: https://docs.mezmo.com/guide-to-pipeline-architecture/basic-data-rehydration-pipeline Build pipelines to archive telemetry data to S3 and restore it to Mezmo Log Analysis using a restoration task. **Completion Time**: 10 Minutes In this tutorial you'll learn how to create basic pipelines for telemetry data archiving and restoration using the [Mezmo Archive Destination](/telemetry-pipelines/mezmo-archive-destination), the [Pipeline Data Restoration](/telemetry-pipelines/pipeline-data-restoration-source) Source, the [Mezmo Log Analysis Destination](/telemetry-pipelines/mezmo-destination), and JSON [Demo Logs](/telemetry-pipelines/demo-logs-source). ## Prerequisites You should have an S3 bucket that you can use as the archiving destination. ## Pipeline Architecture These two Pipettes illustrate the basic configuration of a Pipeline to send telemetry data to an S3 bucket, and then restore that data and send it to Mezmo Log Analysis. ### Archive Pipeline This Pipette sends [Demo Log](/telemetry-pipelines/demo-logs-source) HTTP JSON data directly to a [Mezmo Archive Destination](/telemetry-pipelines/mezmo-archive-destination) for archiving in an S3 bucket. For demonstration purposes this is a two component Pipeline, but you would typically have processor groups [for converting events to metrics](/practioner-guide-data-optimization/pipeline-example--convert-200-events-to-metrics) or others to reduce log volume between the Source and the Archive Destination. Image #### Demo Logs Source Configuration | Configuration Option | Setting | | ------------------------------------------------------------------ | ----------- | | **Interval** (the number of seconds to pause between sending logs) | `1` | | **Format** | `JSON HTTP` | #### Mezmo Archive Destination Configuration | Configuration Option | Setting | | --------------------------- | -------------------------------------------------------- | | **Batch timeout (seconds)** | `300` | | **Archive Provider** | `S3` (note that you can also send archive logs to Azure) | | **Access Key ID** | Access key for the S3 bucket | | **Secret Access Key** | Secret access key for the S3 bucket | | **Bucket** | The name of the S3 bucket | | **Region** | The AWS region where the S3 bucket is located. | ### Restoration Pipeline This Pipeline sends archived data from the [Pipeline Data Restoration Source](/telemetry-pipelines/pipeline-data-restoration-source), passes it through a [Filter Processor](/telemetry-pipelines/filter-processor) to drop data and a [Map Fields Processor](/telemetry-pipelines/map-fields-processor) to make sure that the restored data conforms to the [required schema for the log analysis destination, ](/telemetry-pipelines/required-schema-for-mezmo-log-analysis-destination) and then finally sends it to Mezmo Log Analysis. Note that this Pipeline is not active after being saved and deployed. Data will only begin to stream when the Pipeline is activated during a **Restoration Task**, described in the next section. #### Mezmo Pipeline Data Restoration Source Configuration There is no configuration for the Source other than giving it a **Title**. This is how you will identify where to send the data for the restoration task. #### Filter Processor Configuration This filter is set to only send a subset of the archived data to log analysis. | Configuration Option | Setting | | ------------------------- | ---------------------------------------------------------------------- | | **Action** | Drop events matching this criteria | | **Conditional Statement** | `if (message.status greater_or_equal 200 AND message.status less 300)` | #### Map Fields Processor Configuration This processor maps fields in the restored data to fields conform to the [schema required for Mezmo Log Analysis](/telemetry-pipelines/required-schema-for-mezmo-log-analysis-destination). | Source Field | Target Field | | ------------------ | -------------- | | `message.method` | `message.line` | | `message.referrer` | `.app` | #### Mezmo Log Analysis Destination Configuration The tags you enter in the configuration options are intended to help you easily search for restored data in the Log Viewer. | Configuration Option | Setting | | ------------------------------------------------------ | ------------------------------------------------------ | | **Mezmo Host** | `logs.mezmo.com` | | **Ingestion Key** | The ingestion key for your Mezmo Log Analysis instance | | **Hostname** | `rehydrated-data` | | **Tags** (these will be attached to the restored data) | `{{metadata.query.tags}}` `restored` `restored- data` | | **Scheme** | `Message pass-through` | ## Create the Restoration Task You must have admin privileges within your Mezmo Organization to create and run a restoration task. You should create the restoration task in the same account that is associated with the restoration pipeline. 1. In the Mezmo Web App, go to **Settings > Archiving > Pipeline Restoration**. 2. Click **New Pipeline Restoration Task**. 3. Enter a name for the restoration task. 4. Enter the time period for the data you want to restore. 5. Select the **Pipeline Archive** to restore data from. 6. Select the Pipeline where you want to send the restored data. 7. Click **Start**. You will see data begin to stream into the restoration pipeline, and then in your Mezmo Log Analysis viewer. # Basic Log Analysis Pipeline Source: https://docs.mezmo.com/guide-to-pipeline-architecture/basic-log-analysis-pipeline Configure and use the Mezmo Log Analysis Destination with a Demo Logs Source to stream telemetry data into the Log Analysis viewer. When you sign up for a trial account for the Mezmo Platform, you automatically have access to all the features of Mezmo Telemetry Pipelines. To get access to Log Analysis features, you need to add the [Mezmo Log Analysis Destination ](/telemetry-pipelines/mezmo-destination) to your Pipeline. This topic will show you how to configure and use the Log Analysis Destination with a [Demo Logs Source](/telemetry-pipelines/demo-logs-source). ## Pipeline Architecture This schematic illustrates a basic architecture in which an ingestion source sends telemetry data directly to Mezmo Log Analysis. You would typically have other Processors in the Pipeline to optimize the data to your requirements for Log Analysis, but for demo purposes this Pipeline includes only the Source and Destination. Image ## Build a Basic Log Analysis Pipeline 1. Log in to the [Mezmo Web App](https://app.mezmo.com). 2. Click the **Pipelines** icon in the left-hand navigation. 3. Click **New Pipeline**. 4. Enter a **Name** for the Pipeline, and select **Create a Blank Pipeline**. 5. In the Pipeline Map, click **Add Source**, and select **Demo Logs**. 6. In the Demo Logs configuration panel, use the default settings for **Interval** (1) and **Format** (HTTP JSON). 7. In the Pipeline Map, click **Add Destination** and select **Mezmo Log Analysis**. 8. In the Mezmo Log Analysis configuration panel, use the default settings, then click **Close**. 9. In the Mezmo Log Analysis configuration panel, click **Generate New Ingestion Key,** then click **Close.** 10. Click Deploy Pipeline. You will see the data begin to flow from the Demo Source into Log Analysis in the **Ingestion/Egress by Volume** chart at the top of the Pipeline Map. 11. To view the data in Mezmo Log Analysis, click the **Log Analysis** icon in the lower-right corner of the Mezmo Log Analysis destination node. This will launch the Log Analysis Viewer, and you will begin to see data streaming into the viewer. You will also see Log Analysis features such as **Boards**, **Screens**, and **Views** added to the left-hand navigation. Image # Build a Basic Data Rehydration Pipeline Source: https://docs.mezmo.com/guide-to-pipeline-architecture/build-a-basic-data-rehydration-pipeline Archive telemetry data to an S3 bucket and restore it to Mezmo Log Analysis using the Archive Destination, Pipeline Data Restoration Source, and a restoration task. **Completion Time**: 10 Minutes In this tutorial you'll learn how to create basic pipelines for telemetry data archiving and restoration using the [Mezmo Archive Destination](/telemetry-pipelines/mezmo-archive-destination), the [Pipeline Data Restoration](/telemetry-pipelines/pipeline-data-restoration-source) Source, the [Mezmo Log Analysis Destination](/telemetry-pipelines/mezmo-destination), and JSON [Demo Logs](/telemetry-pipelines/demo-logs-source). ## Prerequisites You should have an S3 bucket that you can use as the archiving destination. ## Pipeline Architecture These two Pipettes illustrate the basic configuration of a Pipeline to send telemetry data to an S3 bucket, and then restore that data and send it to Mezmo Log Analysis. ### Archive Pipeline This Pipette sends [Demo Log](/telemetry-pipelines/demo-logs-source) HTTP JSON data directly to a [Mezmo Archive Destination](/telemetry-pipelines/mezmo-archive-destination) for archiving in an S3 bucket. For demonstration purposes this is a two component Pipeline, but you would typically have processor groups [for converting events to metrics](/practioner-guide-data-optimization/pipeline-example--convert-200-events-to-metrics) or others to reduce log volume between the Source and the Archive Destination. Image #### Demo Logs Source Configuration | Configuration Option | Setting | | ------------------------------------------------------------------ | ----------- | | **Interval** (the number of seconds to pause between sending logs) | `1` | | **Format** | `JSON HTTP` | #### Mezmo Archive Destination Configuration | Configuration Option | Setting | | --------------------------- | -------------------------------------------------------- | | **Batch timeout (seconds)** | `300` | | **Archive Provider** | `S3` (note that you can also send archive logs to Azure) | | **Access Key ID** | Access key for the S3 bucket | | **Secret Access Key** | Secret access key for the S3 bucket | | **Bucket** | The name of the S3 bucket | | **Region** | The AWS region where the S3 bucket is located. | ### Restoration Pipeline This Pipeline sends archived data from the [Pipeline Data Restoration Source](/telemetry-pipelines/pipeline-data-restoration-source), passes it through a [Filter Processor](/telemetry-pipelines/filter-processor) to drop data and a [Map Fields Processor](/telemetry-pipelines/map-fields-processor) to make sure that the restored data conforms to the [required schema for the log analysis destination, ](/telemetry-pipelines/required-schema-for-mezmo-log-analysis-destination) and then finally sends it to Mezmo Log Analysis. Note that this Pipeline is not active after being saved and deployed. Data will only begin to stream when the Pipeline is activated during a **Restoration Task**, described in the next section. #### Mezmo Pipeline Data Restoration Source Configuration There is no configuration for the Source other than giving it a **Title**. This is how you will identify where to send the data for the restoration task. #### Filter Processor Configuration This filter is set to only send a subset of the archived data to log analysis. | Configuration Option | Setting | | ------------------------- | ---------------------------------------------------------------------- | | **Action** | Drop events matching this criteria | | **Conditional Statement** | `if (message.status greater_or_equal 200 AND message.status less 300)` | #### Map Fields Processor Configuration This processor maps fields in the restored data to fields conform to the [schema required for Mezmo Log Analysis](/telemetry-pipelines/required-schema-for-mezmo-log-analysis-destination). | Source Field | Target Field | | ------------------ | -------------- | | `message.method` | `message.line` | | `message.referrer` | `.app` | #### Mezmo Log Analysis Destination Configuration The tags you enter in the configuration options are intended to help you easily search for restored data in the Log Viewer. | Configuration Option | Setting | | ------------------------------------------------------ | ------------------------------------------------------ | | **Mezmo Host** | `logs.mezmo.com` | | **Ingestion Key** | The ingestion key for your Mezmo Log Analysis instance | | **Hostname** | `rehydrated-data` | | **Tags** (these will be attached to the restored data) | `{{metadata.query.tags}}` `restored` `restored- data` | | **Scheme** | `Message pass-through` | ## Create the Restoration Task You must have admin privileges within your Mezmo Organization to create and run a restoration task. You should create the restoration task in the same account that is associated with the restoration pipeline. 1. In the Mezmo Web App, go to **Settings > Archiving > Pipeline Restoration**. 2. Click **New Pipeline Restoration Task**. 3. Enter a name for the restoration task. 4. Enter the time period for the data you want to restore. 5. Select the **Pipeline Archive** to restore data from. 6. Select the Pipeline where you want to send the restored data. 7. Click **Start**. You will see data begin to stream into the restoration pipeline, and then in your Mezmo Log Analysis viewer. # Convert Events to Metrics Source: https://docs.mezmo.com/guide-to-pipeline-architecture/convert-events-to-metrics Reduce log volume by routing routine 200 events through Event to Metric and Aggregate Processors, with example Processor configurations. A simple way to reduce the overall volume of log data is to parse out routine messages, like `Status 200`messages, and then convert that data from events to metrics. Using this method, you can monitor these routine messages through a simple dashboard view, and then take action if you notice or are alerted to any anomalous spikes or decreases in these messages. This topic describe a basic Pipeline architecture and Processor group for converting events to metrics that you can adapt to your own purposes, with examples of Processor configurations. Overview This schematic of the Pipeline illustrates the Processor chain for converting 200 events to metrics. The Processor configurations are described in detail in the sections that match the numbers in the schematic. Image ## Interactive Demo You can see how data is processed and reduced through this Pipeline in this interactive version. View Demo ## 1 - Demo/HTTP Source Use the [HTTP Source](/telemetry-pipelines/http-source) to connect the Pipeline to your incoming telemetry data. The topic [Set Up and Test an HTTP Endpoint Source](/telemetry-pipelines/set-up-and-process-http-endpoint-data) includes tips and examples for configuring your source. This example uses the [Demo Logs](/telemetry-pipelines/demo-logs-source) with the **JSON Logs** option to demonstrate the effects of the Processors on the data stream. You can also [try it out with a sample of your own data. ](/telemetry-pipelines/view-pipeline-data) 1. Log into the Mezmo App, and in the **Pipelines** section, click **New Pipeline**. 2. Add the **Demo Logs** Source, and for **Format**, select **JSON**. 3. Add the **Blackhole** Destination to your Pipeline, and connect it to the Demo Logs. 4. Add the Processors and their configurations as shown in this example. 5. To view the data transformations through the Processors, **Deploy** the Pipeline, and then click the **Tap** for the Source and each Processor to see the data as it egresses from each node. You will also be able to see how the data is reduced on the Pipeline Dashboard. If you don't yet have a Mezmo account, you can [sign up for a 30 Day Free Trial](https://www.mezmo.com/sign-up-pipeline-today) to try us out! ## 2 - Route Processor The [Route Processor](/telemetry-pipelines/route-processor) enables you to set conditions under which telemetry data will be sent to other points in the processing chain. In this case, it is set to send 200 events down the Processor Chain for conversion to metrics, while unmatched data is sent directly to the Destination. This example uses the Blackhole destination, where all data is dropped, but you could send matched and unmatched data to different destinations depending on your use case. | Configuration Parameter | Setting | | ----------------------------------- | ----------------------- | | Conditional Statement for 200 Route | `if(.status equal 200)` | ## 3 - Event to Metric Processor The [Event to Metric Processor](/telemetry-pipelines/event-to-metric-processor) enables you implement a counter for the events sent to it, and attach tags to specified fields. In this case, the tags are sent to capture the values related to the URL and IP Address within the 200 event. | Configuration Parameter | Setting | | ------------------------- | ------------------------ | | Metric Name | `number_hits` | | Kind | `Incremental` | | Type | `Counter` | | Type/Value Type | `New Value` | | Type/Value | `1` | | Type/Namespace/Value Type | `None` | | Tag 1/Name | `url` | | Tag 1/Value Type | `Value from event field` | | Tag 1/Field Value | `.host` | ## 4 - Aggregate Metrics Processor The final Processor in the chain, the [Aggregate Processor](/telemetry-pipelines/aggregate-processor) aggregates multiple metric events into a single metric based on a defined time interval. In this case, it aggregates the value of the 200 metrics over a 10 second interval into a single number. | Configuration Parameter | Setting | | ----------------------- | ------------ | | Interval | `10 seconds` | ## 5 - Blackhole Destination The [Blackhole](/telemetry-pipelines/blackhole-destination) Destination drops all data sent to it. This makes it useful for testing your Processor chain to make sure you are getting the expected results before sending them on to a production destination. Mezmo supports a wide variety of popular destinations including [Mezmo Log Analysis](/telemetry-pipelines/mezmo-destination), [Datadog Metrics](/telemetry-pipelines/datadog-metrics-destination), and [Prometheus Remote Write](/telemetry-pipelines/prometheus-remote-write-destination). ## For More Information For more information on how to understand and optimize your telemetry data, [contact our Solutions Engineering team](https://go.mezmo.com/mezmo-data-profiling?_gl=1*189zkyo*_ga*NDQxOTc0Mzg1LjE2NDE0MTYxODc.*_ga_C3EJ23NJFV*MTcxMTU3ODkyNi45OC4xLjE3MTE1Nzg5MzIuMC4wLjA.) to schedule a free consultation. # Kafka Data Optimization Pipeline Source: https://docs.mezmo.com/guide-to-pipeline-architecture/kafka-data-optimization-pipeline Parse, route, and convert Kafka logs to metrics so routine events are summarized while errors and warnings reach storage at full fidelity. **Estimated Reading Time**: 5 minutes ## The Situation Applications today are often composed of many different components to create a "stack." Each part of the stack is important to the function of the application, but not every part behaves the same way. Open source components like Kafka can generate a substantial volume of logs. Many of the logs can provide valuable operational data in real time needed to understand the application behavior, while others are less relevant and can be sent to storage for later analysis. This Pipeline provides a model for optimizing your log data based on the most common log messages that would be generated from a Kafka cluster, but is also an example of how to create a Pipeline to optimize data from any other system that generates similar log data. This architecture includes [standard best practices](/practioner-guide-data-optimization/optimize-your-observability-data-in-six-steps) such as removing extraneous events from the stream, routing data to specific destinations based on the event type, and converting events to metrics for use in operational dashboards. For Kafka data, the log types we identified that can be easily optimized include **partition management**, **record generation**, and **deletion** information. These log types can be either summarized or rolled into metrics for monitoring without needing additional storage space. **Errors** and **Warnings** are given direct paths to storage and left untouched for full fidelity. ## Architecture Overview ### Sources #### 1 - HTTP Endpoint For this example, the [HTTP Endpoint](/telemetry-pipelines/http-destination) Source includes a data sample that represents Kafka logs. We are using this source because it can accept any log data via an HTTP post request. In practice, you may be using an Agent of some kind, but the same principles apply. ### Processors #### 2 - Parse The [Parse Processor](/telemetry-pipelines/parse-processor) uses a [Grok Pattern](/telemetry-pipelines/using-grok-to-parse) to parse the Kafka logs that the Pipeline will process and standardize their data format. Note that Mezmo has custom Grok expressions, including `%{SQUARE_BRACKET}` , which is used in this example for convenience. ```none theme={null} %{SQUARE_BRACKET}%{TIMESTAMP_ISO8601:timestamp}%{SQUARE_BRACKET} %{LOGLEVEL:level} %{GREEDYDATA:description} ``` #### 3 - Route The [Route Processor](/telemetry-pipelines/route-processor) uses conditional statements to match log data and provide flow control that separates the data for the Metrics and Log Consumer destinations. **Generating Records Route** This statement matches the terms `generating` and `generated` in the `.description` field of the data, and routes it to an [Event to Metric Processor](/telemetry-pipelines/event-to-metric-processor). ```none theme={null} if (.description contains 'generating' OR .description contains 'generated') ``` **Partition Management** This statement matches the term partition in the `.description` field of the data, and routes it to an [Event to Metric Processor](/telemetry-pipelines/event-to-metric-processor). ```none theme={null} if (.description contains 'partition') ``` **Error and Warnings** This statement matches the terms `warn` and `error` in the .`level` field of the data, and routes it directly the Log Consumer destination. ```none theme={null} if (.level equal 'warn' OR 'error') ``` Critical events, like errors and warnings, should be routed directly to storage or your log analysis system to preserve full-fidelity copies for later analysis. If the warnings are especially verbose, you could also convert them to metrics based on your specific needs. **Deleting info** This statement matches the term `deleted` in the .`description` field of the data, and routes it to the [Reduce Processor](/telemetry-pipelines/reduce-processor). ```none theme={null} if (.description contains 'deleted') ``` **Unmatched** Any data that doesn't match the conditional statements is routed directly to the Log Consumer destination. #### 4 - Event to Metric The two [Event to Metric Processor](/telemetry-pipelines/event-to-metric-processor)s are set to take the incoming log events and convert them to metrics, then sends the converted metrics to the [Aggregate Processor](/telemetry-pipelines/aggregate-processor) The configuration of these Processors represents the best practice of reducing repetitive events, like the start and stop of processes, to metrics. The valuable information in these events isn't within the single event itself, but in the total number of operations and the load they place on your systems. The same is true of positive events, like `200-OK` messages. By converting these types of events to metrics, your Pipeline can provide you with useful information while also substantially reducing the volumes of data you send to your monitoring systems. **Generating Records** This Processor creates a counter metric for each event with `generating` or `generated` in the `.description` field, and creates an incremental count starting at 1. **Partition Management** This Processor creates a counter metric for each event with `partition` in the `.description` field, and creates an incremental count starting at 1. #### 5 - Aggregate (Metric) The [Aggregate Processor](/telemetry-pipelines/aggregate-processor) converts the metric counts from the Event to Metric Processor to an aggregated metric based on a count of events over 10 second intervals. When setting time intervals for the **Aggregate** and **Reduce** Processors, you should consider how faithful you need to be to the original data to get the information you need. As a rule of thumb: **30 seconds+** for low fidelity needs, ensuring positive affirmations **10 seconds** for medium fidelity needs **1 second** for high fidelity **\< 1 second** for very high fidelity #### 6 - Reduce Similar to the Event to Metric Processors, the [Reduce Processor](/telemetry-pipelines/reduce-processor) converts the `deleted` events sent to it from the Route Processor into a single event based on an interval of 30 seconds, and appends this as an array to the .description field for consumption by the Log Consumer. ### Destinations For purposes of this example, this Pipeline terminates in two [Blackhole](/telemetry-pipelines/blackhole-destination) destinations. All data sent to a Black Hole is dropped for the purpose counting against your egress volume. This lets you construct a Pipeline and make sure that the data being sent to each destination is in the desired state before sending it to your production systems. In this example, the Blackholes represent two typical destinations for operational information, one that consumes metric data, and another that consumes log/event data. # Kubernetes Data Optimization Pipeline Source: https://docs.mezmo.com/guide-to-pipeline-architecture/kubernetes-data-optimization-pipeline Transform Kubernetes telemetry into metrics with Filter, Route, Event to Metric, and Aggregate Processors while retaining original data in storage. ## The Situation This Pipeline models a typical situation where you have telemetry data originating from a Kubernetes cluster and need to transform it into metric data for consumption by an observability tool, while also retaining a copy of the original data in storage for compliance and later analysis. By using a Pipeline to transform the data as it is streamed, you can reduce the volume of data sent to your tool, and at the same time ensure that data sent to the tool will be optimized to provide useful information. If you would like to try out this Pipeline with your own Kubernetes data, this topic includes configuration information for each Processor. You can find more detailed information about Mezmo Telemetry Pipelines in [our product guide](/telemetry-pipelines/about-mezmo-telemetry-pipelines). If you don't have a Mezmo account yet, [sign up for a free trial](https://www.mezmo.com/sign-up-pipeline-today) so you can try out our product features and start managing your telemetry data! ## Architecture Overview Image ## Sources #### 1 Splunk HEC This Pipeline uses the [Splunk HEC](/telemetry-pipelines/splunk-hec-source) Source as the ingress point for Kubernetes telemetry data, but there are also a variety of [Supported Telemetry Pipeline Sources](/telemetry-pipelines/supported-telemetry-pipeline-sources), including OTel Sources, that you can use. ## Processors ### Container Logs Processing Chain Nodes 2, 3, and 4 represent the chain for processing Kubernetes container logs. #### 2 - Filter Processor The [Filter Processor](/telemetry-pipelines/filter-processor) uses a conditional statement to identify telemetry data specifically related to containers within the Kubernetes cluster, and allows matching data to proceed to the next step of the Processor chain. ```none theme={null} if (exists(metadata.fields."k8s.container.name")) ``` #### 3 - Event to Metric Processor The [Event to Metric Processor](/telemetry-pipelines/event-to-metric-processor) converts the Kubernetes events into metrics representing log entries by node, and log entries by container. | Option | Setting | | -------------------- | -------------------------------------- | | Metric Name | l`og_entry_by_node` | | Kind | `Incremental` | | Type | `Counter` | | Value/Value Type | `New value` | | Value/Value | `1` | | Namespace/Value Type | `None` | | Tags/Name | `node_name` | | Tags/Value Type | `Value from Event Field` | | Tags/Field Value | `metadata.fields."k8snode.name".field` | #### 4 - Aggregate Metrics The [Aggregate Processor](/telemetry-pipelines/aggregate-processor) aggregates multiple metric events into a single metric event based on a defined interval window. In this case, the Processor aggregates all the metric events for the Kubernetes node logs into a single metric over a one minute interval. | Option | Setting | | ------------------------------ | ---------------------------- | | Group by Field Paths | `.name` `.namespace` `.tags` | | Evaluate/Operation | `add` | | Window Type/Type | `tumbling` | | Window Type/Interval (seconds) | `60` | | Event Timestamp | `.timestamp.field` | ### Metric Counters Processing Chain Processors 5 and 6 convert log message events of certain types to metrics and produces a count of each type. #### 5 - Route Processor The [Route Processor](/telemetry-pipelines/route-processor) uses conditional statements to match log messages related to **Errors**, **Exceptions**, and **Negative Sentiment** (Abort, Broken, Kill, etc.) and sends them to specific Event to Metric Processors. | Option | Conditional Statement | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Errors Route | `if (exists(message) AND message contains 'error')` | | Exceptions Route | `if (exists(message) AND message contains 'exception')` | | Negative Sentiment Route | `if (exists(message) AND (message contains 'abort' OR message contains 'broken' OR message contains 'caught' OR message contains 'denied' OR message contains 'exception' OR message contains 'fail' OR message contains 'insufficient' OR message contains 'killed' OR message contains 'malformed' OR message contains 'outofmemory' OR message contains 'panic' OR message contains 'timeout' OR message contains 'undefined' OR message contains 'unsuccessful' OR message contains 'unavailable'))` | #### 6 - Event to Metrics Processors Each of these processors is used to count the type of message event sent to it, and produce an incremental metric for that type. **Error Metrics** | Option | Setting | | -------------------- | -------------------------------------- | | Metric Name | `error_monitoring` | | Kind | `Incremental` | | Type | `Counter` | | Value/Value Type | `New value` | | Value/Value | `1` | | Namespace/Value Type | `None` | | Tags/Name | `container_name` | | Tags/Value Type | `Value from Event Field` | | Tags/Field Value | `metadata.fields."k8s.container.name"` | **Negative Sentiment Metrics** | Option | Setting | | -------------------- | -------------------------------------- | | Metric Name | `negative_sentiment_monitoring` | | Kind | `Incremental` | | Type | `Counter` | | Value/Value Type | `New value` | | Value/Value | `1` | | Namespace/Value Type | `None` | | Tags/Name | `container_name` | | Tags/Value Type | `Value from Event Field` | | Tags/Field Value | `metadata.fields."k8s.container.name"` | **Exceptions Metrics** | Option | Setting | | -------------------- | -------------------------------------- | | Metric Name | `exception_monitoring` | | Kind | `Incremental` | | Type | `Counter` | | Value/Value Type | `New value` | | Value/Value | `1` | | Namespace/Value Type | `None` | | Tags/Name | `container_name` | | Tags/Value Type | `Value from Event Field` | | Tags/Field Value | `metadata.fields."k8s.container.name"` | #### 7 - Enrich Ops Tags All the processed data is sent to the final Processor in the chain, the [Script Execution Processor](/telemetry-pipelines/js-script-processor), which adds descriptive information to the data to identify where and how it was processed. ```bash theme={null} // Modify the event using a subset of the JavaScript language. // The function must return the modified event function processEvent(message, metadata) { message.tags.pipeline_owner = '' message.tags.pipeline_name = '' message.tags.pipeline_url = ' Do not edit the Pipeline without consulting Mezmo Technical Services. ## Pipeline Architecture The Log Analysis Source Pipeline processes data in four steps, with each step designed to optimize your data for use with Mezmo Log Management. Image | Component | Description | Configuration | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------- | | 1. Mezmo Log Analysis Data Source | | | | 2. Route Processor | The [Route Processor](/telemetry-pipelines/route-processor) uses conditional statements to identify Kubernetes enrichment metrics, app/file metadata, and other relevant data for further processing, while unmatched data is sent directly to Log Analysis. | | | 3. Remove Fields Processor | The [Remove Fields Processor](/telemetry-pipelines/drop-fields-processor) drops the .app field when a file is also present. | | | 4. Route Processor | The [Route Processor](/telemetry-pipelines/route-processor) routes empty fields to a [Blackhole](/telemetry-pipelines/blackhole-destination) Destination, where they are dropped. | | | 5. Log Analysis Destination | The processed telemetry data is sent to Mezmo Log Analysis. | | | 6. Blackhole Destination | The [Blackhole](/telemetry-pipelines/blackhole-destination) Destination drops all data sent to it. | | Learn More About Mezmo Telemetry Pipelines Check out these links to learn more about Mezmo Telemetry Pipelines, and how you can optimize your telemetry data to reduce costs, gain insights, and quickly respond to anomalies and incidents. * [Getting Started with Mezmo Telemetry Pipelines](/telemetry-pipelines/getting-started-with-mezmo-telemetry-pipeline) * [About Mezmo Flow](/telemetry-pipelines/about-mezmo-flow) * [Supported Processors](/telemetry-pipelines/supported-processors) * [In-Stream Alerts](/telemetry-pipelines/in-stream-alerts) * [Configure Responsive Pipelines](/telemetry-pipelines/configure-responsive-pipelines) # Mask and Encrypt Data Source: https://docs.mezmo.com/guide-to-pipeline-architecture/mask-and-encrypt-data Build a data compliance processor group that routes events through Redact and Encrypt Field Processors to obfuscate PII such as user IDs and credit card numbers. ## The Situation This Pipette depicts the typical configuration of a Data Compliance processor group within a Telemetry Pipeline. In this situation, the objectives are to send financial transaction and account access data to a storage location in case it is needed for later analysis, and to redact or encrypt Personally Identifying Information (PII). This group uses the [Route Processor](/telemetry-pipelines/route-processor) to send specific events to the [Redact Processor](/telemetry-pipelines/redact-processor) and the [Encrypt Field Processor](/telemetry-pipelines/encrypt-fields-processor), before storage, to obfuscate user IDs and credit card numbers, and to also enable the decryption of credit card numbers in case they are needed for specific analysis. ## Overview This schematic of the Pipette illustrates the Processor chain for redacting and encrypting Personally Identifying Information focusing on login User IDs and credit card numbers. The Processor configurations are described in detail in the sections that match the numbers in the schematic. Image ## 1 - Demo Logs Source This Pipette uses the [Demo Logs](/telemetry-pipelines/demo-logs-source) with the **Financial Data** option to send a sample of data containing PII through the Processor chain. 1. Log into the Mezmo App, and in the **Pipelines** section, click **New Pipeline**. 2. Add the **Demo Logs** Source, and for **Format**, select **Financial Data**. 3. Add the **Blackhole** Destination to your Pipeline, and connect it to the Demo Logs. 4. Add the Processors and their configurations as shown in this example. 5. To view the data transformations through the Processors, **Deploy** the Pipeline, and then click the **Tap** for the Source and each Processor to see the data as it egresses from each node. You will also be able to see how the data is reduced on the Pipeline Dashboard. If you don't yet have a Mezmo account, you can [sign up for a 30 Day Free Trial](https://www.mezmo.com/sign-up-pipeline-today) to try us out! ## 2 - Route Processor The [Route Processor](/telemetry-pipelines/route-processor) enables you to set conditions under which telemetry data will be sent to other points in the processing chain. In this case, it filters three types of events from the incoming data for processing: Access, Transaction, and Boot-up. Any events that don't match these three types are sent directly to the storage location. | Configuration Parameter | Setting | | -------------------------------------------- | --------------------------------- | | Conditional Statement for Boot-up Events | `if (.event equal 'bootup')` | | Conditional Statement for Transaction Events | `if (.event equal 'transaction')` | | Conditional Statement for | `if (.event equal 'access')` | ## 3 - Encrypt Processor The transaction events contain credit card information that should be redacted or encrypted before being sent to storage. In this case, since the credit card numbers may be needed later, for example for fraud analysis, the [Encrypt Field Processor](/telemetry-pipelines/encrypt-fields-processor) is set to encrypt the card numbers, so that they can later be decrypted using the encryption key. | Configuration Parameter | Setting | | --------------------------------------------- | -------------------------------------------------- | | Field | `.transaction.cc.cc_number` | | Encryption algorithm | `AES-256-CFB(key=32 characters, iv=16 characters)` | | Encryption key | `zipadeedoodah777zipadeedoodah888` | | Initialization vector (IV) field | `.creditcardnumber` | | Encode encrypted field and IV as Base 64 text | On | ## 4 - Redact Processor Information that is redacted is obfuscated completely, and cannot be recovered after processing. For this reason, the [Redact Processor](/telemetry-pipelines/redact-processor) should be used to remove PII that is particularly sensitive, but doesn't need to be used for later analysis. In this case, the login User ID from Access events is redacted, since this is information that could be used to hack user accounts, but isn't needed for analysis. The Processor operation is based on searching for specific patterns, such as social security numbers or email addresses, or custom patterns, and then using a hash or replacement pattern to obfuscate the data. In this case, it searches the field`.access.user_id` for a custom pattern based on a regular expression, and then hashes it using the md5 algorithm. | Configuration Parameter | Setting | | ----------------------- | ----------------- | | Field | `.access.user_id` | | Redact Pattern | `Custom Pattern` | | Action | `Hash` | | Algorithm | `md5` | | Expression | `[a-zA-Z0-9@.]+` | ## 5 - Blackhole Destination The [Blackhole](/telemetry-pipelines/blackhole-destination) Destination drops all data sent to it. This makes it useful for testing your Processor chain to make sure you are getting the expected results before sending them on to a production destination. Mezmo supports a wide variety of popular destinations including [Mezmo Log Analysis](/telemetry-pipelines/mezmo-destination), [Datadog Metrics](/telemetry-pipelines/datadog-metrics-destination), and [Prometheus Remote Write](/telemetry-pipelines/prometheus-remote-write-destination). In this case, note that the data volume from the Source to the Destination has increased by almost 22%. It's typical for data volume to increase with these Processors because they add characters to the message strings. However, fine tuning of the algorithms and encryption keys can limit the increase in data volume. ## For More Information For more information on how to implement security modules for your Pipeline data management needs, [contact our Solutions Engineering team](https://go.mezmo.com/mezmo-data-profiling?_gl=1*189zkyo*_ga*NDQxOTc0Mzg1LjE2NDE0MTYxODc.*_ga_C3EJ23NJFV*MTcxMTU3ODkyNi45OC4xLjE3MTE1Nzg5MzIuMC4wLjA.) to schedule a free consultation. # Responsive OTel Pipeline Workshop Source: https://docs.mezmo.com/guide-to-pipeline-architecture/responsive-otel-pipeline-tutorial Hands-on workshop for building responsive OpenTelemetry pipelines that route full-fidelity data during incidents and optimize for cost during normal operations. *** **Time to complete: 30 minutes** In this workshop, you will gain control over your OpenTelemetry logs, metrics, and traces. Learn how Mezmo’s platform helps you build dynamic, "responsive" pipelines that adapt to various conditions by routing comprehensive data during incidents while optimizing for cost during normal operations. Ensuring your data is in the right place precisely when you need it, reducing MTTR, saving on observability spend, and empowering your teams with intelligent telemetry routing. Get hands-on experience directing and controlling OpenTelemetry data across your observability systems. If you run into any issues or have feedback on either the workshop or Pipeline, please reach out to us at [support@mezmo.com](mailto:support@mezmo.com). # Route Data Source: https://docs.mezmo.com/guide-to-pipeline-architecture/route-data Use a Route Processor with conditional statements to direct financial, JSON, and Apache data from multiple sources through specialized processing chains to different destinations. In more complex architectures, you will often have several Sources feeding into the same Pipeline, with the data for each needing different types of processing before being sent to multiple destinations. A key component of these Pipelines is a Route Processor, which uses conditional statements to match data and send it along its particular processing route. This topic describes a typical use of a Route Processor, with examples of the Processor configurations. ## Overview This schematic illustrates the configuration of a Routing group, which includes a Script Execution Processor to format raw strings to JSON, that routes different data types from several sources through specialized processing chains to several destinations. Image ## 1 - Sources The Sources represent three different types of data flowing through the Pipeline that need to be routed to separate processing chains: 1. **Financial Data** that needs to have Personally Identifying Information encrypted before being sent to storage and the observability tool. 2. **JSON Data** that needs to have Status - 200 **events** routed and dropped. 3. **Apache Errors** that need to be converted to JSON format and all info messages dropped. 1) Log into the Mezmo App, and in the **Pipelines** section, click **New Pipeline**. 2) Add three **Demo Logs** Sources, and for **Format**, select 1) **Financial Data** 2) **JSON** 3) **Apache Errors**. 3) Add three **Blackhole** Destinations to your Pipeline to represent 1) **Drop** 2) **Storage** 3) **Observability Tool**. 4) Add the Processors and their configurations as shown in this example. 5) To view the data transformations through the Processors, **Deploy** the Pipeline, and then click the **Tap** for the Source and each Processor to see the data as it egresses from each node. If you don't yet have a Mezmo account, you can [sign up for a 30 Day Free Trial](https://www.mezmo.com/sign-up-pipeline-today) to try us out! ## 2 - Script Execution Processor The [Script Execution Processor](/telemetry-pipelines/js-script-processor) is configured to convert the Apache errors from raw strings to JSON format. ```bash theme={null} function junk(message) { var new_message = {} new_message.message = message return new_message } ``` ## 3 - Route Processor The [Route Processor](/telemetry-pipelines/route-processor) uses three conditional statements to identify and route specific components of all three data types: #### Apache Info Messages ```javascript theme={null} if (exists(.message) AND .message contains 'INFO') ``` Because both the JSON and Apache errors data contain .message fields, this statement uses AND to make sure that .messages that don't contain the INFO event won't generate a "field not found" error. All messages meeting this criteria are sent to the Drop Destination. #### Status 200 Events ```javascript theme={null} if (exists(.status) AND .status equal 200) ``` All events that meet these criteria are sent to the Drop Destination. #### Transaction Events ```bash theme={null} if (exists(.event) AND .event equal 'transaction') ``` All events that meet these criteria are send to the Encrypt Processor. ## 4 - Encrypt Processor Because transaction events contain Personally Identifying Information (PII), such as credit card numbers, this information needs to be encrypted before being sent to storage and observability tools. For more information, check out the topic [Tutorial: Mask and Encrypt Data](/practioner-guide-data-optimization/pipeline-module--security-and-compliance). **Encrypt Processor Configuration** | Configuration Field | Details | | -------------------------------- | ------------------------------------------------- | | Field | .transaction.cc.cc\_number | | Encryption algorithm | AES-256-CFB (key=32 characters, iv-16 characters) | | Encryption key | zipadeedoodah012zipadeedoodah013 | | Initialization vector (IV) field | .creditcardnumber | ## 5 - Destinations The routed data is sent to three destination, represented in this schematic by the [Black Hole Destination](/telemetry-pipelines/blackhole-destination): 1. **Drop**, where the unnecessary INFO and Status - 200 messages are sent. 2. **Storage**, where all unmatched data and encrypted PII data is sent. 3. **Observability Tool,** where all unmatched data and encrypted PII data is sent. The [Blackhole](/telemetry-pipelines/blackhole-destination) Destination drops all data sent to it. This makes it useful for testing your Processor chain to make sure you are getting the expected results before sending them on to a production Destination. Mezmo supports a wide variety of popular Destinations including [Mezmo Log Analysis](/telemetry-pipelines/mezmo-destination), [Datadog Metrics](/telemetry-pipelines/datadog-metrics-destination), and [Prometheus Remote Write](/telemetry-pipelines/prometheus-remote-write-destination). # Types of Telemetry Data Pipelines Source: https://docs.mezmo.com/guide-to-pipeline-architecture/types-of-telemetry-data-pipelines Compare Mezmo pipeline patterns for ingestion, optimization, archiving and rehydration, and responsive processing, with example pipelines for each. In the Mezmo O'Reilly Report [The Fundamentals of Telemetry Pipelines](https://www.mezmo.com/resources/oreilly-report-the-fundamentals-of-telemetry-pipelines), telemetry data is described as a raw resource that must be refined in order to become useful information. The process of refinement is carried out through a telemetry pipeline. There is, however, no one-size-fits-all approach to telemetry pipeline design - you must design the pipeline to produce the type of information that suits your purpose. While the word "pipeline" brings to mind images of pipes, valves, and other plumbing fixtures, a data pipeline is better thought of as an algorithm - a series of operations executed in a specific order to produce a result. Within a telemetry data pipeline, the operations are represented by processors or processor groups that perform specific functions. In this guide you'll find examples of Mezmo Telemetry Pipelines that are designed for specific purposes, along with descriptions of the processors typically used in each type of pipeline. You will also find tutorials for building "Pipettes" using [Mezmo Demo Source Data](/telemetry-pipelines/demo-logs-source), and interactive demos to help you understand how data is transformed into information as it passes through the pipeline. ## Data Ingestion Pipelines Data ingestion pipelines are designed to send log data to Mezmo Log Analysis. | | | | -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Basic Log Analysis Pipeline](/guide-to-pipeline-architecture/basic-log-analysis-pipeline) | This pipeline is designed to optimize data before it is sent to Mezmo Log Analysis. In this case, you would tailor the optimization processes to your specific data type, as shown in the **Data Optimization Pipelines** section. The tutorial shows you how to send source data directly to Mezmo Log Analysis, and then view the live tail of that data. | | [Mezmo Log Analysis Source Pipeline](/guide-to-pipeline-architecture/log-analysis-source-pipeline) | This pipeline was originally designed to provide users of the Mezmo Log Analysis product with a migration path to the Mezmo Platform. | ## Data Optimization Pipelines Data optimization pipelines are designed to optimize specific types of data before sending it to observability tools and storage. These pipelines typically use processors like [Filter](/telemetry-pipelines/filter-processor), [Route](/telemetry-pipelines/route-processor), [Event to Metric](/telemetry-pipelines/event-to-metric-processor), and [Aggregate](/telemetry-pipelines/aggregate-processor) to transform the data into the format required for the destinations. The [Mezmo Data Profiler](/telemetry-pipelines/data-profiler-processor) can help you [understand your data](/guide-to-pipeline-architecture/understanding-your-data-to-optimize-it) and provide recommendations for how to optimize it. | Example Pipelines | Description | | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | | [Basic Data Optimization Pipeline](/guide-to-pipeline-architecture/basic-data-optimization-pipeline) | A basic pipeline to demonstrate the typical data optimization operations. | | [Kafka Data Optimization Pipeline](/guide-to-pipeline-architecture/kafka-data-optimization-pipeline) | A pipeline designed to optimize Kafka data. | | [Kubernetes Data Optimization Pipeline](/guide-to-pipeline-architecture/kubernetes-data-optimization-pipeline) | A pipeline designed to optimize Kubernetes data. | ## Data Archiving and Rehydration Pipelines Data archiving and rehydration pipelines are designed to optimize data for storage by reducing its volume, but then being able to restore or "rehydrate" it as needed for incident or other investigations. The archiving pipeline will typically include elements of a data optimization pipeline, as well as processors to [Mask and Encrypt Data](/guide-to-pipeline-architecture/mask-and-encrypt-data) in situations in which the data potentially includes Personally Identifying Information (PII). The rehydration pipeline will typically include processors like [Filter](/telemetry-pipelines/filter-processor) and [Map Fields](/telemetry-pipelines/map-fields-processor), to make sure the data is in the correct format for your log analysis tool. | Example Pipelines | Description | | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | [Basic Data Rehydration Pipeline](/guide-to-pipeline-architecture/basic-data-rehydration-pipeline) | A basic set of archive and rehydration pipelines to show typical components, and how to create a restoration task. | ## Responsive Pipelines [Responsive Pipelines](/telemetry-pipelines/configure-responsive-pipelines) are designed to change the pipeline's data processing operations when a defined condition is detected in the data, for example a surge in data from a particular source, or the detection of PII data. This enables you to preserve full-fidelity copies of your data during an incident, for example, and is intended to help you reduce Mean Time to Resolution (MTTR). | Example Pipelines | Description | | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------- | | [Responsive Otel Pipeline Workshop](/guide-to-pipeline-architecture/responsive-otel-pipeline-tutorial) | Workshop presented by Mezmo's Braxton Johnston at PlatformCon in July 2025. | # Understanding Your Data to Optimize It Source: https://docs.mezmo.com/guide-to-pipeline-architecture/understanding-your-data-to-optimize-it Use the Mezmo Data Profiler to analyze streaming telemetry through Message Templates, Field Summaries, and Log Metrics before deciding how to optimize it. Before you can optimize your data, you need to understand what it contains, and separate the useful data from the noise. Mezmo's Data Profiler feature enables you to get a detailed, granular view of the most common messages in your log data, and add processor components to handle specific message types. ## Create a Data Profile There are two ways to generate a data profile: 1. As part of the [Mezmo Flow](/telemetry-pipelines/about-mezmo-flow) onboarding process or the Log Volume Reduction pipeline creation process. 2. Through the [Data Profiler Processor](/telemetry-pipelines/data-profiler-processor), which you can set it up as a component within a Pipeline that you build yourself. ## View the Data Profile Once a Data Profile has been generated for the Source, you can access it through both the Processor itself, and the navigation in the Mezmo Web App. Image The Data Profiler analyzes streaming telemetry data using multiple techniques so that you can have improved insights based on the type of telemetry data . Analysis of the telemetry is organized in three different tabs in the report: **Message Templates,** **Field Summaries**, and **Log Metrics.** **Message Templates** provide information on unstructured text messages, where **Field Summaries** provide better insights into structured data such as JSON logs. ### Message Templates **The Message Templates** section provides a report of log patterns discovered in the data. This enables you to understand how much specific log patterns contribute to the overall source volume, expressed as a percentage of the total data volume. With this information, you can determine if the logs matching the pattern are important for investigation and troubleshooting , or if they are low-value logs that can be archived and don't need to be sent to your observability platform. The columns in the Message Templates section, which are all sortable, include: * **Apps,** which shows the number of apps that produced these log patterns. You can hover on the number to see the name of the app(s) that produced the log pattern. * **Template** shows the tokenized log lines with variables that change from message to message, such as IP or Host, and are replaced by`<*>` * **Total Lines** shows the number of log lines that match this pattern * **Total Line Size** shows the sum of all the log lines that match this pattern You can see examples for each matching log sample by clicking the arrow next for each of the message templates. Image ### Field Summaries This is an experimental feature that is still in development. For access to this feature, reach out to your Customer Support Manager or [support@mezmo.com](mailto:support@mezmo.com). Field summaries provide the analysis of telemetry data from the perspective of field values. The Field Summaries section provides a tabular view of all the Fields discovered from the events that are streaming through the pipeline during a profiler run. Click the arrow next to the field name to see the unique values associated with the field. You can also apply Processors to those unique values. Within the Field Summaries report, you will see: | Report Column | Description | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Field Name** | The name of the field. | | **Unique Values** | Represents the count of the unique values found in all the logs during the profiling run. This indicates the cardinality of the field. Some fields can have a large number of unique values, however, the report will only display the first 500 unique values. The value displayed depends on the type of field, as described in the next table. | | **Total Lines** | Similar to the message templates, this column shows:

The number of log events that contain this field.

The percentage of logs that contain this field. The Percentage is calculated based on the total volume of data processed by the profiler during that specific run. | | **Total Size** | Similar to the message templates, this column shows:

**S**ize of all the events containing this field.

The number of events containing this field. | | **Field Size** | This represents the volume contributed by the **field itself** as Bytes and % of the total volume. It includes the field name and value. Using this information, you can decide to drop a field if it contains a large amount of data that is not important. | You can also apply Processors based on **Field** or **Field Value**. At the **Field** level, you can only select Remove Field. At the **Field Value** level, you can apply Processors such as [Filter](/telemetry-pipelines/filter-processor), [Sample Processor](/telemetry-pipelines/sample-processor), [Dedupe Processor](/telemetry-pipelines/dedupe-processor), and [Throttle Processor](/telemetry-pipelines/throttle-processor). If you remove a field at the Field level, the Field Value processors are disabled because they are mutually exclusive. This table describes the value displayed based on the value type: | Value Type | Displayed Value | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Boolean** | The value itself. | | **String** | The value itself, up to the first 50 characters. | | **Array** | The length of the arrays found. For example, `[a, b, c]` is displayed as **3**. | | **Object** | Each unique value is the set of names of the keys. For example,`{a:1, b:2}` is displayed as **a, b**. | | **Float** | Displays the min/max/average, no unique values. | | **Timestamp** | Displays the min/max (no average), no unique values | | **Integer** | If cardinality is > the threshold (for example, 65), displays min/max/average. If cardinality is \< threshold, displays the unique values. | Image ### Log Metrics The **Log Metrics** section provides a report of the profiled logs categorized by **App**, **Host**, **Log Level** and **Log Type**: Image ### # Welcome Source: https://docs.mezmo.com/index Build, optimize, and act on telemetry data with Mezmo.
# Build, optimize, and act on telemetry data with Mezmo.

Start with log management, design Telemetry Pipelines, deploy Mezmo Edge, or jump into the API references for automation.

Quick Start

Follow the shortest path from account setup to your first logs or pipeline API request.

Sign up for Mezmo, create an organization, and review the basics in the Log Management quick start. Use the ingestion guide to pick an agent, integration, API, or pipeline source for your data. Create an access key and send it with the `Authorization` header. ```bash theme={null} export MZM_ACCESS_KEY="sta_1a2b3c4d5e6f7890abcdef1234567890abcdef12" curl -H "Authorization: Token $MZM_ACCESS_KEY" \ -H "Accept: application/json" \ https://api.mezmo.com/v3/pipeline ``` Explore the pipeline architecture guide or the optimization tutorial when you are ready to optimize volume, route data, or deploy repeatable patterns.

Documentation

Browse product guides by the work you are doing: collect, process, deploy locally, or optimize your telemetry data.

Collect logs, parse fields, search events, create views, and alert your team when something changes. Build pipelines that receive, process, route, and deliver telemetry data to the destinations you choose. Run local data pipelines close to your sources for on-premises and edge collection patterns. Walk through source setup, profiling, and optimization for logs, metrics, and traces.

API Reference

Authenticate once, then use the API references and quickstarts to script common Mezmo workflows.

Learn token formats, delegated account context, and request headers. Query and manage Log Analysis resources through the API reference. Create pipelines, add components, publish revisions, and automate pipeline operations.

Resources

Use these references when you are planning architectures or tuning pipelines.

Compare common telemetry pipeline patterns for ingestion, optimization, routing, and rehydration. Learn practical techniques for reducing telemetry volume while preserving useful signal.
## Ready to send data to Mezmo?

Start with log ingestion, then add pipelines and APIs as your telemetry workflows grow.

View Ingestion Options
# HTTP Source for Mezmo Edge Source: https://docs.mezmo.com/mezmo-edge/http-source-for-mezmo-edge Configure any source to send data to a Mezmo Edge Pipeline via a RESTful POST over an allocated HTTP port. ## Description You can configure any source to send data via a RESTful POST to a Mezmo Edge Pipeline. When using the HTTP source, your content must be encoded appropriately, and packaged in a way that enables it to be parsed after ingestion. Structured formats, such as JSON, do not require additional parsing unless you want to further parse a specific value within the JSON. You would typically use an HTTP request as a Source when the type of Source you want to send data from is not supported. For example, you may want to send and process data from an uncommon open source application. As long as you're able to use a RESTful POST transport to send the data to an endpoint, you can send the data into the Pipeline. ## Configuration This Source requires an allocated port that you will forward the data to. You must use a port that has been configured within your Edge instance during set up. The default Edge port range is 8000-8010, unless you modified it during set up. ### Configuration Options | **Setting** | **Description** | | --------------- | ------------------------------------------------------ | | **Title** | A name for your source. | | **Description** | A short description of the source. | | **Port** | The port number to listen on within the Edge instance. | ## Exposing for External Ingress It is highly recommended that a scalable load balancer be used like [Nginx](https://nginx.org). The following is a simple example using the default LoadBalancer for your system which ignores TLS and the like. A simple, non-prod method of exposing the HTTP Source to external ingress is to use the system default LoadBalancer. To do this we will create a load balancer service which exposes the port configured by the HTTP Source. Sample Config: edge-load-balancer.yaml ```yaml theme={null} apiVersion: v1 kind: Service metadata: name: edge-load-balancer spec: ports: - name: http-source-1a port: <> protocol: TCP targetPort: <> selector: app.kubernetes.io/instance: <> app.kubernetes.io/name: <> sessionAffinity: None type: LoadBalancer ``` Replace `HTTP_SOURCE_PORT` with your configured port and `EDGE_INSTANCE` + `EDGE_NAME` with those found in your edge services description. You can find your `instance` and `name` via a command like ```bash theme={null} kubectl get service EDGE_SERVICE_NAME -o yaml > tmp.yaml ``` Finally, apply to your cluster with the following command ```bash theme={null} kubectl apply -f edge-load-balancer.yaml ``` You can now send data to the HTTP Source by routing data to `http://EXTERNAL_IP:HTTP_SOURCE``_PORT`. # Mezmo Edge Deployment Planning Source: https://docs.mezmo.com/mezmo-edge/mezmo-edge-deployment-planning Sizing, scaling, durability, and performance guidance for planning a Mezmo Edge telemetry pipeline deployment in your own environment. Mezmo Telemetry Pipeline is built as a cloud-native SaaS application. For cases where processing data within your environment is a requirement, we created Mezmo Edge. Mezmo Edge lets you run a telemetry data pipeline with the same functionality available in Mezmo Cloud, but locally hosted within your own environment. All of the metrics and management of the pipelines on your Edge instance are still handled by the Cloud infrastructure. Mezmo Edge can be deployed to any Kubernetes cluster using a Helm chart, as described in the topic [Set Up Mezmo Edge in Kubernetes](/mezmo-edge/set-up-mezmo-edge-in-kubernetes). ## Key Deployment Considerations 1. **Data ingestion**: how much data are you planning to send through Edge? 2. **Processing requirements**: are you planning to do any special processing, such as regex, deduplication, or other custom scripts? 3. **Destination targets**: where are you planning to send the data? ## Deployment Model Mezmo Edge uses a hybrid cloud deployment model. When the Edge satellite nodes are deployed they automatically contact the Mezmo Cloud APIs. For this reason, Edge currently requires access to the wide area network (WAN) to function. ## Resource Requirements These numbers are intended for general guidance. Resource requirements depend on many factors. Use the deployment considerations in combination with the sizing guidance to estimate actual sizing. ### Sizing These numbers are effective averages to use in approximation. Your individual event sizes may. Estimations in events per second (EPS) are conservative. These specifications are based on the guidelines published in[ the Vector documentation.](https://vector.dev/docs/setup/going-to-prod/sizing/#estimations) Assumptions 1. Each vCPU is a standard ARM processor without hyper-threading 2. Throughputs are purposefully conservative | Event Type | Typical Use Case | Typical Event Size | Expected Throughput | Event Throughput | | ---------------- | ----------------------------------------------------------- | ------------------ | ------------------- | ---------------- | | Unstructured Log | Parsing, processing, and routing to an external destination | 256 bytes | \~10 MiB/s/vCPU | \~40k EPS/vCPU | | Structured Log | Processing and routing to an external destination | 1 kilobyte | \~25 MiB/s/vCPU | \~25k EPS/vCPU | | Metric | Aggregation and routing to an external destination | 256 bytes | \~25 MiB/s/vCPU | \~100k EPS/vCPU | ## Recommendations Recommendations are divided based on small, medium, and large size deployments. For most use cases, a medium size deployment is sufficient. | Deployment Size | Typical Use Case | Expected Event Throughput | Guidance Specifications | | --------------- | ----------------------------------------------------------------------------------------------------- | ------------------------------------- | --------------------------------- | | Small | Parsing and routing to an external destination | 40-80k EPS

\<1 TB/day | 2 vCPUs

4GB of memory | | Medium | Parsing common unstructured log types, moderate processing, sending to multiple external destinations | 100k-200k EPS

1 - 5 TB/day | 4 vCPUs

8GB of memory | | Large | PII redaction, parsing with regex, processing, and sending to multiple external destinations | 200k-400k EPS

5 TB/day + | 8 vCPUs

16GB of memory | ### Disk Space Disk buffering is not enabled by default within Edge. Disk space is not a bottleneck in terms of performance. High performance disks or over provisioned disk space will only add to cost, with no performance benefit. When enabled, disk buffering can help ensure data receive and/or read in a deployed pipeline isn't lost due to a system restart. It also allows for tracking the last read position on file pointers for sources like the Kubernetes Logs source. ### High Availability High availability can be achieved by using a separate hot failover instance. **We do not recommend this configuration**. If you need to have high availability, we recommend using our Cloud infrastructure. ## Performance Considerations ### Parsing Parsing well structured logs, such as JSON, has a minimal impact on overall utilization. Parsing unstructured logs of known formats into structured logs does utilize CPU resources, though not heavily. Regex or grok parsing can require increased resource utilization, especially when event sizes are larger than average. If you plan on doing heavy parsing, consider targeting a higher deployment size. ### Processing Most transformations are highly efficient and cause minimal CPU load. Regex routing can require significantly increased CPU resources, especially if the event sizes are larger than average. For deduplication, reduction, and aggregation, the buffer size necessitates dedicated memory in order to process. Make sure to err on the conservative side for memory allocation if you’re planning to include these processors. ### Destinations As you add more destinations, the buffering required for sending data to each one increases the total memory load. The I/O operations also utilize CPU to send the packets. If you plan to add many destinations, or duplicate data to multiple destinations, keep in mind that the resource requirements may increase, primarily in terms of memory. ## Auto-scaling ### Vertical Scaling Normal recommendations are to allow for vertical scaling of your Edge instance. This ensures that your data stream remains unaltered. No changes to configuration are needed for this to take effect. ### Horizontal Scaling In default deployments (Kubernetes StatefulSets), there is an option to turn on horizontal pod scaling within the helm configuration. The standard Kubernetes auto-scaling will divide up the workload across the pods, typically allocating in a round robin fashion, depending on how you’ve configured your cluster. However, note that as you add pods and split the workload, certain features like Reduce, Aggregate, and Dedupe may be affected due to the splitting of the event stream across the nodes. ## Durability Durability within Edge is currently limited to end-to-end acknowledgement of events reaching their destination. By default, acknowledgements are on for destinations that support it. If durability is a requirement for your data, you can send the events to the Mezmo Cloud environment, which has built-in durability by default, and then onto the end destination. Mezmo does not collect any of the event data passing through the pipelines. If you use the [Tap](/telemetry-pipelines/simulate-pipeline-data-flows) feature, we will open a tap into the data stream and route that data through our cloud environment. We do not save any of the tapped data. Tapped data persists in your browser cache for a limited period of time. Mezmo does collect metrics regarding the total data throughput and total events passing through every Edge instance. We collect this information for billing purposes as well as to provide monitoring and troubleshooting. **If you block the metrics we collect, your Edge instance may cease to function and process data.** ## Sensitive Data If you are passing sensitive data through your pipeline, keep in mind that while we do not receive this data, tapping a Pipeline with this data could result in sensitive data passing through the Mezmo Cloud and appearing in the browser. If you want to send data from an Edge Pipeline to a Cloud Pipeline, you can do so with our [HTTP](/telemetry-pipelines/http-source) Processor as a destination and source respectively. Note that the metadata, such as any query parameters or headers from your initial Edge ingestion, must be persisted into the payload if you want to keep that intact for later use in the Cloud Pipeline. # Mezmo Edge Pipelines for Local Data Source: https://docs.mezmo.com/mezmo-edge/mezmo-edge-pipelines-for-local-data Run telemetry pipelines locally with Mezmo Edge to access local sources, process data within your network, and use Rust-based regex. Mezmo Edge lets you run a telemetry data pipeline locally in your environment when it's preferable to not egress data to our SaaS solution. You can run any Pipeline as a satellite node within an Edge instance. All of the metrics and management of the Pipeline are still handled by the SaaS infrastructure, making it easy to build, test, and deploy without requiring any additional coding or configuration management. Mezmo Edge can be deployed to any Kubernetes cluster using [a Helm chart](https://helm.sh/docs/intro/quickstart/), as explained in the topic [Set Up Mezmo Edge in Kubernetes](/mezmo-edge/set-up-mezmo-edge-in-kubernetes). By default, Mezmo Edge uses **Horizontal Pod Auto-scaling (HPA)** to scale to your workloads, which means you can take advantage of the scalability Kubernetes offers without having to invest substantial time and effort to manage the scaling. The Helm chart also supports deploying via a **DaemonSet**, which will run on all of the deployed nodes. This comes in handy when using the Kubernetes Pod Logs source. ## What's New in Mezmo Edge Because Mezmo Edge is entirely contained and running within your infrastructure, if offers several capabilities for managing and processing your data. These include: * Accessing local data sources without any changes to your networking, such as punching holes in firewalls * Processing and sending data entirely within your local network * Sending raw syslog directly to your pipelines without needing any additional security plugins * Leveraging the full power of regular expressions (regex) within your pipeline The regex syntax for Mezmo Edge is based on the Rust implementation of RegEx. This regex syntax is similar to other regex engines, but it lacks several features that cannot be efficiently implemented. This includes look-around and back references. However, all Rust regex searches have worst case `O(m * n)` time complexity, where `m` is proportional to the size of the regex and `n` is proportional to the size of the string being searched. You can find more information [in the Rust documentation](https://docs.rs/regex/latest/regex/). # Set Up Mezmo Edge in a Docker Container Source: https://docs.mezmo.com/mezmo-edge/set-up-mezmo-edge-in-a-docker-container Run a single Mezmo Edge instance as a Docker container using Docker Compose, including the compose and processor config files. ## Getting Started You can run Edge as a single Docker container within a suitable environment. In this deployment model, there is no default scaling other than the vertical scaling from the container resource allocation. You should be aware of this when planning your deployment resource allocation. ## Environment You must have access to a Docker environment with sufficient privileges ### Set Environmental Variables Set these default variables, including your Pipeline Service Key. If you need a service key for your Edge instance, in your account go to **Settings** **> Organization > API keys** and find the "Service Keys" section at the bottom of the page. ```bash theme={null} export EDGE_ID= export MEZMO_API_URI=https://api.mezmo.com/v3/pipeline/account/local-deploy export MEZMO_PIPELINE_SERVICE_KEY='your_key_here' ``` ### Add Config Files Add these two `.yaml`files to your working directory. #### compose.yaml ```yaml theme={null} services: edge: image: 'mezmo/vector:8.0.1' environment: MEZMO_LOCAL_DEPLOY_AUTH_TOKEN: ${MEZMO_PIPELINE_SERVICE_KEY} MEZMO_API_URI: ${MEZMO_API_URI} MEZMO_EDGE_ID: ${EDGE_ID} MEZMO_METRICS_ENDPOINT_URL: ${MEZMO_API_URI}/metric/usage?edge_id=${EDGE_ID} MEZMO_TASKS_FETCH_ENDPOINT_URL: ${MEZMO_API_URI}/tasks?edge_id=${EDGE_ID} MEZMO_TASKS_POST_ENDPOINT_URL: ${MEZMO_API_URI}/tasks/:task_id/results MEZMO_RESHAPE_MESSAGE: 1 ports: # these ports are designed to receive traffic from server-listener sources # and are set up in the Mezmo Edge pipeline configuration UI # EX: Fluent, Syslog, OpenTelemetry, Datadog Agent - "8000-8010:8000-8010" # specify the port allocation range volumes: - ${PWD}/processor.yaml:/etc/vector/processor.yaml # change this to a host mount to test specific volumes for disk buffering - ${PWD}/tmp-data:/data/vector command: ["--config-dir", "/etc/vector"] ``` #### processor.yaml ```yaml theme={null} provider: type: http url: ${MEZMO_API_URI}/config poll_interval_secs: 15 request: headers: authorization: "Token ${MEZMO_LOCAL_DEPLOY_AUTH_TOKEN}" payload: | { "deployment_group": "", "edge_id": "${MEZMO_EDGE_ID}", "name": "edge", "namespace": "default", "ports": [8000, 8001, 8002, 8003, 8004, 8005, 8006, 8007, 8008, 8009, 8010], "replica": "0", "version": "edge-8.0.1" } ``` ### Run Docker Compose ```bash theme={null} docker compose up -d ``` Start the Edge instance with this command: ```bash theme={null} docker ps ``` # Set Up Mezmo Edge in Kubernetes Source: https://docs.mezmo.com/mezmo-edge/set-up-mezmo-edge-in-kubernetes Install Mezmo Edge in a Kubernetes cluster with the Mezmo Helm chart, then create, deploy, and monitor an Edge Pipeline. Mezmo Edge is designed to run from within a Kubernetes cluster, but can also be alternatively be run within Docker Desktop. *For more details about configuration options, check out our [helm chart repo on GitHub](https://github.com/mezmo/helm-charts/blob/main/charts/edge/README.md)* ## Requirements To set up a Mezmo Edge Pipeline, you will need: 1. A Kubernetes cluster with WAN access. 2. (Recommended) Support for running Helm charts within the target cluster. Helm charts automate the deployment of Mezmo Edge for simplified set up, but you can also choose to directly pull the Edge image and deploy it with other methods, such as `kubectl`. You can run Mezmo Edge using any existing Kubernetes cluster. If you do not have easy access to a cluster, you can alternatively try it out using the Docker Desktop app with Kubernetes enabled. Go to **Settings -> Kubernetes -> Enable Kubernetes** in your Docker Desktop app to install the necessary dependencies. ## Install Mezmo Edge ### Add the Mezmo Helm Repository Make sure the Mezmo repository is included to install the requirements. ```bash theme={null} helm repo add mezmo https://helm.mezmo.com helm repo update ``` ### Define the Ports to Use Each unique source you intend to send to Edge will need a single port. You can manually define which ports, but if not defined the default port range will be used. The default Edge port range is 8000-8010. ### Run the Helm Chart Install Command Run this Helm command to launch the Edge instance in your local cluster, replacing the port range your own preferred values. These ports will be used to expose ingestion connections for your local sources. If you need a service key for your Edge instance, in your account go to **Settings** **> Organization > API keys** and find the "Service Key" generation section of the page. ```bash theme={null} helm install edge mezmo/edge \ --set mezmoApiAccessToken=MY_PIPELINE_SERVICE_KEY ``` ## Create a Mezmo Edge Pipeline Once Mezmo Edge has been installed and the ports are configured, you can create an Edge pipeline. 1. Log in to [the Mezmo Web App](https://app.mezmo.com/). 2. Go to **Pipelines** and click **New Pipeline**. 3. You will see an option to create either a SaaS or Edge Pipeline. Select **Edge**. 4. You will see the same Pipeline Map that you would use to build a SaaS Pipeline, but with additional options for Sources, and Rust-based regex syntax for the [Parse](/telemetry-pipelines/parse-processor), [Route](/telemetry-pipelines/route-processor), and [Filter Processor](/telemetry-pipelines/filter-processor)s. 5. [Build and deploy ](/telemetry-pipelines/build-deploy-mezmo-pipeline)your Pipeline as you would for a SaaS Pipeline, and you can also use [Simulation Mode](/telemetry-pipelines/simulate-pipeline-data-flows) to test your Pipeline before deploying it. 6. Deploying the Pipeline will generate the Pipeline configuration, which will be pulled down by the satellite Edge instance and become a part of the Edge configuration. This can take up to 15 seconds to complete. 7. You can now send data to your Edge Pipeline through your local Kubernetes cluster service endpoint with this prefix and the value for the source port you set up in the previous step. ```none theme={null} edge.default.svc.cluster.local: ``` ## Troubleshooting and Monitoring If you have any issues with this endpoint, please review your Kubernetes network settings to make sure you have set the appropriate forwarding. You may need to check your services via `kubectl get services` to make sure the right service name is set. If you are sending any sensitive data through your Edge Pipeline, it will be visible during a Pipeline Tap if you are inspecting nodes that do not have this data redacted. We recommend you test without sensitive data first, redact that data in your processing using the [Encrypt Field Processor](/telemetry-pipelines/encrypt-fields-processor), and then use the Pipeline Tap p after it has been redacted. You can monitor the data flow in your deployed Pipeline with [a Pipeline Tap](/mezmo-edge/tap-and-view-mezmo-edge-pipeline-data-on-premises) and make any changes to the Pipeline architecture by clicking **Edit Pipeline**, and then deploying the edited Pipeline again. # Syslog Source for Mezmo Edge Source: https://docs.mezmo.com/mezmo-edge/syslog-source-for-mezmo-edge Send syslog events to a Mezmo Edge Pipeline through a specified port using the Syslog over HTTP source within your own network. ## Description You can send syslog events and data to Mezmo Edge Pipelines directly through a specified port on the Edge instance. ## Configuration Add the [Syslog over HTTP](/telemetry-pipelines/syslog-over-http-pipeline-source) Source to your Edge Pipeline. This Source will automatically convert syslog events to parsed logs when ingested. An API key is not required, and the connection is not protected with encryption. This is intended for use within your own network and isolated from the wide area network. The Edge instances have default port ranges of 8000-8010. You can add port numbers if required. Each Source in Edge requires a unique port number in order to receive data. The port number you use must fall within the range that is set upon deployment. # Tap and View Mezmo Edge Pipeline Data on Premises Source: https://docs.mezmo.com/mezmo-edge/tap-and-view-mezmo-edge-pipeline-data-on-premises Inspect data flowing through a local Mezmo Edge Pipeline using the vector tap CLI or GraphQL, without sending data to the Mezmo Web App. The SaaS version of Mezmo Telemetry Pipelines includes a [Data Tap feature](/telemetry-pipelines/view-pipeline-data) feature that enables you to view the flow of data through your Pipeline, and to confirm that the Processors are transforming and formatting the data as expected. This functionality is also available tapping remotely deployed Mezmo Edge instances. However, some organizations prefer to disable remote tapping to avoid the possibility of sensitive data leaving their network. In this case, you can leverage similar functionality in the `vector tap` CLI, or GraphQL , to view the data for your local Edge Instance, without having to send any data to the Mezmo Web App. ## Set Up Vector Tap with Mezmo Edge The easiest way to get access to `vector tap` is to use a command line interface to connect to a running Edge node, which will have built-in `vector tap` connectivity and tools. ### List Edge Nodes ```none theme={null} kubectl get pods -l app.kubernetes.io/name=edge ``` ### Connect to a Named Node ```none theme={null} kubectl exec -it -- bash ``` If `exec` is not available to your user in the cluster, you can use `vector tap` from any host with network connectivity to the cluster. ### List Services ```none theme={null} kubectl get svc -l app.kubernetes.io/name=edge ``` ### Forward the API Port ```none theme={null} kubectl port-forward 8686 ``` ## Tap Edge Data ### View Data for All Nodes You can use the `vector tap` to view data from combinations of node outputs in the Pipeline. The basic method is to view data at the output of every node: ```none theme={null} vector tap ``` ### View Data for a Specific Node To view the data stream for a specific Pipeline component, you need to provide the `component id`: ```none theme={null} vector tap ``` If tapping remotely is explicitly disabled by your administrator, the **Tap** button in the UI will prompt you with the exact `vector` command to tap that node: Image Click the copy/clipboard button to copy the command, which can be executed on the container or locally (with port forwarding active as described above). Alternatively, if port forwarding is active, you can click the **Vector GraphQL playground** link. This will open a new window that is connected to the GraphQL instance. The subscription query will be pre-populated with the selected node query, as shown in this screenshot: Image ### Manually Constructing a Custom Tap You can find the component id by viewing the Pipeline in the Mezmo Web App. Right-click on the component in the Pipeline Map, then click **inspect**. Image This will open the **Dev Tools** window, which will display a sub div of the component. Find `data-nodeid`, which will the UUID of the component. Image Enter the UUID of the component with wildcard characters and quotes: ```none theme={null} vector tap "*9c0b419c-bee9-11ee-b560-520afa0d7a83*" ``` This method can return multiple components, as seen in this example. This shows both the raw message exiting the component, as well as the same message after further processing. ```none theme={null} {"asdf":"fdsa","source_type":"http_server","timestamp":"2024-02-27T20:51:25.386608710Z"} {"message":{"asdf":"fdsa","timestamp":"2024-02-27T20:51:25.386608710Z"},"metadata":{"headers":{"accept":"*/*","content-length":"16","content-type":"application/x-www-form-urlencoded","host":"localhost:8010","user-agent":"curl/7.74.0"},"query":null},"timestamp":"2024-02-27T20:51:25.386608710Z"} ``` By including the component type, shown in this example as `http`, you can tap just the component itself: ```none theme={null} vector tap "*http*9c0b419c-bee9-11ee-b560-520afa0d7a83*" ``` This will display only the raw message that exits the component: ```none theme={null} {"asdf":"fdsa","source_type":"http_server","timestamp":"2024-02-27T20:57:56.435666086Z"} ``` You can find a list of all component types that you can tap in the **Component Types** section in this topic. ### View Data for a Component with Multiple Outputs #### View the Data for All Outputs To view the data through the various transformations of a component with multiple outputs, like the [Route Processor](/telemetry-pipelines/route-processor), use the `route` command with the UUID of the component. ```none theme={null} vector tap "*route*304b4ee2-d5a1-11ee-9cf9-12129877dec6*" ``` This will display the data for each output of the component: ```none theme={null} {"message":{"asdf":"asdf","timestamp":"2024-02-27T21:12:00.422454671Z"},"metadata":{"headers":{"accept":"*/*","content-length":"16","content-type":"application/x-www-form-urlencoded","host":"localhost:8010","user-agent":"curl/7.74.0"},"query":null},"timestamp":"2024-02-27T21:12:00.422454671Z"} {"message":{"asdf":"asdf","timestamp":"2024-02-27T21:12:00.422454671Z"},"metadata":{"headers":{"accept":"*/*","content-length":"16","content-type":"application/x-www-form-urlencoded","host":"localhost:8010","user-agent":"curl/7.74.0"},"query":null},"timestamp":"2024-02-27T21:12:00.422454671Z"} ``` #### View the Data for a Single Output To view the data for only one of the components outputs, you will need to use the child node ID. As with the node id, you can find the child ID node by using the Dev Tools to inspect the child node ID. Image Hover over the line containing the child node id, and the corresponding route in the Web App interface will be highlighted. You can use this to make sure you are selecting the correct child node id. The syntax of the child `data.nodeid` is of the form `.`. Separate the two ids with an asterisk and add quotations to the full sting: ```none theme={null} vector tap "*304b4ee2-d5a1-11ee-9cf9-12129877dec6*c534e85e*" ``` This will display the output for that specific route: ```none theme={null} {"message":{"asdf":"asdf","timestamp":"2024-02-27T21:36:04.122741881Z"},"metadata":{"headers":{"accept":"*/*","content-length":"16","content-type":"application/x-www-form-urlencoded","host":"localhost:8010","user-agent":"curl/7.74.0"},"query":null},"timestamp":"2024-02-27T21:36:04.122741881Z"} ``` ## Using GraphQL If you prefer, you can use the GraphQL Playground to inspect Edge Pipeline data. Follow the instructions for connecting Vector to your Edge instance described in the first section. After completing the step for Port Forwarding, use you browser to navigate to: [http://localhost:8686/playground](http://localhost:8686/playground) Image Obtain the component IDs as described in the previous section, and then use the`outputEventsByComponentIdPatterns` to inspect the component. Image This code sample shows the component ID within the `outputsPatterns` field: ```none theme={null} subscription { events: outputEventsByComponentIdPatterns( outputsPatterns: ["*http*9c0b419c-bee9-11ee-b560-520afa0d7a83*"], inputsPatterns: [], limit: 100) { ... on Log { type: __typename timestamp metadata: userMetadata message: json(field: ".message") } } } ``` ## Component Types Available for Inspection | Source Type | Processor Type | Destination Type | | ---------------------- | ------------------ | ----------------------- | | azure-event-hub | clustering | alert-message | | demo-logs | compact-fields | azure-blob-storage | | fluent | decrypt-fields | blackhole | | host-metrics | dedupe | clickhouse | | http | drop-fields | cloudwatch-logs | | kafka | encrypt-fields | cloudwatch-metrics | | kubernetes-logs | filter | datadog-logs | | mezmo-agent | flatten-fields | datadog-metrics | | open-telemetry-logs | js-script | elasticsearch | | open-telemetry-metrics | map-fields | gcp-cloud-monitoring | | open-telemetry-traces | mask-pii | gcp-cloud-operations | | splunk-hec | parse-sequentially | gcp-cloud-pubsub | | syslog | parse | gcp-cloud-storage | | | reduce | honeycomb-logs | | | route | http | | | sample | indexed-search | | | stringify | kafka | | | unroll | kinesis-firehose | | | vrl | kinesis-streams | | | | loki | | | | mezmo | | | | new-relic | | | | prometheus-remote-write | | | | pulsar | | | | redis | | | | s3 | | | | splunk-hec-logs | | | | sqs | | | | sumo-logic-logs | | | | sumo-logic-metrics | | | | vector | ## Disabling Tap from the Edge Helm Configuration To disable tap from the Edge itself, simply edit the `statefulset` and set two environment variables to `https://localhost`. This will cause the vector pod to emit error logs when it cannot fetch tasks, but otherwise functionality of the Edge instance is not affected other than to prevent data egress to the SaaS Control Plane from within an environment. ```none theme={null} # edit the statefulset kubectl edit sts/edge ``` # Getting Started with the Mezmo Pipeline API Source: https://docs.mezmo.com/pipeline-api-quickstart/getting-started-with-the-mezmo-pipeline-api Walk through curl examples to create a pipeline, add an HTTP source and access key, attach a transform and S3 destination, then publish. ## Create A New Pipeline ```bash theme={null} curl -X 'POST' \ 'https://api.mezmo.com/v3/pipeline' \ -H 'accept: */*' \ -H 'Authorization: Token ' \ -H 'Content-Type: application/json' \ -d '{ "title": "My API Pipeline", "deploy_type": "saas", "deployment_groups": [ ] }' ``` #### Response ```json theme={null} { "meta": { "pk": "id", "links": { "self": { "list": "/v3/pipeline", "detail": "/v3/pipeline/{pipeline_id}" }, "related": { "source": { "list": null, "detail": "/v3/pipeline/{pipeline_id}/source/{source_id}" }, "transform": { "list": null, "detail": "/v3/pipeline/{pipeline_id}/transform/{transform_id}" }, "sink": { "list": null, "detail": "/v3/pipeline/{pipeline_id}/sink/{sink_id}" } } }, "page": null }, "data": { "id": "", "account_id": "", "partition_id": "gen1", "title": "My API Pipeline", "deploy_type": "saas", "deployment_groups": [], "config": {}, "created_at": "2024-10-28T20:33:56.251Z", "updated_at": null, "published_at": null, "published_revision_id": null, "origin": "ui", "template": null } } ``` This operation creates a new Mezmo Pipeline in your account. Think of this as a blank canvas. We will need to add sources, transforms, and destinations to make use of it. Take note of the "id" under "data" which will be your pipeline id. You will use this in future operations to edit and publish your pipeline. ## Add A Source Now that you have a new pipeline created, it's time to add a Source to it, which defines where your data will be coming from into the pipeline. In this example, we will be creating an HTTP Source ```bash theme={null} curl -X 'POST' \ 'https://api.mezmo.com/v3/pipeline//source' \ -H 'accept: */*' \ -H 'Authorization: Token ' \ -H 'Content-Type: application/json' \ -d '{ "type": "http", "title": "My HTTP Source", "description": "Data sent via agent on server", "user_config": { "capture_metadata": true, "decoding": "auto" } }' ``` #### Response ```bash theme={null} { "meta": { "pk": "id", "type": "source", "links": { "self": { "list": null, "detail": "/v3/pipeline/{pipeline_id}/source/{source_id}" }, "related": { "pipeline": { "list": "/v3/pipeline", "detail": "/v3/pipeline/{pipeline_id}" } } }, "page": null }, "data": { "durability_profile": null, "access_keys": [], "access_keys_user_provided": false, "gateway_route_id": "c8b5911e-956e-11ef-a2a5-9aafefcf6cfb", "id": "c8b5911e-956e-11ef-a2a5-9aafefcf6cfb", "title": "My HTTP Source", "description": "Data sent via agent on server", "account_id": "", "pipeline_id": "", "generation_id": 0, "type": "http", "deploy_type": "saas", "user_config": { "decoding": "auto", "capture_metadata": true }, "outputs": [ { "id": "c8b5911e-956e-11ef-a2a5-9aafefcf6cfb", "label": "Default" } ] } } ``` Take Note of data/id. We will be using this in later steps so that we can connect pipeline nodes together. ## Adding An Access Key Now that we've created an HTTP Source, we need to add an access key to it so that only those authorized can send data to this endpoint. Grab the `gateway_route_id` from the response above as you'll use it in the next request. ```bash theme={null} curl --request POST \ --url https://api.mezmo.com/v3/pipeline/gateway-route//access-key \ --header 'Authorization: Token ' \ --header 'Content-Type: application/json' \ --data '{ "type": "generated", "title": "My Key" }' ``` ### Response ```bash theme={null} { "meta": { "pk": "id", "type": "access-key", "links": { "self": { "list": null, "detail": "/v3/pipeline/gateway-route/{gateway_route_id}/access-key" }, "related": { "gateway_route": { "list": "/v3/pipeline/gateway-route", "detail": "/v3/pipeline/gateway-route/{gateway_route_id}" } } }, "page": null }, "data": { "key": "", "id": "e6a18050-9570-11ef-aabe-9aafefcf6cfb", "account_id": "", "title": "My Key", "gateway_route_id": "", "created_at": "2024-10-28T21:09:20.998Z" } } ``` Success! Your access key has been generated under data/key and you can use this to start ingesting data! But first, let's add a transform to the pipeline to do something with the data coming in. ## Adding a Transform For this example, we're going to add a Remove Fields transform that will remove a set of fields from each payload that is ingested into the pipeline. Note: We also need to add the ID of the source transform above into the "inputs" field to connect these two nodes. This means that immediately after ingesting through your HTTP source, it will start removing fields. Without this, the nodes will be disconnected. ```bash theme={null} curl --request POST \ --url https://api.mezmo.com/v3/pipeline//transform \ --header 'Authorization: Token ' \ --header 'Content-Type: application/json' \ --header 'User-Agent: insomnia/10.1.1' \ --data '{ "type": "drop-fields", "title": "Remove my metadata", "description": "Remove Field", "user_config": { "fields": [".my_meta"] }, "inputs": [ "9c82e974-956f-11ef-bd42-9aafefcf6cfb" ] }' ``` ### Response ```bash theme={null} { "meta": { "pk": "id", "type": "transform", "links": { "self": { "list": null, "detail": "/v3/pipeline/{pipeline_id}/transform/{transform_id}" }, "related": { "pipeline": { "list": "/v3/pipeline", "detail": "/v3/pipeline/{pipeline_id}" } } }, "page": null }, "data": { "id": "01ee81fe-95fe-11ef-8c9b-9aafefcf6cfb", "title": "Remove my metadata", "description": "Remove Field", "account_id": "", "pipeline_id": "", "generation_id": 0, "type": "drop-fields", "deploy_type": "saas", "user_config": { "fields": [ ".my_meta" ] }, "inputs": [ "9c82e974-956f-11ef-bd42-9aafefcf6cfb" ], "outputs": [ { "id": "01ee81fe-95fe-11ef-8c9b-9aafefcf6cfb", "label": "Default" } ] } } ``` Great! Now we have a new transform created that is connected to our HTTP Source node. Next, let's send this data somewhere, like AWS. Take note of data/id in the response. We will be using this in the next step to connect this transform node to the destination of our choosing. ## Creating a Destination Great! Now let's create a place for this data to go to. In this example, we've chosen to configure an S3 Destination. ```bash theme={null} curl --request POST \ --url https://api.mezmo.com/v3//sink \ --header 'Authorization: Token ' \ --header 'Content-Type: application/json' \ --header 'User-Agent: insomnia/10.1.1' \ --data '{ "type": "s3", "user_config": { "batch_timeout_secs": 300, "ack_enabled": true, "auth": { "access_key_id": "123456", "secret_access_key": "7891011" }, "bucket": "my-s3", "prefix": "/mezmo", "encoding": "text", "compression": "none", "region": "us-east-1", "file_consolidation": { "enabled": false, "process_every_seconds": 600, "requested_size_bytes": 500000000, "base_path": "" } }, "inputs": ["01ee81fe-95fe-11ef-8c9b-9aafefcf6cfb"] }' ``` ### Response ```bash theme={null} { "meta": { "pk": "id", "type": "sink", "links": { "self": { "list": null, "detail": "/v3/pipeline/{pipeline_id}/sink/{sink_id}" }, "related": { "pipeline": { "list": "/v3/pipeline", "detail": "/v3/pipeline/{pipeline_id}" } } }, "page": null }, "data": { "id": "7f88617c-97b9-11ef-bdc8-9aafefcf6cfb", "title": null, "description": null, "account_id": "", "pipeline_id": "", "generation_id": 0, "type": "s3", "deploy_type": "saas", "user_config": { "auth": { "access_key_id": "123456", "secret_access_key": "7891011" }, "bucket": "my-s3", "prefix": "/mezmo", "region": "us-east-1", "encoding": "text", "ack_enabled": true, "compression": "none", "batch_timeout_secs": 300, "file_consolidation": { "enabled": false, "base_path": "", "requested_size_bytes": 500000000, "process_every_seconds": 600 } }, "inputs": [ "01ee81fe-95fe-11ef-8c9b-9aafefcf6cfb" ] } } ``` ## Publishing The Pipeline! ```bash theme={null} curl --request POST \ --url https://api.mezmo.com/v3/pipeline//publish \ --header 'Authorization: Token ' \ --header 'User-Agent: insomnia/10.1.1' ``` #### Response ```bash theme={null} { "meta": { "pk": "id", "type": "pipeline", "links": { "self": { "list": null, "detail": "/v3/pipeline/{pipeline_id}/publish" }, "related": { "pipeline": { "list": "/v3/pipeline", "detail": "/v3/pipeline/{pipeline_id}" }, "source": { "list": null, "detail": "/v3/pipeline/{pipeline_id}/source/{source_id}" }, "transform": { "list": null, "detail": "/v3/pipeline/{pipeline_id}/transform/{transform_id}" }, "sink": { "list": null, "detail": "/v3/pipeline/{pipeline_id}/sink/{sink_id}" } } }, "page": null }, "data": { "id": "", "account_id": "", "partition_id": "gen1", "title": "My API Pipeline 3", "deploy_type": "saas", "config": {}, "created_at": "2024-10-28T20:49:35.547Z", "updated_at": "2024-10-31T19:17:13.632Z", "published_at": "2024-10-31T19:17:18.005Z", "published_revision_id": "bea822cc-97bc-11ef-b6dd-9aafefcf6cfb", "deployed_revision_id": null, "loaded_revision_id": null, "processing_status": "enabled", "origin": "ui" } } ``` And that's it! You've now published your first pipeline that you can send data into and out to S3! Stay tuned for part two for more advanced use cases, including removing/modifying nodes, reverting a pipeline version, pausing, and more. # Analysis of Telemetry Data Optimization Techniques Source: https://docs.mezmo.com/practioner-guide-data-optimization/analysis-data-reduction-techniques Summary results and findings from testing data reduction techniques on Priority 1 sources, showing 50%+ volume cuts without quality loss. **Estimated Reading Time**: 3 minutes This topic summarizes the results of an initial round of analysis for data reduction techniques applied to sample data collected from Priority 1 sources. All sample data was generated from internal Mezmo sources, except as noted. ## Priority 1 Telemetry Data Sources | Sample | Input Size | Number of Lines | Avg Reduction | | ----------------------- | ---------- | --------------- | ------------- | | Palo Alto Firewall Logs | 2.8 MB | 2825 | 62% | | Java Logs Kafka | 452.5MB | 441969 | 61% | | Java Logs Kafka (OTel) | 4.4 MB | 22956 | 75% | | Kubernetes Logs | 1.7 MB | 14832 | 77% | | AWS Network Firewall | 502.9 MB | 942248 | 95% | | Prometheus Metrics | 3.65 MB | 4883405 | 96% | ## Overall Findings We tested four representative log samples and one representative metric sample. Our findings are: **Every sample could be reduced in volume by at least 50% (often more) without reducing data quality.** Note that data quality means no substantial loss in the ability to understand the data at a higher level. For example, in combining firewall events, we can realize a substantial savings by merging multiple events into a single event where all of the unique fields are still retained. The original events could be recreated if desired from the merged event. **Dropping redundant events results in a reasonable savings for informational logs such as weblogs. An additional enhancement of including a count of the logs dropped representing the removed lines was introduced as well to aid in visibility.** This technique is most effective when noise reduction is needed and any additional metrics regarding the noise are largely unnecessary. This technique uses a moving window and is the easiest approach for many different log types **Turning logs into metrics based on criteria or extracting numeric values from the event itself to make a metric is highly effective at reducing data size while still retaining trending information.** This technique results in removal of most of the log message, meaning it can lead to permanent data loss if not tuned carefully. Because of this, this technique is best suited for messages where the value of the message content is low, such as informational or notice type messages, but insights can be drawn from looking at the log trend over time. This technique can be taken to extremes, meaning 99% of the data can be thrown out and turned into numeric data, but it would not be likely to be desired in practice due to the loss of data fidelity. **Grouping logs based on similar information is also highly effective at reducing total size, but has the benefit of retaining specific data fields that are valuable.** This must be targeted based on the fields, so there is some moderate complexity in configuring the processors. Downstream tools must also be able to handle the higher complexity of the message structure resulting from how the logs are merged together. **The technique of trimming excess data from the event was only applicable to Java logs in our sample set and resulted in the lowest reduction.** This technique is typically used when payloads are very large, which was not the case in our samples. It also requires complex processor configuration, and so, while not a usual technique, can provide benefits in very specific cases. For metrics, aggregation of incremental values and sampling of gauge values was successful in reducing volume at the cost of granularity of the data. This technique is easy to implement and can also be easily tuned. # Introduction: Telemetry Data Optimization with Mezmo Pipelines Source: https://docs.mezmo.com/practioner-guide-data-optimization/data-optimization-introduction How Mezmo helps SREs understand, optimize, and reduce overwhelming telemetry volume, with an overview of what this practitioner's guide covers. ## A Day in the Life of an SRE: Too Much Data, Man! Today's applications are composed of a multitude of components and micro-services within a "stack," each of which generates its own telemetry data. This results in an overwhelming volume of data to store and analyze. In many cases, this data is sent directly to observability tools, but lacks the optimization necessary to turn it from raw data to useful information. Worse yet, much of this data is not inherently useful and only contributes to "data bloat" that results in high costs for storage and observability tools. ## Mezmo's Approach to Data Optimization Mezmo's approach to the problem of too much data is to provide you with the means to first understand your data, optimize it for your observability tools and storage solutions, and derive new insights into your systems. In our own analysis of typical data samples from sources such as Kubernetes, we have identified specific processing techniques for optimizing and reducing the volume of telemetry data, developed Processor components to match these techniques, and created features that enable you to understand and interact with your data in stream. ## In This Guide In this guide you'll find: * A [deep dive into the five techniques of data optimization](/practioner-guide-data-optimization/optimize-your-observability-data-in-six-steps) and their relationship to Mezmo Telemetry Pipeline Processors * The [summary results and key findings](/practioner-guide-data-optimization/analysis-data-reduction-techniques) from our research into telemetry data optimization * A set of [Pipeline Examples and Tutorials](/practioner-guide-data-optimization/reference-architectures-for-data-optimization-pipelines) so you can learn how to create Pipelines for your own use cases # Demo Pipeline Overview Source: https://docs.mezmo.com/practioner-guide-data-optimization/demo-pipeline-guide8ke Tour the trial Demo Pipeline's sources, Route and Filter processors, and Encrypt Field processor for handling credit card transaction data. When you sign up for a free trial of Mezmo Telemetry Pipeline, a Demo Pipeline is automatically created for you to explore Pipeline features and functionality. This topic describes the architecture of the Demo Pipeline, along with highlights of its Sources, Processors, and Destinations. ## The Scenario The Demo Pipeline is designed to illustrate a typical scenario, where there are multiple pipeline sources, a Route Processor that filters data based on conditional criteria, and additional processors for the filtered data that prepares it for routing to one destination for analysis, and another for archival storage. The specific scenario is a typical processing of JSON and Financial data to identify errors related to credit card transactions, which also requires encryption of credit card data before it can be sent to an analytical tool. In this guide, you'll see how to use the Route, Filter, and Encrypt Processors to accomplish this. ## Architecture Overview To explore the Demo Pipeline in the Mezmo Web App: 1. Log in to [the Mezmo Web App](https://app.mezmo.com). 2. Click **Pipelines**. 3. Under **Deployed**, select **Demo Pipeline**. If necessary, you can also click **Re-start Pipeline** to send the demo source data through the Pipeline. Image ### 1 Sources The Demo Pipeline has two Sources, which are both versions of the [Demo Logs](/telemetry-pipelines/demo-logs-source). You can use this Source to build your Pipelines using sample data before connecting them to live Production Sources, to make sure that your Processors are producing the results you want. #### Financial Data and JSON Use the [Pipeline Tap feature](/telemetry-pipelines/view-pipeline-data) to view the sample Financial and JSON data. You can also download the sample data to view the full JSON, and build your own sample data. Image #### JSON Data Image ### 2 Route Processor The [Route Processor](/telemetry-pipelines/route-processor) uses conditional statements to send data to other processors or destinations. In this case, there are four statements: | Route Name | Purpose | Conditional Statement | | Routed To | | --------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------- | - | ------------------------------ | | Purchase Transactions | Selects transaction events | `if (exists(.event) AND .event contains 'transaction')` | | Allow "Card Denied" Filter | | Login/Logout Events | Selects login and logout events | `if (exists(.event) AND .event contains 'log')` | | Drop Login/Logout Event Filter | | HTTP non-200s | Selects HTTP events that are not 200s (Success) | `if (exists(.status) AND .status greater 200)` | | Long Term Analysis Destination | | Unmatched | Bucket for any data that is not selected by the other conditional statements | None | | Archival Destination | You can test your Route Processor by using a [PIpeline Tap](/telemetry-pipelines/view-pipeline-data) to view the data flowing into it from the Sources, and inserting a tap for each route to make sure that data is passing through as expected. ### 3 and 4 Filter Processors The Route Processor sends matched data to two [Filter Processors](/telemetry-pipelines/filter-processor). | Filter Processor | Purpose | Conditional Statement | Routed To | | ------------------------ | ------------------------------------------------------------------------------------- | ---------------------------------------------------------- | -------------------------------------------- | | Allow "Card Denied" | Filters the Purchase Transactions data to select those with a result of "Card Denied" | `if (.transaction.result_reason __contains 'card_denied')` | Encrypt Card Details Encrypt Field Processor | | Drop Login/Logout Events | Drops the Login/Logout Events matched by the Route Processor | `if (.access.action contains 'log')` | Long Term Analysis Destination | ### 5 Encrypt Field Processor For security compliance, credit card information should be encrypted before reaching the Long Term Analysis destination. With the [Encrypt Field Processor](/telemetry-pipelines/encrypt-fields-processor), you can set encryption for a specific field, along with the encryption algorithm and key, and the Initialization Vector (IV) field. | **Encrypted Field** | `.transaction.cc.cc_number` | | ------------------------------- | ---------------------------------------------------- | | **Encryption Algorithm** | `AES-256-CFB (key = 32 characters, iv=16 characters` | | **Encryption Key** | `keyenrcypt123456789keyenrcypt123` | | **Initialization Vector Field** | `.IVFIELD` | You can use the [Decrypt Field Processor](/telemetry-pipelines/decrypt-fields-processor) with the same settings if you need to later decrypt the data. ### 6 Destinations The routed and filtered data is sent to two versions of the [Blackhole](/telemetry-pipelines/blackhole-destination) destination, one representing Long Term Analysis, the other Archival Storage. As with the Demo Logs Pipeline Source, the Black Hole destination is useful for making sure your log data is processed as expected before connecting it to a Production Destination. # Optimize Your Observability Data in Six Steps Source: https://docs.mezmo.com/practioner-guide-data-optimization/optimize-your-observability-data-in-six-steps A platform-agnostic approach to cutting log volume by 50%+ via archiving, filtering, parsing, merging, metric conversion, and responsive pipelines. **Estimated Reading Time: 10 minutes** ## Introduction: Improving the ROI of Your Observability Data Modern observability platforms are incredibly powerful—but they are also expensive if you send everything, all the time. The goal of data optimization is **not** to lose visibility, but to ensure that the *right data* reaches your observability tools at the *right time*, while everything else is handled more cost‑effectively. This guide walks through a practical, platform‑agnostic approach to optimizing log data volume using the Mezmo pipeline. These principles apply no matter what Observability platforms you use. ## The Six Steps of Observability Data Optimization 1. **Archive** a full fidelity copy of your telemetry data to cheaper, long-term retention solutions for future auditing or analysis, instead of keeping everything in your more expensive Observability platform. 2. **Filter** early and intentionally the duplicate and extraneous events that don’t contribute value to your observability results. 3. **Parse and Structure** events by removing empty values, dropping unnecessary labels, and transforming inefficient data formats into a format specific to your observability destinations. 4. **Merge** events together by grouping messages and combining their fields to retain unique data while removing repetitive data. 5. **Condense events into metrics** to reduce the number of hours and resources dedicated to supporting back-end tools, and convert unstructured data to structured before indexing to make searches more manageable, faster, and efficient. 6. **Configure Responsive Pipelines** to provide your developers and SREs full fidelity data when they need it to troubleshoot and then return to normal optimization when they're done. ## From Steps to Practice Some of these steps may seem obvious, but they are not easy to put into practice. An observability agent alone is insufficient. Agents are neutral forwarders—they collect telemetry and send it downstream, but they do not meaningfully process or optimize data in transit. You *could* implement portions of this approach using open-source tools and custom development, but this typically introduces significant operational cost and complexity. Teams must build and maintain expertise that is not core to their business. The fundamental challenge is that most tools fall into one of two categories: * **Agents**, which only send data * **Observability platforms**, which only receive and analyze data What’s missing is the ability to **process telemetry data in-stream**—to transform, optimize, and route it *as it flows* from source to destination. Mezmo Telemetry Pipelines were designed specifically to address this gap. They give you precise control over the flow of telemetry between data sources and observability tools, allowing you to optimize and shape data *before* it arrives downstream. ## Understanding Pipeline Order Before You Optimize Before applying any optimization techniques, it’s critical to understand that **processor order directly impacts cost, flexibility, and safety**. A poorly ordered pipeline can undo the benefits of even the best filtering strategy. ### Recommended High-Level Pipeline Order 1. **Ingest** – Agents, collectors, forwarders receive raw telemetry 2. **Archive** – Persist a copy of raw logs in low-cost storage 3. **Filter** – Remove clearly low-value noise 4. **Parse / Structure** – Extract fields from logs you intend to keep 5. **Merge** – Condense multiple events into one while maintaining meaning 6. **Convert** – Create metrics based on remaining events 7. **Route** – Deliver data to one or more destinations #### Why this order matters Expensive operations like parsing and enrichment should only be applied to data that has already proven its value. Archiving early gives you freedom to optimize aggressively without fear of permanent data loss. ## Putting the Five Steps into Practice ### 1: Archive The foundation of any safe optimization strategy is **archiving all raw logs before making any destructive decisions**. Archiving means writing an unmodified copy of every log event to low-cost, durable object storage such as Amazon S3, Azure Blob Storage, or Google Cloud Storage. #### Why Archiving Comes First Archiving transforms your object store into the **system of record for logs**. Observability platforms become optimized analysis tools rather than long-term retention systems. With a complete archive, you gain: * Confidence to filter aggressively downstream * A forensic record for audits, security investigations, and compliance * The ability to reprocess or replay logs if requirements change **Best Practice** Always archive **raw, unparsed, and unfiltered logs**. This preserves maximum future flexibility. #### How to Implement Archiving * Write logs to object storage immediately after ingestion * Partition data by date, environment, and service * Apply lifecycle policies to transition older data to colder tiers (e.g., Glacier, Archive) **Common Pitfall** Archiving parsed or enriched logs increases storage cost and permanently locks in today’s schema decisions. To implement archiving in your telemetry pipeline, review the tutorial on [Creating a Basic Data Archiving and Restoration Pipeline](/telemetry-pipelines/archive-restore-data-tutorial), which demonstrates how to add an archive destination and rehydrate data when needed. ### 2: Filter Once logs are safely archived, filtering becomes the most impactful way to reduce data volume and observability spend. Filtering removes logs that are **high frequency but low diagnostic value**—data that rarely contributes to troubleshooting, alerting, or root cause analysis. #### What Makes a Log a Good Filtering Candidate? Logs are strong candidates for filtering if they: * Occur continuously during healthy operation * Are never referenced during incidents * Duplicate signals already captured by metrics Common examples include health checks, load balancer probes, Kubernetes liveness checks, and verbose debug output in production. **Rule of Thumb** If a log has never helped you resolve an incident, it probably shouldn’t be sent to your observability platform. #### How to Identify What to Filter Use Mezmo analytics or the [Data Profiler](/telemetry-pipelines/data-profiler-processor) to identify: * Top log-producing services * Most frequent message templates * Dominant log levels by volume Start by filtering the most obvious noise, then iterate gradually. **Best Practice** Apply filters as early as possible in the pipeline, before parsing or enrichment. Filtering is easily accomplished using a [Filter](/telemetry-pipelines/filter-processor) processor, which allows you to include or drop events based on conditions or Log Analysis queries. Multiple conditions can be defined in a single processor—there is no need to create separate processors for each rule, which would add unnecessary overhead. ### 3: Parse and Structure Many applications pack excessive information into single log lines—often including stack traces or serialized data objects intended only for debugging. These large, semi-structured messages increase storage cost and make searching inefficient. Parsing converts raw log lines into structured fields that enable powerful querying and alerting. With Mezmo’s [Parse Processor](/telemetry-pipelines/parse-processor), you can extract the fields that matter using regex or grok, then remove unnecessary data. For example, stack traces can often be reduced to just the originating source location while preserving diagnostic value. #### Why Selective Parsing Matters Parsing logs that will later be discarded wastes processing capacity and cost. Instead, parsing should be reserved for logs that provide clear operational value. #### How to Approach Parsing * Parse logs **after filtering**, not before * Extract only fields that are actively queried or alerted on * Avoid deeply parsing rarely used nested structures Examples of high-value fields include request IDs, error codes, user identifiers, and severity levels. **Common Pitfall** Over-parsing everything “just in case” often increases cost without improving outcomes. ### 4: Merge Many applications emit multiple log lines to describe what is logically a *single event*. Common examples include firewall logs from systems like Palo Alto and AWS Firewalls that generate a high volume of log events. Often these logs share a number of fields that are non-unique. However, you would not want to simply drop the logs due to the importance of the information from a security perspective. Left un-optimized, these patterns dramatically increase log volume while making troubleshooting harder, not easier. With Mezmo’s [Reduce Processor](/telemetry-pipelines/reduce-processor) you can merge multiple log input events into a single log event based on specified criteria. For example, Threat and Traffic logs from the firewall share 70% of the same fields, and are tied to the same events by a common `sessionid` field. #### Why Log Reduction Matters When related log lines are merged: * Log volume is reduced without losing information * Context is preserved in a single event * Queries and investigations become simpler **Why This Is Helpful** Five uncorrelated log lines are harder to reason about—and more expensive—than one well-structured event. #### Common Use Cases for Reduce The Reduce processor is particularly effective for: * Multiline stack traces and exceptions * Logs grouped by a shared request ID or trace ID * Sequential logs that represent a single operation * Framework-generated logs with predictable patterns #### How to Apply the Reduce Processor * Apply Reduce **after filtering**, so you only reduce logs you intend to keep * Configure grouping keys such as request ID, trace ID, or container ID * Define a time window to collect related log lines The result is a single log event that contains the full context of the original sequence. **Best Practice** Use Reduce to *increase signal density*, not to obscure detail. The merged log should be easier to understand than the originals. 5: Condense Events to Metrics Not all operational signals need to remain as logs. Many high-volume logs exist primarily to answer quantitative questions such as *how often*, *how long*, or *how many*. In these cases, converting logs into **metrics** preserves the signal while dramatically reducing data volume. #### When Logs Should Become Metrics Logs are strong candidates for metric conversion when they: * Occur at very high frequency * Represent counts, durations, or rates * Are primarily used for dashboards or alerts Common examples include request counts, error rates, latency measurements, and job success/failure totals. **Rule of Thumb** If you aggregate it every time you query it, it should probably be a metric. #### How Log-to-Metric Conversion Helps * Metrics are far more storage- and query-efficient than logs * Dashboards and alerts become faster and cheaper * Logs can be filtered once the metric is emitted #### Practical Approach * Identify log fields that represent numeric values or discrete outcomes * Emit counters, gauges, or histograms from those logs * Retain only error or anomaly logs for deep inspection **Common Pitfall** Keeping both full logs and derived metrics indefinitely often defeats the cost-saving benefit. Use [the Mezmo Event to Metric Processor](/telemetry-pipelines/event-to-metric-processor) to convert logs metrics and visualize them on an operational dashboard, providing valuable business insights while also helping reduce the inefficiencies that SRE teams and others have when accessing information they want. [Tutorial: Convert Events to Metrics](/practioner-guide-data-optimization/pipeline-example--convert-200-events-to-metrics) provides an overview of an event-to-metric Pipeline, along with information on Processor configuration. ### 6: Configure Responsive Pipelines Static pipelines force teams into a permanent trade-off: optimize for cost *or* optimize for visibility. A **responsive pipeline** removes that trade-off by allowing the pipeline to switch operating modes based on operational context. At the heart of a responsive pipeline is the ability to **bypass filters and transforms on demand**. ### What Switching Modes Really Means When a pipeline switches modes, it does not simply send *more* data—it **changes execution paths** inside the pipeline: * Filters are bypassed so no log events are dropped * Reduce, parse, enrich, and transform processors are skipped or minimized * Raw (or near-raw) logs are forwarded directly to observability platforms This ensures that, during an incident, you see *exactly what the application emitted*, without optimization logic getting in the way. Developers and SREs do not need to worry that your optimization efforts will deny them the data they need to identify, diagnose, and remediate application issues. c **Why This Matters** Filters and transforms are designed for efficiency. During incidents, fidelity matters more than efficiency. #### Normal Mode (Cost-Optimized) In Normal Mode, the pipeline prioritizes signal-to-noise ratio and cost control: * Archiving is always enabled * Filters aggressively remove known noise * Reduce merges related log lines * Logs are parsed, enriched, and transformed * Optimized events are sent to observability platforms This mode supports day-to-day operations at scale without unnecessary spend. #### Incident Mode (Fidelity-First) In Incident Mode, the pipeline prioritizes completeness and speed of investigation: * **Filters are bypassed** (no logs are dropped) * **Transforms and reductions are bypassed** to preserve raw detail * Minimal processing is applied * Raw logs are forwarded directly to observability tools This provides maximum visibility when teams are actively troubleshooting. **Best Practice** Incident Mode should favor raw data over perfect structure. Structure can always be added later. #### How to Implement Pipeline Mode Switching There are two ways to change a pipeline's mode. 1. It can be changed manually in the interface, using the drop down selector in the top left of the pipeline window pane. 2. It can be changed programmatically using the pipeline APIs. You can do this using a script processor or the Notification Channel destination processor. See [Configure Responsive Pipelines](/telemetry-pipelines/configure-responsive-pipelines#set-the-pipeline-operational-state) for more details and an example. **Common Pitfall** Leaving Incident Mode enabled indefinitely negates the benefits of optimization. Research and Findings ### Methodology To test these techniques and substantiate our data reduction claims, we undertook [a research project with our customer engineering and product team](/practioner-guide-data-optimization/analysis-data-reduction-techniques)s. Data was collected from internal Mezmo sources where available to make it as close to representative of real world data as possible. Data collected from external sources was sourced from Kaggle.com and other open source locations, such as GitHub. Data was then groomed via scripting as needed to flatten for loading into Snowflake. Each log schema was parsed and given its own table for storage and comparison. In parallel, Telemetry Pipelines were created in a production environment with a standard account tied to the individual source types. Data was injected into each pipeline for each sample through an HTTP source. Each pipeline attempted to follow the Snowflake queries, though variations in the technologies required some alterations. Data samples sent into the pipeline were forwarded to HTTP destinations for comparison in the byte count from input to output. Due to how pipelines and network layer traffic work, this naturally introduces variation versus the Snowflake analysis, so the results were not expected to match perfectly. However, these results more closely resemble real world cases because network layer translation would always be a part of any functioning log / metric system. ### Key Findings The net findings are that following these steps can reduce the volume of telemetry data by **50% or more** without impacting your observability data, and that this is true across the many data sources we tested. * Using the **Filter** technique and dropping redundant events with deduplication criteria resulted in a **62% reduction from standard web logs such as Apache and nginx by matching based on the IP, URL, and request type**. * Using the **Route** technique, we were able to separate more than 67% of Kubernetes logs by routing them to cold storage. * Using the **Trim and transform** technique, we were able to **reduce** **Kafka logs 50%** **by extracting common message data including process status updates, topic creation, and messages from the Controller.** Note that we still kept information fidelity in case it was needed for troubleshooting. * Using the **Merge** technique, we were able to **reduce Firewall Log volume by 94% by removing unnecessary fields and grouping events based on source and destination IPs.** * **Converting logs to metrics** can result in **over 90%** **reduction in total volume for all informational logs**, but the process must be carefully tuned to avoid the risk of losing potentially valuable data while avoiding an explosion of tag cardinality. Our Sales Engineering team can provide more information based on your data sources and observability needs. ## Conclusion By following the six steps described in this paper in the design of your Telemetry Pipeline, you can realize significant data optimization to reduce the cost of your observability data. If you want to know more about our research and findings, or to find out how our steps can be applied to your telemetry data, [reach out to our Solutions Engineering team](https://go.mezmo.com/schedule-a-demo?utm*medium=docs\&utm*source=docs-paid\&utm*campaign=practitioners-guide). # Pipeline Example: Kafka Telemetry Data Optimization Source: https://docs.mezmo.com/practioner-guide-data-optimization/pipeline-architecture-for-data-reduction An example pipeline that parses, routes, reduces, and converts Kafka logs to metrics while preserving errors and warnings at full fidelity. **Estimated Reading Time**: 5 minutes ## The Situation Applications today are often composed of many different components to create a "stack." Each part of the stack is important to the function of the application, but not every part behaves the same way. Open source components like Kafka can generate a substantial volume of logs. Many of the logs can provide valuable operational data in real time needed to understand the application behavior, while others are less relevant and can be sent to storage for later analysis. This Pipeline provides a model for optimizing your log data based on the most common log messages that would be generated from a Kafka cluster, but is also an example of how to create a Pipeline to optimize data from any other system that generates similar log data. This architecture includes [standard best practices](/practioner-guide-data-optimization/optimize-your-observability-data-in-six-steps) such as removing extraneous events from the stream, routing data to specific destinations based on the event type, and converting events to metrics for use in operational dashboards. For Kafka data, the log types we identified that can be easily optimized include partition management, record generation, and deletion information. These log types can be either summarized or rolled into metrics for monitoring without needing additional storage space. Errors and Warnings are given direct paths to storage and left untouched for full fidelity. ## Architecture Overview ### Sources #### 1 - HTTP Endpoint For this example, the [HTTP Endpoint](/telemetry-pipelines/http-destination) Source includes a data sample that represents Kafka logs. We are using this source because it can accept any log data via an HTTP post request. In practice, you may be using an Agent of some kind, but the same principles apply. * To view this sample within the Mezmo Web App, go to \*\*Pipelines > Mezmo Java Demo > Sample Management, then click on the sample. * You can run this sample by selecting the HTTP Endpoint Source, then select **Simulate Pipeline**. * To view the effect of the Processors on the data, select a Processor, then select **Tap egress**. ### Processors #### 2 - Parse The [Parse Processor](/telemetry-pipelines/parse-processor) uses a [Grok Pattern](/telemetry-pipelines/using-grok-to-parse) to parse the Kafka logs that the Pipeline will process and standardize their data format. Note that Mezmo has custom Grok expressions, including `%{SQUARE_BRACKET}` , which is used in this example for convenience. ```none theme={null} %{SQUARE_BRACKET}%{TIMESTAMP_ISO8601:timestamp}%{SQUARE_BRACKET} %{LOGLEVEL:level} %{GREEDYDATA:description} ``` #### 3 - Route The [Route Processor](/telemetry-pipelines/route-processor) uses conditional statements to match log data and provide flow control that separates the data for the Metrics and Log Consumer destinations. **Generating Records Route** This statement matches the terms `generating` and `generated` in the `.description` field of the data, and routes it to an [Event to Metric Processor](/telemetry-pipelines/event-to-metric-processor). ```none theme={null} if (.description contains 'generating' OR .description contains 'generated') ``` **Partition Management** This statement matches the term partition in the `.description` field of the data, and routes it to an [Event to Metric Processor](/telemetry-pipelines/event-to-metric-processor). ```none theme={null} if (.description contains 'partition') ``` **Error and Warnings** This statement matches the terms `warn` and `error` in the .`level` field of the data, and routes it directly the Log Consumer destination. Critical events, like errors and warnings, should be routed directly to storage or your log analysis system to preserve full-fidelity copies for later analysis. If the warnings are especially verbose, you could also convert them to metrics based on your specific needs. ```none theme={null} if (.level equal 'warn' OR 'error') ``` **Deleting info** This statement matches the term `deleted` in the .`description` field of the data, and routes it to the [Reduce Processor](/telemetry-pipelines/reduce-processor). ```none theme={null} if (.description contains 'deleted') ``` **Unmatched** Any data that doesn't match the conditional statements is routed directly to the Log Consumer destination. #### 4 - Event to Metric The two [Event to Metric Processor](/telemetry-pipelines/event-to-metric-processor)s are set to take the incoming log events and convert them to metrics, then sends the converted metrics to the [Aggregate Processor](/telemetry-pipelines/aggregate-processor) The configuration of these Processors represents the best practice of reducing repetitive events, like the start and stop of processes, to metrics. The valuable information in these events isn't within the single event itself, but in the total number of operations and the load they place on your systems. The same is true of positive events, like `200-OK` messages. By converting these types of events to metrics, your Pipeline can provide you with useful information while also substantially reducing the volumes of data you send to your monitoring systems. **Generating Records** This Processor creates a counter metric for each event with `generating` or `generated` in the `.description` field, and creates an incremental count starting at 1. **Partition Management** This Processor creates a counter metric for each event with `partition` in the `.description` field, and creates an incremental count starting at 1. #### 5 - Aggregate (Metric) The [Aggregate Processor](/telemetry-pipelines/aggregate-processor)converts the metric counts from the Event to Metric Processor to an aggregated metric based on a count of events over 10 second intervals. When setting time intervals for the **Aggregate** and **Reduce** Processors, you should consider how faithful you need to be to the original data to get the information you need. As a rule of thumb:> > **30 seconds+** for low fidelity needs, ensuring positive affirmations> > **10 seconds** for medium fidelity needs> > **1 second** for high fidelity> > **\< 1 second** for very high fidelity #### 6 - Reduce Similar to the Event to Metric Processors, the [Reduce Processor](/telemetry-pipelines/reduce-processor) converts the `deleted` events sent to it from the Route Processor into a single event based on an interval of 30 seconds, and appends this as an array to the .description field for consumption by the Log Consumer. ### Destinations For purposes of this example, this Pipeline terminates in two [Blackhole](/telemetry-pipelines/blackhole-destination) destinations. All data sent to a Black Hole is dropped for the purpose counting against your egress volume. This lets you construct a Pipeline and make sure that the data being sent to each destination is in the desired state before sending it to your production systems. In this example, the Blackholes represent two typical destinations for operational information, one that consumes metric data, and another that consumes log/event data. # Pipeline Example: Kubernetes Telemetry Data Optimization Source: https://docs.mezmo.com/practioner-guide-data-optimization/pipeline-architecture-for-kubernetes-data-optimization An example pipeline that filters, converts to metrics, aggregates, and routes Kubernetes container logs while archiving originals for compliance. ## The Situation This Pipeline models a typical situation where you have telemetry data originating from a Kubernetes cluster and need to transform it into metric data for consumption by an observability tool, while also retaining a copy of the original data in storage for compliance and later analysis. By using a Pipeline to transform the data as it is streamed, you can reduce the volume of data sent to your tool, and at the same time ensure that data sent to the tool will be optimized to provide useful information. If you would like to try out this Pipeline with your own Kubernetes data, this topic includes configuration information for each Processor. You can find more detailed information about Mezmo Telemetry Pipelines in [our product guide](https://docs.mezmo.com). If you don't have a Mezmo account yet, [sign up for a free trial](https://www.mezmo.com/sign-up-pipeline-today) so you can try out our product features and start managing your telemetry data! ## Architecture Overview ## Sources #### 1 Splunk HEC This Pipeline uses the [Splunk HEC](/telemetry-pipelines/splunk-hec-source) Source as the ingress point for Kubernetes telemetry data, but there are also a variety of [Supported Telemetry Pipeline Sources](/telemetry-pipelines/supported-telemetry-pipeline-sources), including OTel Sources, that you can use. ## Processors ### Container Logs Processing Chain Nodes 2, 3, and 4 represent the chain for processing Kubernetes container logs. #### 2 - Filter Processor The [Filter Processor](/telemetry-pipelines/filter-processor) uses a conditional statement to identify telemetry data specifically related to containers within the Kubernetes cluster, and allows matching data to proceed to the next step of the Processor chain. ```none theme={null} if (exists(metadata.fields."k8s.container.name")) ``` #### 3 - Event to Metric Processor The [Event to Metric Processor](/telemetry-pipelines/event-to-metric-processor) converts the Kubernetes events into metrics representing log entries by node, and log entries by container. | Option | Setting | | -------------------- | -------------------------------------- | | Metric Name | l`og_entry_by_node` | | Kind | `Incremental` | | Type | `Counter` | | Value/Value Type | `New value` | | Value/Value | `1` | | Namespace/Value Type | `None` | | Tags/Name | `node_name` | | Tags/Value Type | `Value from Event Field` | | Tags/Field Value | `metadata.fields."k8snode.name".field` | #### 4 - Aggregate Metrics The [Aggregate Processor](/telemetry-pipelines/aggregate-processor) aggregates multiple metric events into a single metric event based on a defined interval window. In this case, the Processor aggregates all the metric events for the Kubernetes node logs into a single metric over a one minute interval. | Option | Setting | | ------------------------------ | ---------------------------- | | Group by Field Paths | `.name` `.namespace` `.tags` | | Evaluate/Operation | `add` | | Window Type/Type | `tumbling` | | Window Type/Interval (seconds) | `60` | | Event Timestamp | `.timestamp.field` | ### Metric Counters Processing Chain Processors 5 and 6 convert log message events of certain types to metrics and produces a count of each type. #### 5 - Route Processor The [Route Processor](/telemetry-pipelines/route-processor) uses conditional statements to match log messages related to **Errors**, **Exceptions**, and **Negative Sentiment** (Abort, Broken, Kill, etc.) and sends them to specific Event to Metric Processors. | Option | Conditional Statement | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Errors Route | `if (exists(message) AND message contains 'error')` | | Exceptions Route | `if (exists(message) AND message contains 'exception')` | | Negative Sentiment Route | `if (exists(message) AND (message contains 'abort' OR message contains 'broken' OR message contains 'caught' OR message contains 'denied' OR message contains 'exception' OR message contains 'fail' OR message contains 'insufficient' OR message contains 'killed' OR message contains 'malformed' OR message contains 'outofmemory' OR message contains 'panic' OR message contains 'timeout' OR message contains 'undefined' OR message contains 'unsuccessful' OR message contains 'unavailable'))` | #### 6 - Event to Metrics Processors Each of these processors is used to count the type of message event sent to it, and produce an incremental metric for that type. **Error Metrics** | Option | Setting | | -------------------- | -------------------------------------- | | Metric Name | `error_monitoring` | | Kind | `Incremental` | | Type | `Counter` | | Value/Value Type | `New value` | | Value/Value | `1` | | Namespace/Value Type | `None` | | Tags/Name | `container_name` | | Tags/Value Type | `Value from Event Field` | | Tags/Field Value | `metadata.fields."k8s.container.name"` | **Negative Sentiment Metrics** | Option | Setting | | -------------------- | -------------------------------------- | | Metric Name | `negative_sentiment_monitoring` | | Kind | `Incremental` | | Type | `Counter` | | Value/Value Type | `New value` | | Value/Value | `1` | | Namespace/Value Type | `None` | | Tags/Name | `container_name` | | Tags/Value Type | `Value from Event Field` | | Tags/Field Value | `metadata.fields."k8s.container.name"` | **Exceptions Metrics** | Option | Setting | | -------------------- | -------------------------------------- | | Metric Name | `exception_monitoring` | | Kind | `Incremental` | | Type | `Counter` | | Value/Value Type | `New value` | | Value/Value | `1` | | Namespace/Value Type | `None` | | Tags/Name | `container_name` | | Tags/Value Type | `Value from Event Field` | | Tags/Field Value | `metadata.fields."k8s.container.name"` | #### 7 - Enrich Ops Tags All the processed data is sent to the final Processor in the chain, the [Script Execution Processor](/telemetry-pipelines/js-script-processor), which adds descriptive information to the data to identify where and how it was processed. ```bash theme={null} // Modify the event using a subset of the JavaScript language. // The function must return the modified event function processEvent(message, metadata) { message.tags.pipeline_owner = '' message.tags.pipeline_name = '' message.tags.pipeline_url = ' Image ## 1 - Demo/HTTP Source Use the [HTTP Source](/telemetry-pipelines/http-source) to connect the Pipeline to your incoming telemetry data. The topic [Set Up and Test an HTTP Endpoint Source](/telemetry-pipelines/set-up-and-process-http-endpoint-data) includes tips and examples for configuring your source. This example uses the [Demo Logs](/telemetry-pipelines/demo-logs-source) with the **JSON Logs** option to demonstrate the effects of the Processors on the data stream. You can also [try it out with a sample of your own data. ](/telemetry-pipelines/view-pipeline-data) 1. Log into the Mezmo App, and in the **Pipelines** section, click **New Pipeline**. > 2. Add the **Demo Logs** Source, and for **Format**, select **JSON**. > 3. Add the **Blackhole** Destination to your Pipeline, and connect it to the Demo Logs. > 4. Add the Processors and their configurations as shown in this example.> 5. To view the data transformations through the Processors, **Deploy** the Pipeline, and then click the **Tap** for the Source and each Processor to see the data as it egresses from each node. You will also be able to see how the data is reduced on the Pipeline Dashboard.> > If you don't yet have a Mezmo account, you can [sign up for a 30 Day Free Trial](https://www.mezmo.com/sign-up-pipeline-today) to try us out! ## 2 - Route Processor The [Route Processor](/telemetry-pipelines/route-processor) enables you to set conditions under which telemetry data will be sent to other points in the processing chain. In this case, it is set to send 200 events down the Processor Chain for conversion to metrics, while unmatched data is sent directly to the Destination. This example uses the Blackhole destination, where all data is dropped, but you could send matched and unmatched data to different destinations depending on your use case. | Configuration Parameter | Setting | | ----------------------------------- | ----------------------- | | Conditional Statement for 200 Route | `if(.status equal 200)` | ## 3 - Event to Metric Processor The [Event to Metric Processor](/telemetry-pipelines/event-to-metric-processor) enables you implement a counter for the events sent to it, and attach tags to specified fields. In this case, the tags are sent to capture the values related to the URL and IP Address within the 200 event. | Configuration Parameter | Setting | | ------------------------- | ------------------------ | | Metric Name | `number_hits` | | Kind | `Incremental` | | Type | `Counter` | | Type/Value Type | `New Value` | | Type/Value | `1` | | Type/Namespace/Value Type | `None` | | Tag 1/Name | `url` | | Tag 1/Value Type | `Value from event field` | | Tag 1/Field Value | `.host` | ## 4 - Aggregate Metrics Processor The final Processor in the chain, the [Aggregate Processor](/telemetry-pipelines/aggregate-processor) aggregates multiple metric events into a single metric based on a defined time interval. In this case, it aggregates the value of the 200 metrics over a 10 second interval into a single number. | Configuration Parameter | Setting | | ----------------------- | ------------ | | Interval | `10 seconds` | ## 5 - Blackhole Destination The [Blackhole](/telemetry-pipelines/blackhole-destination) Destination drops all data sent to it. This makes it useful for testing your Processor chain to make sure you are getting the expected results before sending them on to a production destination. Mezmo supports a wide variety of popular destinations including [Mezmo Log Analysis](/telemetry-pipelines/mezmo-destination), [Datadog Metrics](/telemetry-pipelines/datadog-metrics-destination), and [Prometheus Remote Write](/telemetry-pipelines/prometheus-remote-write-destination). ## For More Information For more information on how to understand and optimize your telemetry data, [contact our Solutions Engineering team](https://go.mezmo.com/mezmo-data-profiling?_gl=1*189zkyo*_ga*NDQxOTc0Mzg1LjE2NDE0MTYxODc.*_ga_C3EJ23NJFV*MTcxMTU3ODkyNi45OC4xLjE3MTE1Nzg5MzIuMC4wLjA.) to schedule a free consultation. # Tutorial: Route Data Source: https://docs.mezmo.com/practioner-guide-data-optimization/pipeline-module--route Build a routing group that uses a Route Processor and conditional statements to send financial, JSON, and Apache data down separate processing chains. In more complex architectures, you will often have several Sources feeding into the same Pipeline, with the data for each needing different types of processing before being sent to multiple destinations. A key component of these Pipelines is a Route Processor, which uses conditional statements to match data and send it along its particular processing route. This topic describes a typical use of a Route Processor, with examples of the Processor configurations. ## Overview This schematic illustrates the configuration of a Routing group, which includes a Script Execution Processor to format raw strings to JSON, that routes different data types from several sources through specialized processing chains to several destinations. ## 1 - Sources The Sources represent three different types of data flowing through the Pipeline that need to be routed to separate processing chains: 1. **Financial Data** that needs to have Personally Identifying Information encrypted before being sent to storage and the observability tool. 2. **JSON Data** that needs to have Status - 200 **events** routed and dropped. 3. **Apache Errors** that need to be converted to JSON format and all info messages dropped. 1) Log into the Mezmo App, and in the **Pipelines** section, click **New Pipeline**.> 2. Add three **Demo Logs** Sources, and for **Format**, select 1) **Financial Data** 2) **JSON** 3) **Apache Errors**. > 3. Add three **Blackhole** Destinations to your Pipeline to represent 1) **Drop** 2) **Storage** 3) **Observability Tool**. > 4. Add the Processors and their configurations as shown in this example.> 5. To view the data transformations through the Processors, **Deploy** the Pipeline, and then click the **Tap** for the Source and each Processor to see the data as it egresses from each node. > > If you don't yet have a Mezmo account, you can [sign up for a 30 Day Free Trial](https://www.mezmo.com/sign-up-pipeline-today) to try us out! ## 2 - Script Execution Processor The [Script Execution Processor](/telemetry-pipelines/js-script-processor) is configured to convert the Apache errors from raw strings to JSON format. ```bash theme={null} function junk(message) { var new_message = {} new_message.message = message return new_message } ``` ## 3 - Route Processor The [Route Processor](/telemetry-pipelines/route-processor) uses three conditional statements to identify and route specific components of all three data types: #### Apache Info Messages ```javascript theme={null} if (exists(.message) AND .message contains 'INFO') ``` Because both the JSON and Apache errors data contain .message fields, this statement uses AND to make sure that .messages that don't contain the INFO event won't generate a "field not found" error. All messages meeting this criteria are sent to the Drop Destination. #### Status 200 Events ```javascript theme={null} if (exists(.status) AND .status equal 200) ``` All events that meet these criteria are sent to the Drop Destination. #### Transaction Events ```bash theme={null} if (exists(.event) AND .event equal 'transaction') ``` All events that meet these criteria are send to the Encrypt Processor. ## 4 - Encrypt Processor Because transaction events contain Personally Identifying Information (PII), such as credit card numbers, this information needs to be encrypted before being sent to storage and observability tools. For more information, check out the topic [Tutorial: Mask and Encrypt Data](/practioner-guide-data-optimization/pipeline-module--security-and-compliance). **Encrypt Processor Configuration** | Configuration Field | Details | | -------------------------------- | ------------------------------------------------- | | Field | .transaction.cc.cc\_number | | Encryption algorithm | AES-256-CFB (key=32 characters, iv-16 characters) | | Encryption key | zipadeedoodah012zipadeedoodah013 | | Initialization vector (IV) field | .creditcardnumber | ## 5 - Destinations The routed data is sent to three destination, represented in this schematic by the [Black Hole Destination](/telemetry-pipelines/blackhole-destination): 1. **Drop**, where the unnecessary INFO and Status - 200 messages are sent. 2. **Storage**, where all unmatched data and encrypted PII data is sent. 3. **Observability Tool,** where all unmatched data and encrypted PII data is sent. The [Blackhole](/telemetry-pipelines/blackhole-destination) Destination drops all data sent to it. This makes it useful for testing your Processor chain to make sure you are getting the expected results before sending them on to a production Destination. Mezmo supports a wide variety of popular Destinations including [Mezmo Log Analysis](/telemetry-pipelines/mezmo-destination), [Datadog Metrics](/telemetry-pipelines/datadog-metrics-destination), and [Prometheus Remote Write](/telemetry-pipelines/prometheus-remote-write-destination). # Tutorial: Mask and Encrypt Data Source: https://docs.mezmo.com/practioner-guide-data-optimization/pipeline-module--security-and-compliance Build a compliance processor group that routes events to Encrypt Field and Redact processors to protect PII like card numbers and user IDs. ## The Situation This Pipette depicts the typical configuration of a Data Compliance processor group within a Telemetry Pipeline. In this situation, the objectives are to send financial transaction and account access data to a storage location in case it is needed for later analysis, and to redact or encrypt Personally Identifying Information (PII). This group uses the [Route Processor](/telemetry-pipelines/route-processor) to send specific events to the [Redact Processor](/telemetry-pipelines/redact-processor) and the [Encrypt Field Processor](/telemetry-pipelines/encrypt-fields-processor), before storage, to obfuscate user IDs and credit card numbers, and to also enable the decryption of credit card numbers in case they are needed for specific analysis. ## Overview This schematic of the Pipette illustrates the Processor chain for redacting and encrypting Personally Identifying Information focusing on login User IDs and credit card numbers. The Processor configurations are described in detail in the sections that match the numbers in the schematic. Image ## 1 - Demo Logs Source This Pipette uses the [Demo Logs](/telemetry-pipelines/demo-logs-source) with the **Financial Data** option to send a sample of data containing PII through the Processor chain. 1. Log into the Mezmo App, and in the **Pipelines** section, click **New Pipeline**. > 2. Add the **Demo Logs** Source, and for **Format**, select **Financial Data**. > 3. Add the **Blackhole** Destination to your Pipeline, and connect it to the Demo Logs. > 4. Add the Processors and their configurations as shown in this example.> 5. To view the data transformations through the Processors, **Deploy** the Pipeline, and then click the **Tap** for the Source and each Processor to see the data as it egresses from each node. You will also be able to see how the data is reduced on the Pipeline Dashboard.> > If you don't yet have a Mezmo account, you can [sign up for a 30 Day Free Trial](https://www.mezmo.com/sign-up-pipeline-today) to try us out! ## 2 - Route Processor The [Route Processor](/telemetry-pipelines/route-processor) enables you to set conditions under which telemetry data will be sent to other points in the processing chain. In this case, it filters three types of events from the incoming data for processing: Access, Transaction, and Boot-up. Any events that don't match these three types are sent directly to the storage location. | Configuration Parameter | Setting | | -------------------------------------------- | --------------------------------- | | Conditional Statement for Boot-up Events | `if (.event equal 'bootup')` | | Conditional Statement for Transaction Events | `if (.event equal 'transaction')` | | Conditional Statement for | `if (.event equal 'access')` | ## 3 - Encrypt Processor The transaction events contain credit card information that should be redacted or encrypted before being sent to storage. In this case, since the credit card numbers may be needed later, for example for fraud analysis, the [Encrypt Field Processor](/telemetry-pipelines/encrypt-fields-processor) is set to encrypt the card numbers, so that they can later be decrypted using the encryption key. | Configuration Parameter | Setting | | --------------------------------------------- | -------------------------------------------------- | | Field | `.transaction.cc.cc_number` | | Encryption algorithm | `AES-256-CFB(key=32 characters, iv=16 characters)` | | Encryption key | `zipadeedoodah777zipadeedoodah888` | | Initialization vector (IV) field | `.creditcardnumber` | | Encode encrypted field and IV as Base 64 text | On | ## 4 - Redact Processor Information that is redacted is obfuscated completely, and cannot be recovered after processing. For this reason, the [Redact Processor](/telemetry-pipelines/redact-processor) should be used to remove PII that is particularly sensitive, but doesn't need to be used for later analysis. In this case, the login User ID from Access events is redacted, since this is information that could be used to hack user accounts, but isn't needed for analysis. The Processor operation is based on searching for specific patterns, such as social security numbers or email addresses, or custom patterns, and then using a hash or replacement pattern to obfuscate the data. In this case, it searches the field`.access.user_id` for a custom pattern based on a regular expression, and then hashes it using the md5 algorithm. | Configuration Parameter | Setting | | ----------------------- | ----------------- | | Field | `.access.user_id` | | Redact Pattern | `Custom Pattern` | | Action | `Hash` | | Algorithm | `md5` | | Expression | `[a-zA-Z0-9@.]+` | ## 5 - Blackhole Destination The [Blackhole](/telemetry-pipelines/blackhole-destination) Destination drops all data sent to it. This makes it useful for testing your Processor chain to make sure you are getting the expected results before sending them on to a production destination. Mezmo supports a wide variety of popular destinations including [Mezmo Log Analysis](/telemetry-pipelines/mezmo-destination), [Datadog Metrics](/telemetry-pipelines/datadog-metrics-destination), and [Prometheus Remote Write](/telemetry-pipelines/prometheus-remote-write-destination). In this case, note that the data volume from the Source to the Destination has increased by almost 22%. It's typical for data volume to increase with these Processors because they add characters to the message strings. However, fine tuning of the algorithms and encryption keys can limit the increase in data volume. ## For More Information For more information on how to implement security modules for your Pipeline data management needs, [contact our Solutions Engineering team](https://go.mezmo.com/mezmo-data-profiling?_gl=1*189zkyo*_ga*NDQxOTc0Mzg1LjE2NDE0MTYxODc.*_ga_C3EJ23NJFV*MTcxMTU3ODkyNi45OC4xLjE3MTE1Nzg5MzIuMC4wLjA.) to schedule a free consultation. # Pipeline Examples and Tutorials Source: https://docs.mezmo.com/practioner-guide-data-optimization/reference-architectures-for-data-optimization-pipelines Index of example Mezmo Telemetry Pipelines and data optimization tutorials, including processor configuration settings. These topics include examples of Mezmo Telemetry Pipelines and tutorials for data optimization techniques, including configuration settings for Processors. # About Mezmo Flow Source: https://docs.mezmo.com/telemetry-pipelines/about-mezmo-flow Learn how Mezmo Flow profiles your telemetry data and recommends Processors to build a volume-reducing Pipeline in four steps. Mezmo Flow provides an easy onboarding experience focused on helping you gain an understanding of your data, and then recommending Processors based on common patterns and message types. With Mezmo Flow, you're four steps away from creating a telemetry data Pipeline that will substantially reduce the volume of telemetry data sent to your storage locations and observability tools, saving both on costs and the mental toil required to optimize your data for your observability requirements. 1. Mezmo Flow begins when you [create your organization](/docs/organization-management-overview) in the Mezmo Web App, and then [set up a Data Source](/telemetry-pipelines/supported-telemetry-pipeline-sources) to start sending your data to Mezmo. Mezmo Flow will also automatically set up [Mezmo Log Analysis](/telemetry-pipelines/mezmo-destination) as the data Destination for your Pipeline. 2. As the telemetry data from your Source is ingested, the [Data Profiler](/telemetry-pipelines/data-profiling) will analyze it and present you with an overview of the most common message patterns, and metrics for the apps and hosts that are generating the most log data. 3. From there you can select [Processors ](/telemetry-pipelines/supported-processors) to apply to message patterns, and see the way in which each Processor affects the reduction of your overall log volume. 4. When you're satisfied with the results, you can apply your selected Processors to the telemetry data, and Mezmo Flow will generate a Pipeline that includes your selected Processors. Once your Pipeline is active, you can use the [Pipeline Tap](/telemetry-pipelines/view-pipeline-data) feature to examine the transformations to the data in stream and make changes to the Processor configurations as needed. Your Pipeline will also include the [Data Profiler Processor](/telemetry-pipelines/data-profiler-processor) that you can use to examine your data profile, or generate a new one after making changes to your Source or your Processor configurations. # About Mezmo Telemetry Pipelines Source: https://docs.mezmo.com/telemetry-pipelines/about-mezmo-telemetry-pipelines Overview of how Mezmo Telemetry Pipelines ingest, process, route, and monitor telemetry data across diverse sources and destinations. Mezmo Telemetry Pipeline enables organizations to take a new approach towards managing telemetry data. * Incorporate data from your existing Log Analysis environment, or from entirely new data streams like [AWS Kinesis Firehose](/telemetry-pipelines/kinesis-firehose-source), and the [Mezmo Agent](/telemetry-pipelines/mezmo-agent-source) to capture a diverse range of telemetry data, with new sources being added regularly * You can use Processors to [encrypt](/telemetry-pipelines/encrypt-fields-processor), [filter](/telemetry-pipelines/filter-processor), [route](/telemetry-pipelines/route-processor), and automate other transformations of your data to make sure that it is tailored to your specific needs for storage and analysis * Send your processed data to destinations like [AWS S3 Storage](/telemetry-pipelines/s3-destination), [ElasticSearch](/telemetry-pipelines/elasticsearch-destination), and [Mezmo Log Analysis](/telemetry-pipelines/mezmo-destination) so you always know that you will have the data you need in the right place at the right time * Use [Pipeline Taps](/telemetry-pipelines/monitor-data-pipelines) to monitor the flow of data in your Pipeline, and [sample Pipeline data in real-time](/telemetry-pipelines/view-pipeline-data) to use in constructing new Pipelines ## Video Overview