Build a Mezmo Telemetry Pipeline with Terraform
This feature is in Beta development.
Introduction
Terraform is an infrastructure-as-code tool from Hashicorp that lets you define, create, and manage resources using a code-based approach. Mezmo has implemented a Terraform provider that gives you the ability to build and manage your telemetry data Pipelines with Terraform. This makes managing your pipelines easier and simpler by allowing you to:
- Store your Pipelines as code
- Version your Pipelines in your source control
- Have multiple users collaborate on Pipelines outside of a user interface
- Quickly and easily replicate Pipelines
Getting Started
You will need:
- A Pipeline Service Key. Create a new Pipeline Service Key by accessing Settings > API Keys > Pipeline Service Keys.
- A local Terraform installation for your system
Once have these items, you can add the Mezmo Terraform provider by referencing the provider in any TerraForm project using this code:
terraform required_providers mezmo = source = "registry.terraform.io/mezmo/mezmo" version = "4.3.0" provider "mezmo" # Configuration options auth_key = "<YOUR TOKEN>"Unitll you are comfortable working with Mezmo Terraform for Pipelines, we recommend you start a new Terraform project for your Pipelines that is separate from any existing Terraform installations you may have.
You will need to add the configuration option for your pipeline service key to authenticate and use the Terraform provider API. Use the auth_key = "<YOUR TOKEN>" configuration option in the provider configuration object as shown here:
provider "mezmo" auth_key = "<YOUR TOKEN>"Your First Terraform Pipeline
Now that you've installed the Mezmo Terraform provider, you can generate Pipelines directly within your account based on the generated Pipeline service key. In the public repository on github.com/mezmo, you will find examples of Pipelines and other resources you can use to get started.
We are showing the example in a single file, but we generally recommend splitting it out into individual files for your providers, sources, destinations, pipelines, and variables.
This is is a simple placeholder example you can try out, which will then reflect in your environment.
terraform required_providers mezmo = source = "registry.terraform.io/mezmo/mezmo" required_version = ">= 1.1.0"provider "mezmo" auth_key = "<YOUR TOKEN>"resource "mezmo_pipeline" "pipeline1" title = "My Terraform pipeline"resource "mezmo_http_source" "source1" pipeline_id = mezmo_pipeline.pipeline1.id title = "My HTTP source" description = "This receives data from my webhook" decoding = "json"resource "mezmo_http_source" "shared_source" pipeline_id = mezmo_pipeline.pipeline1.id title = "A shared HTTP source" description = "This source can be used across pipelines" decoding = "json" gateway_route_id = mezmo_http_source.source1.gateway_route_idresource "mezmo_sample_processor" "processor1" pipeline_id = mezmo_pipeline.pipeline1.id title = "My second sample processor" description = "Let's sample some data while keeping other events intact" inputs = mezmo_http_source.shared_source.id rate = 100 always_include = field = ".my_app_id" operator = "greater" value_number = 10 Once you've created your example, run:
terraform initterraform planTerraform will examine the local state and compute the necessary changes to enact your configuration. The plan command does not execute the configuration changes, but computes how they would be enacted. Practically this means that Terraform will list the changes to apply. You should see an output similar to this example:
# mezmo_http_source.shared_source will be created + resource "mezmo_http_source" "shared_source" { + capture_metadata = false + decoding = "json" + description = "This source can be used across pipelines" + gateway_route_id = (known after apply) + generation_id = (known after apply) + id = (known after apply) + pipeline_id = (known after apply) + title = "A shared HTTP source" } # mezmo_http_source.source1 will be created + resource "mezmo_http_source" "source1" { + capture_metadata = false + decoding = "json" + description = "This receives data from my webhook" + gateway_route_id = (known after apply) + generation_id = (known after apply) + id = (known after apply) + pipeline_id = (known after apply) + title = "My HTTP source" } # mezmo_pipeline.pipeline1 will be created ~ resource "mezmo_pipeline" "pipeline1" { ~ created_at = (known after apply) ~ id = (known after apply) ~ title = "My Terraform pipeline" } # mezmo_sample_processor.processor1 will be created ~ resource "mezmo_sample_processor" "processor1" { + always_include = { + field = ".my_app_id" + operator = "greater" + value_number = 10 } ~ description = "Let's sample some data while keeping other events intact" ~ generation_id = (known after apply) ~ id = (known after apply) ~ inputs = [ + (known after apply), ] ~ pipeline_id = (known after apply) ~ rate = 100 ~ title = "My second sample processor" } Plan: 4 to addVerify that the plan returns the expected changes. If this is your first Pipeline, it will list creation of all of the resources.
Once you're ready to deploy your Terraform Pipeline, use the apply command. This command results in the execution of the computed changes between your Terraform file and your environment. You will have to confirm you want to make the changes to your environment.
terraform applyYou've just created your first Terraform pipeline! However, be aware the Pipeline is not deployed automatically. You will need to log into the the Mezmo Web App to test and deploy your Pipeline.
Once you've created your first Pipeline, we recommend studying the documentation found in our repository to learn more about the available resources, including Sources, Processors, and Destinations.
State is an important concept in Terraform. Terraform assumes that you will only manage your resources via Terraform itself. This means if you edit your configuration anywhere else than Terraform, you risk getting out of sync with the state of your resources. If you try to edit your Pipelines in the Mezmo UI, a similar issue can occur, because your resources may become out of sync with your Terraform state. This can lead to unexpected behavior. For this reason, we've restricted editing Pipelines created by Terraform in the UI. This way you can view and reference the Pipelines in the UI, but limit the risk of having unexpected changes because of the Terraform state. .