4 - Analyze the Source Data

​

Create a Log Explorer Pipeline

1. In the Mezmo Web App, go to Pipelines and click New Pipeline.
2. Select Create a blank pipeline.
3. For Pipeline Name, enter Log Explorer.
4. Under Deployment Options, select SaaS.
5. Under Select a path, select Create a blank pipeline.
6. Click Continue.

​

Add the OpenTelemtry Log Source

1. In the Pipeline Map, click Add Source.
2. Under Shared Sources, select the OTel Log Source.
3. Click Save. The Source will be added to the Pipeline Map.

​

Add the OTel Mapping Script

1. In the Profile Map, click Add Processor.
2. Select Script Execution.
3. Copy and paste this script into the Script field.
4. Click Save.
5. Connect the Source to the Script Execution Processor.

function processEvent(message, metadata, timestamp, annotations) {
  
  let line = message
  let app = metadata.resource.attributes["container.name"]
  let host = metadata.resource.attributes["container.hostname"]
  let level = metadata.level
  
  if( app == null || app == '' ){
    app = metadata.resource["service.name"]
  }
  if( app == null || app == '' ){
    app = metadata.resource["service_name"]
  }
  if( app == null || app == '' ){
    app = metadata.scope.name
  }
  if( app == null || app == '' ){
    app = 'na'
  }

  if( host == null || host == '' ){
    host = metadata.headers["x-kafka-partition-key"]
  }
  if( host == null || host == '' ){
    host = metadata.attributes["log.file.path"]
  }
  if( host == null || host == '' ){
    host = 'na'
  }
  
  if( level == null || level == '' ){
    level = annotations.level
  }

  let new_msg = {
    "line":line,
    "app":app,
    "host":host,
    "level":level
  }

  // Extract metadata to top level fields
  for( const meta of Object.entries(metadata) ){
    let meta_name = 'metadataotel_' + meta[0].toString()
    let meta_val = meta[1]
    new_msg[meta_name] = meta_val
  }

  return new_msg
}

​

Add a Data Profiler Processor

1. In the Pipeline Map, click Add Processor.
2. Select Data Profiler, and give it the name OTel Demo Log Exploration.
3. Click Save.
4. Connect the Script Execution Processor to the Data Profiler Processor.

​

Deploy the Pipeline and View the Data Profile

1. The load-generator service is sending a huge volume of logs simply stating a homepage is being flooded. This is standard behavior of the OpenTelemetry Demo using the Feature Flag: loadgeneratorFloodHomepage , but this data is noisy and costly to retain.

2. There are unparsed events that appear to be custom Apache logs being sent from the frontend-proxy service. While these are defined in the demo code here, we can take steps to make sure this data is structured and parsed properly to be searchable in any downstream Observability system.