Parsed Log Line Components

Mezmo ingestion automatically parses your log lines based on key components that you can then use to search and analyze your log data

📘

LogDNA is now Mezmo!

LogDNA has recently become Mezmo. As you access technical resources like our API, Code Libraries, and GitHub repositories, you will continue to see references to LogDNA for a short time as we update our new name across all our resource channels.

As Mezmo ingests your logs, it automatically parses information from your log lines, including string components, source information, application information, JSON objects, and user-specified metadata. This topic describes the various types of information that Mezmo parses, along with notes on how it is parsed.

🚧

Consistent Value Types Expected

If your parsed fields contain inconsistent value types, field parsing may fail, but the line will be preserved if possible. For example, if a line is passed with a meta object, such as meta.myfield of type String, any subsequent lines with meta.myfield must have `String as the value type. This applies to all parsed fields, including JSON.

Log Line String Components

Most log line strings contain three components: Message, Timestamp, and Log Level.

Message

Message is a string that represents the core descriptive component of a log line. It is usually preceded by timestamp and log level. A message typically contains a mixture of static and variable substrings, and is human-readable. For example:

User [email protected] requested /API/accountdetails/

Timestamp

Timestamp is required for all ingested log lines. For Mezmo log ingestion to correctly parse a timestamp, it should follow the ISO 8601 format.

Log Level

Log level typically follows timestamp and is automatically parsed. Mezmo log ingestion parses common log level formats, such as a timestamp followed by a separator followed by the log level. Common log levels include:

  * CRITICAL
  * DEBUG
  * EMERGENCY
  * ERROR
  * FATAL
  * INFO
  * SEVERE
  * TRACE
  * WARN
  * ALERT

Source Information Metadata

Mezmo also parses source information metadata from loglines, which is listed in the All Sources menu in the web app. The only required parameter is hostname.

Hostname

A hostname is the name of the log line source, and is automatically parsed by [the Mezmo Agent](doc:introducing-the-agent}, as well as Syslog-based ingestion. However, when you are sending log lines for ingestion with the REST API or a code library , you must specify the host name.

Tags

You can use a tag to group lines, and more than one tag can be applied to a single line. Tags are listed in the All Tags menu in the web app. Tagging is supported by both the [the Mezmo Agent](doc:introducing-the-agent} as well as custom-template supported Syslog-based ingestion, such as rsyslog or syslog-ng.

Other information

Other optional source information includes:

  * IP address
  * MAC address

The Mezmo Agent automatically parses this information, and you specify it for the REST API. The Mezmo Agent also parses some instance metadata, such as instance type.

Application Information Metadata

In addition to source information, LogDNA can also parse application information from log lines. The [Mezmo Agent](doc:introducing-the-agent} automatically parses the application name as the filename (for example: error.log) while Syslog-based ingestion uses the syslog-generated APP-NAME tag. For the REST API and code libraries, you must specify the app name.

Automatic and Custom Parsing for Field Search

Mezmo automatically parses certain types of log lines that enable the use of field search for those lines. For a list of supported log types, check out the topic for Auto-Parsing Logs, while Custom Parsing & How to Parse Logs provides more information about custom configurations.

JSON Parsing

📘

Data Size Measurement for JSON Parsing

Be aware that the size of sent log data can increase after the JSON string is parsed in Node.js. Measurement is based on how much data is ingested into Mezmo, after it is parsed as JSON, and not how much data is sent in a line.

As long as the log message ends with a right curly-brace character, } , Mezmo will parse your last JSON object in the log message, even if the JSON object does not span the entire message. If you don’t want Mezmo to parse your JSON object, you can append an additional character after the right curly brace, such as a period: }.

If your JSON contains a message field, Mezmo will use that field for display and search in the log viewer. If you include a level field, Mezmo will also parse out, and override any existing, log levels.

Reserved and Protected Fields

For JSON parsed lines, Mezmo uses a number of reserved fields to keep track of specific types of data. Using the reserved fields in your root JSON object will result in an underscore, _ , prepended to those fields inside the context menu. For example, status is stored as _status. However, for the purpose of searching in the Mezmo web app, you don’t need to include the underscore character to search for these fields. For example, if you search for status:200, this will return results for both status and ``_status```.

These are common reserved fields:

  * _source
  * _type
  * auth
  * bytes
  * connect
  * method
  * namespace
  * path
  * pod
  * request
  * response
  * service
  * space
  * status
  * timestamp
  * user

Protected field names cannot be used in your object, and are removed by Mezmo when encountered. The protected field names are:

  * _account
  * _retention

Metadata

Metadata is a field reserved for custom information associated with a log line. Sending metadata is currently supported by the REST API ), as well as our Node.JS, and Python code libraries.