Skip to main content

​
Enabling tracing

OpenTelemetry support is enabled by default via the otel feature. Configure your OTLP endpoint using standard environment variables (for example OTEL_EXPORTER_OTLP_ENDPOINT) to export traces. No endpoint set means no OTel layer is installed. AURA emits spans using the OpenInference semantic convention (llm.*, tool.*, input.*, output.*) rather than the gen_ai.* conventions. Any gen_ai.* attributes from underlying provider libraries (Rig.rs) are automatically translated to OpenInference equivalents at export time. This makes AURA traces natively compatible with Phoenix and other OpenInference-aware observability tools.

​
Trace structure

Every request produces two traces:
  1. HTTP trace: covers the request/response lifecycle
  2. Agent trace: covers the LLM/tool execution
The agent trace is rooted at agent.stream with parent: None so Phoenix sees it as an independent trace root with all LLM I/O attributes.

​
HTTP trace (both modes)

​
Single-agent mode

​
Orchestration mode

​
Span attributes

​
Agent root (agent.stream)

user.id, session.id, metadata, input.value, output.value, llm.token_count.prompt, llm.token_count.completion, llm.token_count.total

​
Orchestration spans

The agent.turn spans under each orchestration phase (planning and worker) record token usage (llm.token_count.*). The phase spans themselves don’t.

​
Tool spans (execute_tool)

For HITL-gated tool calls, execute_tool carries a decision_id attribute whose value is the UUID of the approval decision that gated the call. This is the same decision_id that appears on the approval webhook payload, the SSE lifecycle events, and the approval ingress endpoint (see Human-in-the-Loop Approval Gates). The attribute is stamped before the approval outcome is known, so every gated call carries it regardless of outcome: approved, denied, timed_out, cancelled, or errored. It is stamped for gated calls only; ungated tool calls carry no decision_id attribute. decision_id sits on the execute_tool (TOOL) parent, not on the nested mcp.tool_call child, where the tool’s own status and errors land.

​
MCP Tool Call Spans

Each mcp.tool_call span identifies the tool it ran with these attributes: AURA never adds the namespace to tool.name. See Scope a Pattern to One MCP Server for how namespaces work in tool-name patterns.

​
OpenInference span kinds

AURA exports its agent-level spans as LLM spans. These are agent.stream, agent.prompt, agent.chat, orchestration.planning, and orchestration.worker. As LLM spans, they show their recorded tools and invocation parameters in the Phoenix LLM span view. Only agent.turn spans carry token counts, so Phoenix computes cost from the turns. chat, agent.prompt, and agent.chat are non-streaming entry points; the trace diagrams above cover the streaming path, which is the one AURA’s HTTP/CLI backends use.

​
Span parenting

  • agent.stream is created with parent: None to break the link from the HTTP handler trace, making it an independent trace root in Phoenix.
  • The producer task is instrumented with agent.stream so Rig’s agent.turn becomes a direct child.
  • In orchestration mode, the orchestrator instruments its spawned task with the agent.stream span so all orchestration child spans nest under the trace root.
  • Tool execution propagates the current span into its background task so mcp.tool_call nests under Rig’s execute_tool.

​
Content recording

When OTEL_RECORD_CONTENT=true, prompt/completion text and tool arguments/results are recorded as span attributes, truncated to OTEL_CONTENT_MAX_LENGTH (default 1000 bytes, rounded to a UTF-8 boundary).

​
Export at Shutdown

AURA exports a span only after the span ends. At shutdown, AURA flushes the spans that are still buffered:
  • Web server: After graceful shutdown completes, the server flushes buffered spans and waits up to 5 seconds for the export. The server first aborts any request still running after the drain window, so those spans end and are part of the flush.
  • Standalone CLI: On SIGTERM, the CLI flushes buffered spans and then exits with code 143. For details, see SIGTERM Handling. The CLI doesn’t stop a query that is still running. That query’s agent.stream root span and any other open spans never end, so AURA doesn’t export them. Spans that already ended, such as finished agent.turn and mcp.tool_call spans, arrive in your trace backend without their root.

​
Known limitations

  • Tool error propagation: Tool errors are only recorded on the mcp.tool_call child span, not on Rig’s execute_tool parent. This is intentional. mcp.tool_call is the canonical TOOL span for Phoenix.