Request Lifecycle

Overview

AURA manages per-request state (cancellation tokens, subscriptions) across streaming SSE connections. This document covers the lifecycle, timeout configuration, and known limitations.

Request Flow

Client POST /v1/chat/completions
         │
         ▼
┌─────────────────────────────────────┐
│ Shutdown middleware check           │
│   (503 if shutdown_token cancelled) │
└─────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────┐
│ Generate request_id (UUID)          │
└─────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────┐
│ Spawn producer task                 │
│   - Register cancellation token     │
│   - Subscribe to progress events    │
│   - Subscribe to tool events        │
└─────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────┐
│ Stream chat with TimeoutHook        │
│   - Tool calls set thread context   │
│   - Tool results clear context      │
└─────────────────────────────────────┘
         │
         ├──────────────────┬──────────────────┐
         ▼                  ▼                  ▼
┌────────────────┐  ┌──────────────────┐  ┌──────────────────┐
│ Normal         │  │ Timeout/         │  │ Shutdown         │
│ completion     │  │ Disconnect       │  │ (grace expired)  │
└────────────────┘  └──────────────────┘  └──────────────────┘
         │                  │                  │
         │                  ▼                  ▼
         │          ┌──────────────────┐  ┌──────────────────┐
         │          │ Cancel token     │  │ Cancel hook +    │
         │          │ Send MCP cancel  │  │  registry        │
         │          │ Evict from pool  │  │ Send [DONE]      │
         │          └──────────────────┘  │ Then MCP cleanup │
         │                  │             │ (no pool evict)  │
         │                  │             └──────────────────┘
         └──────────┬───────┴──────────────────┘
                    ▼
┌─────────────────────────────────────┐
│ Cleanup (RAII guard)                │
│   - Unregister cancellation         │
│   - Unsubscribe progress            │
│   - Unsubscribe tool events         │
└─────────────────────────────────────┘

Timeout Configuration

Production Defaults

Setting	Default	Env Variable	Purpose
First chunk timeout	90 sec	`FIRST_CHUNK_TIMEOUT_SECS`	Max wait for first provider chunk
Stream timeout	15 min	`STREAMING_TIMEOUT_SECS`	Max request duration
Stream inactivity timeout	Disabled (`0`)	`STREAM_INACTIVITY_TIMEOUT_SECS`	Max silence between stream items after the first chunk
Shutdown grace period	30 sec	`SHUTDOWN_TIMEOUT_SECS`	Time for in-flight requests to finish on shutdown
Heartbeat	15 sec	—	Disconnect detection

Rationale

First chunk timeout (90 sec): Catches provider connection failures early. If the LLM hasn’t sent any data within this window, the request is aborted rather than hanging for the full stream timeout. The window is sized for reasoning models, which can legitimately take over a minute before the first chunk.
Stream inactivity timeout (disabled by default): Closes the gap between the first-chunk timeout, which guards only the opening chunk, and the stream timeout, which is the whole-request budget. Once streaming starts, a provider that goes silent between chunks runs all the way to the stream timeout by default. Enable this to fail a stalled stream sooner.
Stream timeout (15 min): Supports long-running MCP tools. Set to 0 to disable (not recommended).
Heartbeat (15 sec): Standard SSE keepalive. Detects disconnect during silent tool execution.

Tuning Stream Inactivity Timeouts

There are two knobs at two layers, both disabled by default (0). The TOML stream_inactivity_timeout_secs (in [orchestration.timeouts], see configuration reference) governs coordinator and worker orchestration streams. The server STREAM_INACTIVITY_TIMEOUT_SECS (see web server reference) governs single-agent streaming requests. The TOML knob exempts tool execution from its timer. The TOML deadline suspends during tool execution and re-arms on each stream item, so slow MCP tool calls and paused HITL approvals do not trip it. This makes it the reliable knob for orchestrated deployments. When the timer fires, the behavior depends on the layer. For orchestration, the affected task fails with a no-stream-progress error (an agent-timeout failure) and the run completes with partial results instead of dying mid-wave. For the server single-agent path, the request terminates as a timeout.

Orchestrated worker activity reaches the server layer as tool and progress events. Those events re-arm the server deadline but cannot suspend it, so a worker tool call that goes quiet past the window can still trip the server-layer timeout. For orchestrated deployments, rely on the TOML stream_inactivity_timeout_secs. If you also set the server STREAM_INACTIVITY_TIMEOUT_SECS, size it above the TOML window.

Aura pins OpenAI to Chat Completions, which streams nothing while the model is thinking, so a long think reads as inactivity and can trip the deadline. Anthropic, Gemini, and OpenAI-compatible reasoning_content streams re-arm while thinking, so this affects first-party OpenAI only. Size the window above worst-case think time for those models, or leave the knob off.

Tool Event Correlation

Rig spawns tool execution in separate tokio tasks. The hook context (where the LLM decides to call a tool) and the execution context (where MCP actually runs) are decoupled, requiring a mechanism to correlate tool_call_id across these boundaries — AURA does this with a per-request FIFO queue. This relies on Rig’s streaming mode executing tools sequentially within a request; see Rig Fork Changes for the validation methodology if you’re tracking Rig upstream compatibility.

Cleanup Mechanism

Request resources (cancellation registration, progress/tool-event subscriptions) are cleaned up via an RAII guard that ensures cleanup runs even on panic. If the async runtime is already gone (process exit), cleanup is skipped — resources are reclaimed with the process.

MCP Cancellation

On client disconnect or timeout:

The cancellation token signals all waiting code
Aura sends notifications/cancelled to MCP servers
The agent is evicted from its connection pool to prevent stale connections

MCP servers receive the cancellation notification and can abort in-progress operations.

Graceful Shutdown

The server uses a two-phase shutdown with separate cancellation tokens:

Token	Cancelled	Purpose
`shutdown_token`	Immediately on signal	Middleware rejects new requests with 503
`stream_shutdown_token`	After `SHUTDOWN_TIMEOUT_SECS` grace period	Terminates remaining in-flight streams

Shutdown Sequence

SIGTERM/SIGINT received
Phase 1 (immediate): shutdown_token cancelled — middleware returns 503 for all new requests
Grace period: In-flight streams continue running for up to SHUTDOWN_TIMEOUT_SECS (default 30s). Streams that complete naturally during this window are unaffected.
Phase 2 (drain): stream_shutdown_token cancelled — remaining streams:
- Cancel hook + request registry (stops in-flight MCP tool execution)
- Send [DONE] to client (before MCP cleanup, so client gets clean termination)
- Run MCP cleanup (send notifications/cancelled, close connections)
- No pool eviction (pool is dying with the server)
Server stops: Workers have 10s to complete Phase 2 cleanup

Shutdown vs Disconnect/Timeout

Behavior	Disconnect/Timeout	Shutdown
Pool eviction	Yes	No (pool is dying)
`[DONE]` timing	After MCP cleanup (Timeout) / skipped (Disconnect)	Before MCP cleanup
Grace period	None — immediate cancel	Configurable (`SHUTDOWN_TIMEOUT_SECS`)

Getting Started

Configuration

Features

CLI Reference

Web Server Reference

Overview

Request Flow

Timeout Configuration

Production Defaults

Rationale

Tuning Stream Inactivity Timeouts

Tool Event Correlation

Cleanup Mechanism

MCP Cancellation

Graceful Shutdown

Shutdown Sequence

Shutdown vs Disconnect/Timeout

​Overview

​Request Flow

​Timeout Configuration

​Production Defaults

​Rationale

​Tuning Stream Inactivity Timeouts

​Tool Event Correlation

​Cleanup Mechanism

​MCP Cancellation

​Graceful Shutdown

​Shutdown Sequence

​Shutdown vs Disconnect/Timeout

Overview

Request Flow

Timeout Configuration

Production Defaults

Rationale

Tuning Stream Inactivity Timeouts

Tool Event Correlation

Cleanup Mechanism

MCP Cancellation

Graceful Shutdown

Shutdown Sequence

Shutdown vs Disconnect/Timeout