Module Deep Dive

Analytics & Logs

Three layers of observability — what each one records, what it deliberately does not, and how long any of it survives.

Overview

Read this first — the three layers answer different questions, and reaching for the wrong one wastes time.

Every agent run leaves evidence in three places. A run record is a structured document in the database: queryable, aggregatable, and what every dashboard is built on. A trace log is a per-run file containing the full detail of what each stage did. The application log is process-level — the layer that tells you the server itself is healthy.

They exist separately because they have incompatible constraints. A database document has a size limit and must stay small enough to aggregate across thousands of runs, so it stores previews and counts. A log file has no such limit, so it keeps complete step inputs and outputs. And the application log has to survive a crash, so it is written by a different mechanism entirely.

One rule holds across all three: no raw sensitive value is ever written. Guardrail outcomes are recorded as rule types and counts, API request headers are redacted before they are stored, and the file logger strips credentials by key name and by value pattern before anything reaches disk.

Three layers

One run, three destinations, each with its own retention.

Different size limits and different failure modes, so three mechanisms rather than one

Which layer to read

Question	Layer
What did this cost, and how many runs failed?	Run record — aggregated in dashboards
Why did this specific step produce that output?	Trace log — full step input and output
Did a user like the answer?	Run record — feedback is written back onto it
Is the server healthy, and did a connection fail?	Application log
What did a guardrail block, and how often?	Run record, plus the violations collection

Modules

The Run Record

One document per run, written to a single collection whether the run was an agent or a team. A runKind field distinguishes them, and a trigger field records what started it — a user, an API call, a channel message or the scheduler.

Ingestion is fire-and-forget by design. The record is written without blocking the response, so a slow or failing write never delays the user. The trade-off is that a rejected document is invisible to the caller: the run succeeds and simply never appears in the dashboard. That is why a rejection logs the exact field paths that failed validation — a single invalid subdocument fails the whole document, so the path is the only thing that identifies the culprit.

Status and error classification

Status is resolved in a fixed precedence: stopped beats blocked beats success. The middle case matters — a guardrail-blocked request completes the graph normally, returning a block message rather than throwing, so without an explicit check it would be recorded as a plain success.

When a run throws, the error is mapped onto one of eight stable codes rather than stored as a raw message. That is what makes "how often do we hit rate limits" a query instead of a text search.

Feedback

Thumbs up and down are written back onto the run record after the fact, keyed by message id.

Why the session id matters here. A semantic-cache hit reuses the original cached message id, so the same id can legitimately appear on more than one record — the original generation and the later cache-hit turn. Passing the session id narrows the update to the turn the user actually reacted to. Tagging zero documents is not an error; the run may not have been an analytics-logged turn at all.

Also captured per run

Eight feature flags — RAG, MCP, channel, web, API, both semantic-search paths, stopped
Token usage split by phase, not just a total
Cost frozen at write time from the model's rates
Per-direction guardrail outcome — metadata only
The static action plan, so skipped nodes still render
Team strategy, for team runs
A pointer to the run's trace log file
Partial-output metadata when a user stopped generation

Execution Detail

Alongside the summary, the record carries six arrays of per-operation detail — one per kind of thing a run can do. Each entry carries its own status, duration and timestamp, so a trace view can reconstruct the run without re-reading the log file.

toolExecutions

Each MCP tool call — name, arguments, response size, output, duration.

ragExecutions

Each retrieval attempt: how many passages cleared the relevance gate, how many queries were issued, the best score, and the combined character length.

webExecutions

Each URL fetched, with response size and outcome.

apiRegistryExecutions

The full exchange — method, URL, headers, query parameters, body, HTTP status, and a capped response preview. Secret-looking values are redacted using the same rules as the file logger.

channelExecutions

Each outbound channel action, with the channel and its id.

stepExecutions

One entry per workflow step attempt — step id, type, attempt number, which steps it consumed, the resource it used, and an output preview. For team runs it also carries the block name and kind, the containing loop, the loop pass number, and per-agent feature flags and call counts.

A bug worth knowing about, because the fix shapes the schema. Retrieval reports two different shapes: a success summarises across the passages it merged, while a failure reports an empty passage and a zero size. Marking those failure-path fields as required silently rejected every successful RAG run — and since one invalid subdocument fails the whole document, it took the entire run's analytics down with it. Only fields common to both paths are required now.

Trace Logs

Each run with logging enabled writes its own file, named from a collision-proof run identifier. Both Agent Studio and Agent Teams write to the same directory in the same format, so one parser serves both.

Levels

Six levels — trace, debug, info, warn, error, fatal — selected as a set rather than a threshold, so you can keep errors without the noise of debug. The default set is info and above. An empty set disables logging for that run entirely, which is also how the feature is switched off.

Node lifecycle events do not have to name a level: one is inferred from the action. Anything mentioning failure or an exception becomes an error; rate limits, retries, truncation, skips and blocks become warnings; starts, completions and step boundaries become info; everything else is debug.

Step entries

The action executor writes a structured entry per step attempt — id, type, status, attempt number, which steps fed it, its configuration with sensitive keys redacted, input, output, duration and token usage. Status maps to level: success is info, skipped is a warning, error is an error.

Step entries are exempt from the size cap. Ad-hoc lifecycle traces are truncated at 8 KB so they cannot bloat the file, but a structured step entry is written in full — the file has no document-size limit, unlike the database record, so this is the one place complete step input and output survive. Per-step cost is frozen into the line too, using the run's model rates.

Housekeeping

File handles closed after 30 seconds of inactivity
Expired files swept every six hours
An initial sweep runs on first use
The sweep timer never keeps the process alive
Buffered writes flushed on shutdown
Concurrent removal during a sweep is tolerated

Application Logs

Two process-level files: a general application log and a separate log for model interactions. These are the layer that answers "is the service itself working".

Writes go through append streams, not synchronous appends. Every line used to be written with a blocking call, which stalls the event loop for the duration of the syscall. On a multi-agent team run that is hundreds of blocking writes per turn — and because it blocks the loop rather than just the caller, it serialised work that is otherwise concurrent: parallel sub-agents, and every other tenant's in-flight request. Streams keep identical bytes and ordering while handing the syscall off the loop.

Formatting deliberately stays synchronous on the caller's stack. Deferring it would let callers mutate the object being logged before serialisation, silently changing what ends up on disk. And a stream error is swallowed on purpose — a logging failure must never take the process down.

Secret redaction

Credentials were once written verbatim. Redaction now happens before anything is serialised, using two independent mechanisms.

By key name — an exact list covering authorization, tokens of every kind, API keys, secrets, passwords, cookies, credentials and private keys, matched case- and separator-insensitively. Beyond the exact list, a key is treated as sensitive if it ends in a secret-ish word.

The plural exception is the clever part. A suffix rule for "token" would also match inputTokens and outputTokens — redacting the very metrics analytics depends on. Keys ending in tokens or count are therefore explicitly excluded, so the singular check catches credentials while token-usage numbers survive intact.

By value pattern — bearer tokens and recognisable key prefixes are stripped out of free text, so a credential embedded in a message body is caught even when no key name gives it away. The walk is cycle-safe and never mutates the caller's object. The same exported helper is reused for API Registry request headers and payloads, so the database and the log file redact identically.

Size control

A scheduled cleanup trims the application log by keeping the newest proportion of lines, but only once the file exceeds a minimum line count — so a quiet deployment is never truncated pointlessly. Interval, path, retained ratio and threshold are all configurable, and the scheduler can be disabled. The same pass also clears temporary browser-automation files.

Retention & TTL

Retention windows are operator-configurable, which turns out to be harder than it sounds — because a database treats index options as immutable.

Creating a TTL index twice with different windows throws. Re-running creation with an existing name but a changed expiry fails outright. Since the windows are configurable and do change, every TTL index has to be reconciled rather than merely created — which is also why these indexes are not declared on the schema, whose auto-indexing only ever attempts creation.

How reconciliation works

No index yet — create it.
Wrong key, or not a TTL index at all — rebuild. A live retune cannot change an index's key, and on older database versions cannot convert a plain index into a TTL one.
Right key, different window — retune in place. Preferred over drop-and-recreate so the collection is never left briefly unguarded.
Already correct — verify and move on.

"Lifetime" is the absence of an index, not a large number. There is no infinite expiry, and zero would expire documents immediately — so a retention of zero drops the index instead. Index names deliberately match the auto-generated ones, so an index created before reconciliation existed is retuned in place rather than duplicated.

Trace log files follow the same clock by a different mechanism: a periodic sweep deletes files whose modification time is older than the analytics window, so the file layer and the database layer expire together.

Dashboards

Four views are built on top of the records, each behind its own permission.

Agent overview

High-level metrics across every run — agents and teams together, since both live in one collection. Paginated, with a configurable day window.

Per-agent detail

One agent's runs, resolving the models, templates, knowledge bases, MCP servers and API endpoints it touched.

Team analytics

The same collection filtered to team runs, with per-run drill-down.

Cost management

Spend aggregation behind a separate cost permission, so usage figures can be shared without exposing commercial ones.

Knowledge-base analytics

Ingestion cost in its own collection — tokens consumed, chunks produced, the embedding model and strategy used, per file. Kept separate so ingestion spend is auditable independently of run spend.

Template analytics

Prompt-enhancement usage per template: re-enhance count, running token totals, and cost frozen per enhancement.

Reading a trace from a dashboard

A run record points at its trace file, and the viewer parses that file into structured lines. Continuation lines from pretty-printed payloads are folded back into the preceding entry, so a multi-line object does not become several bogus rows.

Older runs can still be traced. Runs recorded before the file pointer was stored have no direct link — but the run wrote its own identity into the file: the opening line carries the session id, which the record also stores. Matching on that is identity, not guesswork. Only when no file names the session does it fall back to nearest modification time, and that result is reported as approximate rather than presented as certain.

How It Works — Flow Examples

Four diagrams: the write path, status resolution, TTL reconciliation, and trace recovery.

Input / output Data handling Decision Rule / gate Result

Example 1 — Writing the run record Ingestion

The write happens off the response path, which is exactly why a rejection has to name the offending field paths.

Fire-and-forget keeps the response fast; named field paths keep the failure debuggable

Example 2 — How a run's status is decided Status

A guardrail block returns a message rather than throwing, so it has to be caught explicitly or it would look like an ordinary success.

Fixed precedence, checked top to bottom — the first match wins

Example 3 — Reconciling a retention window TTL

Because index options cannot be changed by re-creating them, every configured window is reconciled against what already exists.

Retuning in place means the collection is never left briefly unguarded

Example 4 — Finding a run's trace file Recovery

Newer runs carry a direct pointer. Older ones are matched on the session id the run wrote into its own opening line.

The fallback is still offered, but never presented as certain

Reference Tables

Error codes, log levels, the detail arrays, and what each layer retains.

Error codes

Code	Raised when
TOKEN_LIMIT_EXCEEDED	Input too large, or the output limit was exceeded
MCP_CONNECTION_FAILED	An MCP server could not be reached
TEAM_INVALID	The team graph failed validation
SAFETY_POLICY_VIOLATION	A global safety violation, or a rejected request
RATE_LIMIT_EXCEEDED	A rate limit was hit
PROVIDER_ERROR	The model provider was unavailable
TIMEOUT	The call timed out
SYSTEM_ERROR	Anything unclassified

Trace log levels

Level	In the default set	Inferred from actions mentioning
trace	No	—
debug	No	Everything unmatched
info	Yes	start, complete, invoke, step, prompt, route, node
warn	Yes	rate limit, retry, truncation, skip, block
error	Yes	error, fail, exception
fatal	Yes	fatal

What each layer keeps

Layer	Granularity	Size handling	Retention
Run record	One document per run	Previews and counts, to stay aggregatable	Configurable TTL index; zero means keep forever
Trace log	One file per run	8 KB cap on ad-hoc traces; step entries in full	Swept every six hours against the analytics window
Application log	Process-wide	Trimmed to the newest proportion of lines	Scheduled cleanup, above a minimum line count
KB ingestion	One document per file embedded	Counts and metadata only	Own collection, audited separately