Tracing¶

The SDK provides OTel-native (OpenTelemetry) tracing that records every LLM call, tool execution, and chain step. Traces are stored locally in SQLite by default and can be exported to any OTel-compatible backend (Jaeger, Datadog, Grafana, etc.).

Quick Start¶

Tracing is automatic — every agent and chain execution creates spans:

from fastaiagent import Agent, LLMClient

agent = Agent(
    name="support-bot",
    system_prompt="Be helpful.",
    llm=LLMClient(provider="openai", model="gpt-4.1"),
)

# This execution is automatically traced
result = agent.run("Hello")

# Every result includes a trace_id
print(result.trace_id)  # e.g. "b6acf1ef2c2779bbc2fcf80802ae0534"

# Use it to replay and debug later
from fastaiagent.trace import Replay
replay = Replay.load(result.trace_id)

Manual Tracing with Context Manager¶

Wrap any code block in a trace span:

from fastaiagent.trace import trace_context

with trace_context("my-operation") as span:
    span.set_attribute("custom.key", "value")
    # ... your code here ...
    result = do_work()

Nested Spans¶

Spans nest automatically — inner spans become children of the outer span:

with trace_context("parent-operation") as parent:
    parent.set_attribute("step", "start")

    with trace_context("step-1") as child1:
        # This span is a child of parent-operation
        do_step_1()

    with trace_context("step-2") as child2:
        do_step_2()

This creates a trace tree:

parent-operation
├── step-1
└── step-2

Local Storage¶

All traces are stored automatically in a local SQLite database at .fastaiagent/local.db. No configuration needed.

Sensitive data in local.db

Traces capture prompts, tool inputs/outputs, and LLM responses verbatim, plus image / PDF attachment bytes when trace_full_images=True. Anything a user types — names, emails, tickets, screenshots — lands in local.db as written. Treat the file as you would any database backing your app:

The SDK creates .fastaiagent/local.db with mode 0o600 and the parent directory with 0o700 (POSIX) so other users on the same machine can't read it. Don't widen those permissions without a reason.
Before sharing a project directory or cutting a backup, scrub historical traces with the CLI:

# Delete everything older than 30 days, including attachments.
fastaiagent traces purge --older-than-days 30 --attachments

# Wipe the whole trace store (interactive prompt).
fastaiagent traces purge

Set FASTAIAGENT_TRACE_PAYLOADS=0 (see Payload Gating) to record only span structure — names, timing, model, tokens, cost — and leave prompt text and outputs out of the DB entirely.

Querying Traces¶

from fastaiagent.trace import TraceStore

store = TraceStore()

# List recent traces, newest first
traces = store.list_traces(last_hours=24)
for t in traces:
    print(f"{t.trace_id[:12]}  {t.name}  spans={t.span_count}  {t.start_time}")

# Narrow it down
store.list_traces(last_hours=None, limit=500)          # all time, up to 500
store.list_traces(name_filter="chain.sales-sdr")       # trace name starts with…
store.list_traces(agent_name="support-bot")            # traces this agent ran in

# Get a specific trace with all spans
trace = store.get_trace("abc123def456...")
print(f"Name: {trace.name}")
print(f"Status: {trace.status}")
print(f"Spans: {len(trace.spans)}")

for span in trace.spans:
    print(f"  {span.name}  {span.start_time} → {span.end_time}")
    print(f"    Attributes: {span.attributes}")

# Search traces by name or attributes
results = store.search("support-bot")

# Export as JSON
json_str = store.export("abc123def456...", format="json")

list_traces() returns traces that started within last_hours (default 24; None for all time), newest first, at most limit (default 100). agent_name matches an agent anywhere in the trace — the root, or a child span inside a swarm, chain or supervisor. Until 1.81.0 every argument was ignored and the call returned the newest 100 traces of all time.

Tailing spans as they land¶

list_traces() and get_trace() are both trace-shaped: to watch spans arrive you would have to poll each trace and diff it. list_spans() reads across traces directly, oldest first, from a cursor.

import time

from fastaiagent.trace import TraceStore

store = TraceStore()
cursor = 0

while True:
    page = store.list_spans(since=cursor, limit=100)
    for record in page:
        print(f"{record.span.name}  {record.span.status}")
        cursor = record.cursor          # advance past what you handled
    if not page:
        time.sleep(0.5)                 # nothing new yet — keep the cursor

since is a cursor, not a timestamp. Pass 0 first, then the cursor of the last record you handled. Each call returns only rows written after it, so a span is read exactly once even while new ones keep arriving.

That distinction matters. A long span starts before a short one that finishes first, so it is written second while carrying the earlier start time. A timestamp cursor would step straight over it; a write-order cursor cannot.

An empty page means "nothing new yet", not "end of stream" — keep the cursor and call again.

Two optional filters narrow the stream:

store.list_spans(trace_id="abc123def456...")   # one trace
store.list_spans(execution_id="run-42")        # one durable run, across traces

execution_id matches the chain.execution_id attribute, so a durable run is followed across every trace it spans.

SpanRecord¶

Returned by list_spans():

Field	Type	Description
`cursor`	`int`	Opaque write-order position. Pass it back as `since`.
`span`	`SpanData`	The span itself

The cursor is deliberately not a field on SpanData. That model is serialized whole when spans are exported to the Platform, so a field added to it would put a new key on the wire for something only local readers use.

TraceSummary¶

Returned by list_traces() and search():

Field	Type	Description
`trace_id`	`str`	Unique trace identifier
`name`	`str`	Root span name. Falls back to the lexicographically smallest span name only when the trace has no root span.
`start_time`	`str`	ISO timestamp
`status`	`str`	OK, ERROR, UNSET
`span_count`	`int`	Number of spans
`duration_ms`	`int`	Total duration

name became the root span in 1.68.0

It was the smallest span name in the trace, full stop — so a swarm whose spans were swarm.pair and agent.alpha came back as agent.alpha, because agent.* sorts ahead of swarm.*. That hit most multi-agent traces, and the same expression backs the Local UI's traces list and Home page, so a run you knew by a child agent's name now lists under its runner's.

TraceData¶

Returned by get_trace():

Field	Type	Description
`trace_id`	`str`	Unique trace identifier
`name`	`str`	Root span name
`start_time`	`str`	ISO timestamp
`end_time`	`str`	ISO timestamp
`status`	`str`	OK, ERROR, UNSET
`metadata`	`dict`	Trace-level metadata
`spans`	`list[SpanData]`	All spans in the trace

SpanData¶

Field	Type	Description
`span_id`	`str`	Unique span identifier
`trace_id`	`str`	Parent trace
`parent_span_id`	`str \\| None`	Parent span (None for root)
`name`	`str`	Span name (e.g., "llm.chat_completion")
`start_time`	`str`	ISO timestamp
`end_time`	`str`	ISO timestamp
`status`	`str`	OK, ERROR, UNSET
`attributes`	`dict`	Key-value metadata
`events`	`list[dict]`	Span events

GenAI Semantic Conventions¶

The SDK follows the OpenTelemetry GenAI semantic conventions for LLM-related attributes:

Attribute	Description	Example
`gen_ai.system`	LLM provider	`"openai"`, `"anthropic"`
`gen_ai.request.model`	Model name	`"gpt-4.1"`
`gen_ai.request.temperature`	Temperature	`0.7`
`gen_ai.request.max_tokens`	Max tokens	`1000`
`gen_ai.usage.input_tokens`	Prompt tokens	`150`
`gen_ai.usage.output_tokens`	Completion tokens	`45`
`gen_ai.response.finish_reasons`	Stop reasons	`["stop"]`

FastAIAgent Custom Attributes¶

Attribute	Description
`agent.name`	Agent name
`fastaiagent.chain.name`	Chain name
`fastaiagent.chain.node_id`	Current node in chain
`fastaiagent.chain.iteration`	Cycle iteration count
`fastaiagent.tool.name`	Tool being executed
`fastaiagent.checkpoint.id`	Checkpoint ID
`fastaiagent.guardrail.name`	Guardrail name — what the platform keys the guardrail row on
`fastaiagent.guardrail.passed`	Whether the guardrail passed
`fastaiagent.guardrail.position`	`input` / `tool_call` / `tool_result` / `output`
`fastaiagent.guardrail.errored`	The check couldn't run; `passed` reflects `on_error`, not a verdict
`fastaiagent.guardrail.checks`	JSON `[{"name": ..., "result": "pass"｜"block"｜"error"}]`
`fastaiagent.cost.total_usd`	Estimated USD cost of that one LLM call, priced from the model id and the provider's token counts. Set by `LLMClient` on every `llm.` span since 1.67.0, and by the LangChain / CrewAI / Pydantic-AI integrations before that. Sum it across a trace for the run's cost. Absent* — not `0.0` — when the model has no rate, so a reader can fall back to its own estimate instead of trusting a fabricated zero.
`fastaiagent.template.kind`	Flagship-template marker on root span (e.g. `"deep-research"`) — set via `set_template_kind()`. Lets the UI badge / filter trace lists by template.

OpenInference standard attributes¶

Spans are also classified with the OpenInference openinference.span.kind, so any consumer of that ecosystem understands them without knowing FastAIAgent:

Attribute	Set on	Notes
`openinference.span.kind = "GUARDRAIL"`	every guardrail span	The `fastaiagent.guardrail.` fields above are the outcome* convention riding under this kind — OpenInference standardizes the kind, not the fields. Guardrail spans also still carry a legacy `span_type="guardrail"`; that dual-write is transitional.
`openinference.span.kind = "EVALUATOR"` + `evaluation.{name,score,label,explanation,annotator_kind}`	an inline eval-score span	`evaluation.score` is a 0..1 scale. Nothing in `agent.run` emits this — you emit it when you score a turn yourself, via `set_evaluation_attributes()` / `emit_evaluation()`.

Both are plain attributes on the open OTel envelope, so they need no wire change. See Guardrails & evals without the runtime for emitting them from a runtime that isn't fa.Agent.

Marking template traces¶

Flagship example templates (e.g. examples/deep-research-agent) stamp a kebab-case marker on their root span so the UI can identify them without parsing span names:

from fastaiagent.trace import trace_context
from fastaiagent.trace.span import set_template_kind

with trace_context("deep_research.session") as span:
    set_template_kind(span, "deep-research")
    # ...

Filter via SQL:

SELECT trace_id, json_extract(attributes, '$.fastaiagent.research.topic')
FROM spans
WHERE json_extract(attributes, '$.fastaiagent.template.kind') = 'deep-research';

Convention: the kind matches the template's directory under examples/. Any new template (customer-support, meeting-notes, …) can adopt the same marker for free.

Agent Reconstruction Attributes (used by Replay)¶

Every agent.run() root span carries enough metadata for Agent Replay to reconstruct the agent from a stored trace and rerun it. These are always captured (structural, not payload):

Attribute	Description
`agent.name`	Agent name
`agent.input`	Input passed to `agent.run()`
`agent.output`	Final output
`agent.tokens_used`	Tokens across every LLM call of the run (each tool-loop turn and every re-ask), not just the last response. Whole-run since 1.68.0.
`agent.latency_ms`	Wall-clock duration
`swarm.tokens_used`	Tokens across a streamed swarm run, summed over every hop. Set by `Swarm.stream()` only — the non-streaming path reports the same number on the result. New in 1.68.0.
`supervisor.tokens_used`	The supervisor's own turns, not its workers' — each worker reports its spend on its own result. Set by `Supervisor.stream()` only. New in 1.68.0.
`agent.config`	JSON-encoded `AgentConfig` (max_iterations, temperature, max_tokens, etc.)
`agent.tools`	JSON-encoded list of tool schemas (name, description, parameters)
`agent.guardrails`	JSON-encoded list of guardrails (name, position, blocking, type)
`agent.llm.provider`	LLM provider (`openai`, `anthropic`, ...)
`agent.llm.model`	Model id
`agent.llm.config`	JSON-encoded `LLMClient.to_dict()` (api_key stripped)

Tool invocations emit their own tool.{name} span with:

Attribute	Description
`tool.name`	Tool name
`tool.origin`	`function` / `rest` / `mcp` / `kb` / `custom` / `unknown`
`tool.status`	`ok` / `error` / `unknown`
`tool.args`	JSON-encoded arguments (payload-gated — see below)
`tool.result`	JSON-encoded return value (payload-gated)
`tool.error`	Error string when status is `error`
`fastaiagent.runner.type`	Always `tool` — classifies the span as a tool call
`fastaiagent.tool.replay_class`	`read_only` / `idempotent` / `side_effecting` — the tool's replay-safety class (default `side_effecting`)

The last two are always captured (structural, not payload-gated) so the Replay engine can classify the span and pick inject-vs-execute even with payloads disabled. An unmarked tool resolves to side_effecting.

LLM calls emit llm.{provider}.{model} spans with standard GenAI attributes plus payload-gated gen_ai.request.messages, gen_ai.request.tools, gen_ai.response.content, gen_ai.response.tool_calls, and gen_ai.response.finish_reason.

Payload Gating (`FASTAIAGENT_TRACE_PAYLOADS`)¶

Payload-bearing attributes — LLM messages, LLM response content, tool arguments, tool results, and resolved system prompts — can contain sensitive data. They default to captured so replay reconstruction works out of the box, but you can turn them off globally:

export FASTAIAGENT_TRACE_PAYLOADS=0

With payloads disabled: - Structural metadata (agent.config, agent.tools, agent.guardrails, agent.llm.config, gen_ai.system, gen_ai.request.model, token counts, finish reasons, tool.name/tool.status) is still captured — traces remain useful for monitoring and performance analysis. - Free-text payloads (messages, responses, prompts, tool args/results) are skipped. - Replay reconstruction still works for agent config and tool schemas, but reruns lose the original resolved prompt if your code relied on span-captured prompts.

Defaults to 1 (on). Set to 0 in production environments handling PII if you do not otherwise scrub traces at the exporter layer.

Setting Attributes Programmatically¶

from fastaiagent.trace.span import set_genai_attributes, set_fastai_attributes

with trace_context("my-llm-call") as span:
    set_genai_attributes(
        span,
        system="openai",
        model="gpt-4.1",
        input_tokens=150,
        output_tokens=45,
    )
    set_fastai_attributes(
        span,
        **{"agent.name": "support-bot", "cost.total_usd": 0.003},
    )

Exporting to External Backends¶

OTLP (Jaeger, Grafana, Datadog)¶

from fastaiagent.trace import add_exporter
from fastaiagent.trace.export import create_otlp_exporter

# HTTP exporter (most common)
exporter = create_otlp_exporter(
    endpoint="http://localhost:4318/v1/traces",
    headers={"Authorization": "Bearer my-token"},
)
add_exporter(exporter)

# gRPC exporter
exporter = create_otlp_exporter(
    endpoint="http://localhost:4317",
    protocol="grpc",
)
add_exporter(exporter)

Requires: pip install fastaiagent[otel-export]

Any OTel SpanExporter¶

from fastaiagent.trace import add_exporter

# Use any OTel-compatible exporter
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
add_exporter(ConsoleSpanExporter())

Traces are always stored locally AND sent to exporters — adding an exporter doesn't replace local storage.

Custom Storage Path¶

Traces are always stored as SQLite. The database path can point to any filesystem location — local disk or a cloud-mounted volume.

Local¶

export FASTAIAGENT_LOCAL_DB=/data/my-project/local.db

from fastaiagent.trace import TraceStore
store = TraceStore(db_path="/data/my-project/local.db")

Cloud-Mounted Filesystems¶

Mount a cloud volume and point the trace path to it. SQLite works on any POSIX-compatible filesystem mount:

Cloud Provider	Mount Tool	Example Path
Azure Files	Azure File Share (SMB/NFS)	`/mnt/azure-share/local.db`
AWS S3	Mountpoint for S3 or s3fs-fuse	`/mnt/s3-bucket/local.db`
AWS EFS	NFS mount	`/mnt/efs/local.db`
GCS	Cloud Storage FUSE	`/mnt/gcs-bucket/local.db`

# Azure Files example
export FASTAIAGENT_LOCAL_DB=/mnt/azure-share/local.db

# S3 via Mountpoint
export FASTAIAGENT_LOCAL_DB=/mnt/s3-bucket/local.db

# Or set programmatically
store = TraceStore(db_path="/mnt/azure-share/local.db")

Note: SQLite requires a filesystem that supports file locking. Most cloud-mounted POSIX filesystems (Azure Files, EFS, GCS FUSE) support this. Object-storage mounts (S3 Mountpoint, s3fs-fuse) work for single-writer scenarios — avoid concurrent writes from multiple processes to the same SQLite file on these mounts.

See Example 10 for a runnable demo of trace querying with custom storage paths.

CLI Commands¶

# List recent traces
fastaiagent traces list
fastaiagent traces list --last-hours 1

# Export a trace as JSON
fastaiagent traces export <trace_id>
fastaiagent traces export abc123def456 --format json

Disabling Tracing¶

export FASTAIAGENT_TRACE_ENABLED=false   # 0 / no / off all work too

This is the master switch: the SDK hands out OpenTelemetry's no-op tracer, so no span is built, nothing is written to local.db, no attachment bytes are stored, and there is nothing to export. The Local UI and Replay go with it — there is no trace to read. result.trace_id becomes the all-zero id rather than raising.

This started working in 1.67.0

The variable was parsed and documented from the beginning but read by nothing. If you have it set, your next upgrade stops capturing traces — which is what you asked for, but it may not be what you have been getting.

To keep local traces while stopping content from leaving the machine, use FASTAIAGENT_TRACE_PAYLOADS=0 instead — see Security Posture.

Or pass trace=False to agent/chain execution, which suppresses tracing for that call only:

result = agent.run("Hello", trace=False)

In code, the same switch is fastaiagent.config.trace_enabled — set it before the first traced run (see SDK configuration).

Resetting the Tracer¶

For testing or reconfiguration:

from fastaiagent.trace import reset

reset()  # Shuts down existing provider, clears singleton
# Next trace operation creates a fresh provider

Architecture¶

Your Code
    │
    ▼
OTel TracerProvider (singleton)
    │
    ├── LocalStorageProcessor → SQLite (.fastaiagent/local.db)
    │
    ├── BatchSpanProcessor → OTLP Exporter (Jaeger, Datadog, etc.)
    │
    └── BatchSpanProcessor → Any additional exporters

LocalStorageProcessor writes every span to SQLite as it completes
BatchSpanProcessor batches spans for efficient export to remote backends
Multiple exporters can run simultaneously
The TracerProvider is a singleton — initialized on first use, reused globally

Platform Export¶

When connected to the FastAIAgent Platform, traces are automatically sent to the platform dashboard alongside local SQLite storage. No code changes needed.

import fastaiagent as fa

fa.connect(api_key="fa-...", project="my-project")

# Every agent.run() now sends traces to both local SQLite and platform
result = agent.run("Help me")
# View in platform dashboard: execution traces, token costs, latency

Export is local-first and durable: every span is written to local SQLite, then drained to the platform on a background thread. Transient failures (connection errors, timeouts, HTTP 5xx) are retried with bounded backoff, and any spans not yet acknowledged are buffered and re-sent on the next export — so a platform outage never loses traces and never blocks agent.run(). Re-sends are idempotent (/traces/ingest dedups by span_id). The re-send queue is bounded (~10k spans / ~7 days); spans beyond the bound are dropped from the queue but kept in local SQLite. See Offline / Disconnected Behavior.

Manual backfill — publish existing local traces to the platform:

trace_store = TraceStore()
traces = trace_store.list_traces(last_hours=None, limit=100)  # all time
for t_summary in traces:
    trace_data = trace_store.get_trace(t_summary.trace_id)
    trace_data.publish()  # sends to platform

If the platform is unreachable, traces are safe in local SQLite. No operation fails because the platform is down.

Tagging a run with metadata¶

Attach your own key/values to a run (MLflow-style tags) and they land on the run's root span as fastaiagent.meta.* attributes — queryable in the trace store and, when connected, on the plane. You own the keys, the values, and any PII implications.

agent.run(
    "Refund my order",
    metadata={"customer": "acme", "env": "prod", "ticket": 4271},
)

Guardrail CHECKS¶

Each guardrail that runs on a turn emits a child guardrail.* span (on pass and block) carrying its outcome, so a trace shows a per-span CHECKS row. Nothing extra in your code — just add guardrails=[…] to the agent.

Pruning the local buffer¶

Traces are buffered in local.db. Reclaim space by deleting already-sent (acked/abandoned) spans:

fastaiagent traces prune               # delete all acked spans
fastaiagent traces prune --older-than-days 7

Internals¶

For contributors who need to modify the tracing layer, add new span attributes, debug missing spans, or understand the dual-sink model (SQLite + platform), see Tracing Architecture (Internals).

Next Steps¶

Replay — Debug agent execution with fork-and-rerun
Integrations — Auto-trace OpenAI, Anthropic, LangChain, CrewAI
Capture any OTel / OpenInference framework — enable_otel_capture() for third-party instrumentors
Agents — Build agents with automatic tracing

Examples¶

Example 09 — Export traces to OTel collectors (Jaeger, Datadog)
Example 10 — Query, search, and export local traces with custom storage paths