Tracing¶

The SDK provides OTel-native (OpenTelemetry) tracing that records every LLM call, tool execution, and chain step. Traces are stored locally in SQLite by default and can be exported to any OTel-compatible backend (Jaeger, Datadog, Grafana, etc.).

Quick Start¶

Tracing is automatic — every agent and chain execution creates spans:

from fastaiagent import Agent, LLMClient

agent = Agent(
    name="support-bot",
    system_prompt="Be helpful.",
    llm=LLMClient(provider="openai", model="gpt-4.1"),
)

# This execution is automatically traced
result = agent.run("Hello")

# Every result includes a trace_id
print(result.trace_id)  # e.g. "b6acf1ef2c2779bbc2fcf80802ae0534"

# Use it to replay and debug later
from fastaiagent.trace import Replay
replay = Replay.load(result.trace_id)

Manual Tracing with Context Manager¶

Wrap any code block in a trace span:

from fastaiagent.trace import trace_context

with trace_context("my-operation") as span:
    span.set_attribute("custom.key", "value")
    # ... your code here ...
    result = do_work()

Nested Spans¶

Spans nest automatically — inner spans become children of the outer span:

with trace_context("parent-operation") as parent:
    parent.set_attribute("step", "start")

    with trace_context("step-1") as child1:
        # This span is a child of parent-operation
        do_step_1()

    with trace_context("step-2") as child2:
        do_step_2()

This creates a trace tree:

parent-operation
├── step-1
└── step-2

Local Storage¶

All traces are stored automatically in a local SQLite database at .fastaiagent/local.db. No configuration needed.

Sensitive data in local.db

Traces capture prompts, tool inputs/outputs, and LLM responses verbatim, plus image / PDF attachment bytes when trace_full_images=True. Anything a user types — names, emails, tickets, screenshots — lands in local.db as written. Treat the file as you would any database backing your app:

The SDK creates .fastaiagent/local.db with mode 0o600 and the parent directory with 0o700 (POSIX) so other users on the same machine can't read it. Don't widen those permissions without a reason.
Before sharing a project directory or cutting a backup, scrub historical traces with the CLI:

# Delete everything older than 30 days, including attachments.
fastaiagent traces purge --older-than-days 30 --attachments

# Wipe the whole trace store (interactive prompt).
fastaiagent traces purge

Set FASTAIAGENT_TRACE_PAYLOADS=0 (see Payload Gating) to record only span structure — names, timing, model, tokens, cost — and leave prompt text and outputs out of the DB entirely.

Querying Traces¶

from fastaiagent.trace import TraceStore

store = TraceStore()

# List recent traces
traces = store.list_traces(last_hours=24)
for t in traces:
    print(f"{t.trace_id[:12]}  {t.name}  spans={t.span_count}  {t.start_time}")

# Get a specific trace with all spans
trace = store.get_trace("abc123def456...")
print(f"Name: {trace.name}")
print(f"Status: {trace.status}")
print(f"Spans: {len(trace.spans)}")

for span in trace.spans:
    print(f"  {span.name}  {span.start_time} → {span.end_time}")
    print(f"    Attributes: {span.attributes}")

# Search traces by name or attributes
results = store.search("support-bot")

# Export as JSON
json_str = store.export("abc123def456...", format="json")

TraceSummary¶

Returned by list_traces() and search():

Field	Type	Description
`trace_id`	`str`	Unique trace identifier
`name`	`str`	Root span name
`start_time`	`str`	ISO timestamp
`status`	`str`	OK, ERROR, UNSET
`span_count`	`int`	Number of spans
`duration_ms`	`int`	Total duration

TraceData¶

Returned by get_trace():

Field	Type	Description
`trace_id`	`str`	Unique trace identifier
`name`	`str`	Root span name
`start_time`	`str`	ISO timestamp
`end_time`	`str`	ISO timestamp
`status`	`str`	OK, ERROR, UNSET
`metadata`	`dict`	Trace-level metadata
`spans`	`list[SpanData]`	All spans in the trace

SpanData¶

Field	Type	Description
`span_id`	`str`	Unique span identifier
`trace_id`	`str`	Parent trace
`parent_span_id`	`str \\| None`	Parent span (None for root)
`name`	`str`	Span name (e.g., "llm.chat_completion")
`start_time`	`str`	ISO timestamp
`end_time`	`str`	ISO timestamp
`status`	`str`	OK, ERROR, UNSET
`attributes`	`dict`	Key-value metadata
`events`	`list[dict]`	Span events

GenAI Semantic Conventions¶

The SDK follows the OpenTelemetry GenAI semantic conventions for LLM-related attributes:

Attribute	Description	Example
`gen_ai.system`	LLM provider	`"openai"`, `"anthropic"`
`gen_ai.request.model`	Model name	`"gpt-4.1"`
`gen_ai.request.temperature`	Temperature	`0.7`
`gen_ai.request.max_tokens`	Max tokens	`1000`
`gen_ai.usage.input_tokens`	Prompt tokens	`150`
`gen_ai.usage.output_tokens`	Completion tokens	`45`
`gen_ai.response.finish_reasons`	Stop reasons	`["stop"]`

FastAIAgent Custom Attributes¶

Attribute	Description
`agent.name`	Agent name
`fastaiagent.chain.name`	Chain name
`fastaiagent.chain.node_id`	Current node in chain
`fastaiagent.chain.iteration`	Cycle iteration count
`fastaiagent.tool.name`	Tool being executed
`fastaiagent.checkpoint.id`	Checkpoint ID
`fastaiagent.guardrail.name`	Guardrail name — what the platform keys the guardrail row on
`fastaiagent.guardrail.passed`	Whether the guardrail passed
`fastaiagent.guardrail.position`	`input` / `tool_call` / `tool_result` / `output`
`fastaiagent.guardrail.errored`	The check couldn't run; `passed` reflects `on_error`, not a verdict
`fastaiagent.guardrail.checks`	JSON `[{"name": ..., "result": "pass"｜"block"｜"error"}]`
`fastaiagent.cost.total_usd`	Accumulated cost
`fastaiagent.template.kind`	Flagship-template marker on root span (e.g. `"deep-research"`) — set via `set_template_kind()`. Lets the UI badge / filter trace lists by template.

OpenInference standard attributes¶

Spans are also classified with the OpenInference openinference.span.kind, so any consumer of that ecosystem understands them without knowing FastAIAgent:

Attribute	Set on	Notes
`openinference.span.kind = "GUARDRAIL"`	every guardrail span	The `fastaiagent.guardrail.` fields above are the outcome* convention riding under this kind — OpenInference standardizes the kind, not the fields. Guardrail spans also still carry a legacy `span_type="guardrail"`; that dual-write is transitional.
`openinference.span.kind = "EVALUATOR"` + `evaluation.{name,score,label,explanation,annotator_kind}`	an inline eval-score span	`evaluation.score` is a 0..1 scale. Nothing in `agent.run` emits this — you emit it when you score a turn yourself, via `set_evaluation_attributes()` / `emit_evaluation()`.

Both are plain attributes on the open OTel envelope, so they need no wire change. See Guardrails & evals without the runtime for emitting them from a runtime that isn't fa.Agent.

Marking template traces¶

Flagship example templates (e.g. examples/deep-research-agent) stamp a kebab-case marker on their root span so the UI can identify them without parsing span names:

from fastaiagent.trace import trace_context
from fastaiagent.trace.span import set_template_kind

with trace_context("deep_research.session") as span:
    set_template_kind(span, "deep-research")
    # ...

Filter via SQL:

SELECT trace_id, json_extract(attributes, '$.fastaiagent.research.topic')
FROM spans
WHERE json_extract(attributes, '$.fastaiagent.template.kind') = 'deep-research';

Convention: the kind matches the template's directory under examples/. Any new template (customer-support, meeting-notes, …) can adopt the same marker for free.

Agent Reconstruction Attributes (used by Replay)¶

Every agent.run() root span carries enough metadata for Agent Replay to reconstruct the agent from a stored trace and rerun it. These are always captured (structural, not payload):

Attribute	Description
`agent.name`	Agent name
`agent.input`	Input passed to `agent.run()`
`agent.output`	Final output
`agent.tokens_used`	Total tokens consumed
`agent.latency_ms`	Wall-clock duration
`agent.config`	JSON-encoded `AgentConfig` (max_iterations, temperature, max_tokens, etc.)
`agent.tools`	JSON-encoded list of tool schemas (name, description, parameters)
`agent.guardrails`	JSON-encoded list of guardrails (name, position, blocking, type)
`agent.llm.provider`	LLM provider (`openai`, `anthropic`, ...)
`agent.llm.model`	Model id
`agent.llm.config`	JSON-encoded `LLMClient.to_dict()` (api_key stripped)

Tool invocations emit their own tool.{name} span with:

Attribute	Description
`tool.name`	Tool name
`tool.origin`	`function` / `rest` / `mcp` / `kb` / `custom` / `unknown`
`tool.status`	`ok` / `error` / `unknown`
`tool.args`	JSON-encoded arguments (payload-gated — see below)
`tool.result`	JSON-encoded return value (payload-gated)
`tool.error`	Error string when status is `error`
`fastaiagent.runner.type`	Always `tool` — classifies the span as a tool call
`fastaiagent.tool.replay_class`	`read_only` / `idempotent` / `side_effecting` — the tool's replay-safety class (default `side_effecting`)

The last two are always captured (structural, not payload-gated) so the Replay engine can classify the span and pick inject-vs-execute even with payloads disabled. An unmarked tool resolves to side_effecting.

LLM calls emit llm.{provider}.{model} spans with standard GenAI attributes plus payload-gated gen_ai.request.messages, gen_ai.request.tools, gen_ai.response.content, gen_ai.response.tool_calls, and gen_ai.response.finish_reason.

Payload Gating (`FASTAIAGENT_TRACE_PAYLOADS`)¶

Payload-bearing attributes — LLM messages, LLM response content, tool arguments, tool results, and resolved system prompts — can contain sensitive data. They default to captured so replay reconstruction works out of the box, but you can turn them off globally:

export FASTAIAGENT_TRACE_PAYLOADS=0

With payloads disabled: - Structural metadata (agent.config, agent.tools, agent.guardrails, agent.llm.config, gen_ai.system, gen_ai.request.model, token counts, finish reasons, tool.name/tool.status) is still captured — traces remain useful for monitoring and performance analysis. - Free-text payloads (messages, responses, prompts, tool args/results) are skipped. - Replay reconstruction still works for agent config and tool schemas, but reruns lose the original resolved prompt if your code relied on span-captured prompts.

Defaults to 1 (on). Set to 0 in production environments handling PII if you do not otherwise scrub traces at the exporter layer.

Setting Attributes Programmatically¶

from fastaiagent.trace.span import set_genai_attributes, set_fastai_attributes

with trace_context("my-llm-call") as span:
    set_genai_attributes(
        span,
        system="openai",
        model="gpt-4.1",
        input_tokens=150,
        output_tokens=45,
    )
    set_fastai_attributes(
        span,
        **{"agent.name": "support-bot", "cost.total_usd": 0.003},
    )

Exporting to External Backends¶

OTLP (Jaeger, Grafana, Datadog)¶

from fastaiagent.trace import add_exporter
from fastaiagent.trace.export import create_otlp_exporter

# HTTP exporter (most common)
exporter = create_otlp_exporter(
    endpoint="http://localhost:4318/v1/traces",
    headers={"Authorization": "Bearer my-token"},
)
add_exporter(exporter)

# gRPC exporter
exporter = create_otlp_exporter(
    endpoint="http://localhost:4317",
    protocol="grpc",
)
add_exporter(exporter)

Requires: pip install fastaiagent[otel-export]

Any OTel SpanExporter¶

from fastaiagent.trace import add_exporter

# Use any OTel-compatible exporter
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
add_exporter(ConsoleSpanExporter())

Traces are always stored locally AND sent to exporters — adding an exporter doesn't replace local storage.

Custom Storage Path¶

Traces are always stored as SQLite. The database path can point to any filesystem location — local disk or a cloud-mounted volume.

Local¶

export FASTAIAGENT_LOCAL_DB=/data/my-project/local.db

from fastaiagent.trace import TraceStore
store = TraceStore(db_path="/data/my-project/local.db")

Cloud-Mounted Filesystems¶

Mount a cloud volume and point the trace path to it. SQLite works on any POSIX-compatible filesystem mount:

Cloud Provider	Mount Tool	Example Path
Azure Files	Azure File Share (SMB/NFS)	`/mnt/azure-share/local.db`
AWS S3	Mountpoint for S3 or s3fs-fuse	`/mnt/s3-bucket/local.db`
AWS EFS	NFS mount	`/mnt/efs/local.db`
GCS	Cloud Storage FUSE	`/mnt/gcs-bucket/local.db`

# Azure Files example
export FASTAIAGENT_LOCAL_DB=/mnt/azure-share/local.db

# S3 via Mountpoint
export FASTAIAGENT_LOCAL_DB=/mnt/s3-bucket/local.db

# Or set programmatically
store = TraceStore(db_path="/mnt/azure-share/local.db")

Note: SQLite requires a filesystem that supports file locking. Most cloud-mounted POSIX filesystems (Azure Files, EFS, GCS FUSE) support this. Object-storage mounts (S3 Mountpoint, s3fs-fuse) work for single-writer scenarios — avoid concurrent writes from multiple processes to the same SQLite file on these mounts.

See Example 10 for a runnable demo of trace querying with custom storage paths.

CLI Commands¶

# List recent traces
fastaiagent traces list
fastaiagent traces list --last-hours 1

# Export a trace as JSON
fastaiagent traces export <trace_id>
fastaiagent traces export abc123def456 --format json

Disabling Tracing¶

export FASTAIAGENT_TRACE_ENABLED=false

Or pass trace=False to agent/chain execution:

result = agent.run("Hello", trace=False)

Resetting the Tracer¶

For testing or reconfiguration:

from fastaiagent.trace import reset

reset()  # Shuts down existing provider, clears singleton
# Next trace operation creates a fresh provider

Architecture¶

Your Code
    │
    ▼
OTel TracerProvider (singleton)
    │
    ├── LocalStorageProcessor → SQLite (.fastaiagent/local.db)
    │
    ├── BatchSpanProcessor → OTLP Exporter (Jaeger, Datadog, etc.)
    │
    └── BatchSpanProcessor → Any additional exporters

LocalStorageProcessor writes every span to SQLite as it completes
BatchSpanProcessor batches spans for efficient export to remote backends
Multiple exporters can run simultaneously
The TracerProvider is a singleton — initialized on first use, reused globally

Platform Export¶

When connected to the FastAIAgent Platform, traces are automatically sent to the platform dashboard alongside local SQLite storage. No code changes needed.

import fastaiagent as fa

fa.connect(api_key="fa-...", project="my-project")

# Every agent.run() now sends traces to both local SQLite and platform
result = agent.run("Help me")
# View in platform dashboard: execution traces, token costs, latency

Export is local-first and durable: every span is written to local SQLite, then drained to the platform on a background thread. Transient failures (connection errors, timeouts, HTTP 5xx) are retried with bounded backoff, and any spans not yet acknowledged are buffered and re-sent on the next export — so a platform outage never loses traces and never blocks agent.run(). Re-sends are idempotent (/traces/ingest dedups by span_id). The re-send queue is bounded (~10k spans / ~7 days); spans beyond the bound are dropped from the queue but kept in local SQLite. See Offline / Disconnected Behavior.

Manual backfill — publish existing local traces to the platform:

trace_store = TraceStore()
traces = trace_store.list_traces(limit=100)
for t_summary in traces:
    trace_data = trace_store.get_trace(t_summary.trace_id)
    trace_data.publish()  # sends to platform

If the platform is unreachable, traces are safe in local SQLite. No operation fails because the platform is down.

Tagging a run with metadata¶

Attach your own key/values to a run (MLflow-style tags) and they land on the run's root span as fastaiagent.meta.* attributes — queryable in the trace store and, when connected, on the plane. You own the keys, the values, and any PII implications.

agent.run(
    "Refund my order",
    metadata={"customer": "acme", "env": "prod", "ticket": 4271},
)

Guardrail CHECKS¶

Each guardrail that runs on a turn emits a child guardrail.* span (on pass and block) carrying its outcome, so a trace shows a per-span CHECKS row. Nothing extra in your code — just add guardrails=[…] to the agent.

Pruning the local buffer¶

Traces are buffered in local.db. Reclaim space by deleting already-sent (acked/abandoned) spans:

fastaiagent traces prune               # delete all acked spans
fastaiagent traces prune --older-than-days 7

Internals¶

For contributors who need to modify the tracing layer, add new span attributes, debug missing spans, or understand the dual-sink model (SQLite + platform), see Tracing Architecture (Internals).

Next Steps¶

Replay — Debug agent execution with fork-and-rerun
Integrations — Auto-trace OpenAI, Anthropic, LangChain, CrewAI
Capture any OTel / OpenInference framework — enable_otel_capture() for third-party instrumentors
Agents — Build agents with automatic tracing

Examples¶

Example 09 — Export traces to OTel collectors (Jaeger, Datadog)
Example 10 — Query, search, and export local traces with custom storage paths