Skip to content

Connected eval export (Agent-CI verdicts)

Agent CI gates your build locally and works fully standalone — no platform dependency. When you fa.connect() to an Enterprise control plane, the SDK additionally reports each gated run's verdict so the org can answer "which of our agents passed their gates this week?" and treat an agent that produces no eval evidence as a governance finding rather than a blank space.

Mark, don't coerce

Eval export is egress, not enforcement, and the SDK keeps those in separate lanes:

Who decides Local off-switch
Enforcement — plane-authored guardrails plane authors, edge enforces no (by design)
Egress — traces, eval verdicts your runtime yes, and it is final

connect(export_evals=False) is honored, always. The plane cannot override it, exactly as with export_traces. What a connected plane does instead is mark agents that produce no eval evidence, and it can refuse to promote an agent that cannot produce any — you are never blocked locally, and nothing is forced out of your machine.

Because the SDK attests its posture at enroll, the plane distinguishes two states that otherwise look identical:

  • "eval export disabled" — you run evals and keep them local (a deliberate choice)
  • "no eval data in N days" — no evals are running at all

What is reported

Metadata only. Run aggregates, the gate verdict, thresholds, git provenance, and per-case scorer verdicts travel. Case inputs, expected outputs and actual outputs never do — the plane joins content through each case's trace_id against traces it already ingested, so a verdict can be opened against real content without shipping a second copy of it.

Preview the literal payload before anything leaves:

fastaiagent eval export --dry-run     # the exact JSON
fastaiagent eval export --status      # posture + how many runs are queued

Wire protocol

POST {target}/public/v1/eval/runs/ingest · headers X-API-Key, Content-Type: application/json · wire v1.6

Batched, at-least-once, idempotent on run_id. The response counts new runs only, so a re-send is free:

{ "ingested": 2 }

Request

{
  "runs": [
    {
      "run_id": "76fd558996c54dc598b5d0652d5dbbfc",
      "run_name": "pytest::northwind-support",
      "dataset_name": "cases.jsonl",
      "agent_name": "northwind-support",
      "started_at": "2026-08-23T09:14:02.511034+00:00",
      "finished_at": "2026-08-23T09:14:19.884120+00:00",
      "pass_count": 18,
      "fail_count": 2,
      "pass_rate": 0.9,
      "errored_count": 1,
      "error_rate": 0.0476,
      "gate_outcome": "failed",
      "thresholds": { "overall.pass_rate": 0.95 },
      "scorers": ["exact_match", "faithfulness"],
      "git_sha": "9f3c1ab7e2d4c5f6a8b9c0d1e2f3a4b5c6d7e8f9",
      "git_branch": "feature/refund-policy",
      "baseline": {
        "run_id": "d41c9b2e77a34f0e8b6c1a5d3e9f7b28",
        "pass_rate_delta": -0.05,
        "regressed_count": 2
      },
      "sdk_version": "1.49.0",
      "instance_id": "be42fb1d3f7847e98e0ace31b2a05f40",
      "cases": [
        {
          "case_id": "0b1caf35e17c45d5baeb0fde33551e39",
          "ordinal": 0,
          "per_scorer": {
            "exact_match": { "passed": true, "score": 1.0, "reason": null }
          },
          "trace_id": "ae077737d85c4f2b9e1a6c3d5f8b0a2c",
          "error": null
        },
        {
          "case_id": "7c2e9f11ab3d40a6b8e5d4c7f1a92b60",
          "ordinal": 1,
          "per_scorer": {},
          "trace_id": null,
          "error": "provider 500"
        }
      ]
    }
  ]
}

Run fields

Field Type Meaning
run_id string idempotency key — SDK-generated; the plane dedupes on it
run_name string? e.g. pytest::<rootdir>, or --eval-run-name
dataset_name string? the dataset the cases came from
agent_name string? resolved to a real plane agent at ingest — this is what makes the run evidence for a specific system. Inferred from the callable (agent.run → Agent.name); null when a suite spans several agents
started_at / finished_at ISO 8601? evidence freshness / control decay
pass_count / fail_count / pass_rate int/int/float? the scored result: checks (applicable scorer verdicts) passed / failed, and passed over passed + failed. Since 1.89.0 an errored or skipped verdict is in neither count; pass_rate is null when no verdict applied
errored_count / error_rate int/float? cases whose outcome is errored — unscored, never counted as quality misses. Since 1.89.0 that includes a case whose scorer could not run, or that no scorer applied to, not only a case whose agent raised
gate_outcome passed|failed|invalid required. invalid = infra disqualified the run
thresholds object? what the gate demanded, keyed <metric>.<field>. {} = evals ran, no gate demanded
scorers string[]? scorer names used
git_sha / git_branch string? which version was tested
baseline object? {run_id, pass_rate_delta, regressed_count} — evidences regressions tracked across versions, not merely "evals ran"
sdk_version / instance_id string? provenance; instance_id joins the enrollment record
cases array per-case verdicts (below)

Case fields

Field Type Meaning
case_id string? SDK-generated; the row key
ordinal int? position in the dataset
per_scorer object? {scorer: {passed, score, reason}}, plus errored: true (the check could not run), skipped: true (it did not apply) and details (non-content extras, e.g. a judge's raw_score) only when set (1.89.0). Empty for a case whose agent raised. reason is scrubbed of credentials, and null when FASTAIAGENT_TRACE_PAYLOADS=0
trace_id string? the join key — lets the plane corroborate this verdict against a trace it ingested independently
error string? infra failure detail; presence means the case was never scored. Scrubbed of credentials; with FASTAIAGENT_TRACE_PAYLOADS=0 it is the fixed text errored (detail withheld: FASTAIAGENT_TRACE_PAYLOADS=0)

Never on the wire

input, expected_output, actual_output. A test freezes the exact payload shape and asserts these three are absent — adding them later is a privacy regression, not an enhancement.

Note that run_name, dataset_name and especially git_branch do travel, and branch names routinely carry ticket IDs or customer names.

Enrollment attestation

The posture rides along on the existing enroll call so the plane can tell a deliberate opt-out from an absence of evals:

POST /public/v1/governance/enroll
{
  "instance_id": "be42fb1d3f7847e98e0ace31b2a05f40",
  "sdk_version": "1.49.0",
  "fail_mode": "open",
  "protocol_version": "1",
  "export_evals": true
}

Errors

Status Meaning SDK behavior
2xx ingested mark runs synced
403 key lacks eval:execute, or domain lacks connected_state_plane terminal — warn once, leave buffered to age out
404 plane predates wire v1.6 terminal — warn once
5xx / transport transient retry ×3 with 0.5s, 1.0s backoff, then leave buffered

eval:execute is not a default scope; mint keys with it explicitly.

Delivery model

Runs are queued in local.db (eval_runs.synced = 0) and pushed by a background drain. A run flips to synced = 1 only after a confirmed 2xx — so an outage simply buffers, and the next gated run drains the backlog. The buffer is bounded (50k runs / 30 days); beyond that the oldest un-acked runs are abandoned, which marks them handled without deleting anything — they stay in local.db and the Local UI.

A run becomes exportable when its gate verdict is recorded, not when it is persisted: the plane requires gate_outcome, and the gate necessarily runs after persistence. The CLI (fastaiagent eval run) and the pytest plugin record the verdict for you; a pytest session with no gate options records passed with thresholds: {}, because "evals ran, no gate demanded" is evidence too. A bare evaluate() in Python is exported only once you call record_gate_result() (evaluate → gate → record_gate_result); until then fastaiagent eval export --dry-run shows "gate_outcome": null and marks the run not exportable.

The CLI and the pytest plugin record the verdict but do not call connect(). Their runs are sent only when something in the same process connected (the agent module or conftest.py); otherwise they wait in local.db and go with the next connected gated run. fastaiagent eval export --status shows whether the process is connected and how many runs are waiting. eval export itself never sends.

A run a scripted model answered is never sent (since 1.104.0). If a TestModel or FunctionModel answered or judged any case, evaluate() and the pytest plugin store scripted_models: true in the run's local metadata, the run is written synced = 1, and record_gate_result() logs a warning when connected. Its verdict scores what the test scripted, and on the plane it would read as evidence about the agent. This came from a coding-assistant test: with no model key the assistant scripted the answers, scored 100%, and recorded a passing verdict on the plane. The key stays in local.db; nothing new goes on the wire.

Changed in 1.89.0. Before, a run waiting for its verdict was sent with an invented passed whenever any span flush drained the outbox, and the plane keeps the first copy of a run_id, so the real verdict arriving later was discarded. A process running an older SDK against the same local.db can still send those runs; upgrade every process that shares the file.

One drain runs at a time per local.db. The gate's background kick, a span flush and fastaiagent eval export used to drain the same rows at once and post a run twice. The plane answered the duplicate with 409, and the SDK logged a rejection for a run that had landed. A drain that finds one in flight now asks it to go round once more instead (since 1.99.0).

record_gate_result() raises ValueError for any outcome other than passed, failed or invalid. The plane would reject it with a 4xx, which is never retried, and the bad run would hold back every run batched with it.

When export is off, runs are written synced = 1 and never queued, so a disabled install never grows an outbox it cannot drain. Turning it on later ships only new runs, never a backlog dump.

See also