Skip to content

Reasoning and prompt caching

Two LLMClient settings, added in 1.91.0, that every provider spells differently:

from fastaiagent import Agent, LLMClient

llm = LLMClient(provider="gemini", model="gemini-3.8-flash", reasoning="low", cache=True)
agent = Agent(name="support", system_prompt=LONG_POLICY, llm=llm, tools=[...])
  • reasoning= sets how hard the model thinks before answering.
  • cache= turns on prompt caching on the providers that need to be asked.

Both are off by default. An unset client sends exactly the request it sent before 1.91.0.

Reasoning

llm = LLMClient(provider="anthropic", model="claude-haiku-5-5", reasoning="medium")
response = llm.complete([UserMessage("A bat and a ball cost $1.10 ...")])

response.content                      # "$0.05" — the answer, never the reasoning
response.reasoning                    # what the model showed of its thinking, or None
response.usage["reasoning_tokens"]    # the part of completion_tokens spent thinking

llm.complete(messages, reasoning="high")   # one call can override the client

One vocabulary, lowest first: "none", "minimal", "low", "medium", "high", "xhigh", "max". A typo raises ValueError before anything is sent. Each provider gets the level in its own field:

Provider What is sent
Gemini 3.5+ / 4 thinkingConfig.thinkingLevel (minimal–high; xhigh/max send high) and includeThoughts, so a thought summary comes back
Gemini 2.5 thinkingConfig.thinkingBudget: none 0, minimal 512, low 1024, medium 8192, high and above 24576
OpenAI and OpenAI-compatible reasoning_effort, as given
Claude 5.x, Opus 4.6–4.8, Sonnet 4.6 output_config.effort plus adaptive thinking with display: "summarized" (5.x models hide the text otherwise). minimal sends low, Anthropic's lowest
Claude Opus 4.5 effort and a thinking budget_tokens
Claude Haiku 4.5, Sonnet 4.5, older extended thinking with a budget_tokens (low 2048 … max 63999, always below max_tokens)
Claude, "none" thinking: {"type": "disabled"}; on Sonnet 5.5, between_tools (its lowest setting)
Bedrock (Claude models) the Claude fields, as additionalModelRequestFields
Ollama think: a level for gpt-oss, true/false otherwise

Which levels a model accepts varies. For example, gpt-6-luna takes none through xhigh, Gemini 3.8 Flash has no minimal, and Opus 5.5 cannot turn thinking off. The SDK passes your choice through and lets the provider answer, because its 400 names the levels it supports:

Unsupported value: 'reasoning_effort' does not support 'max' with this model.
Supported values are: 'none', 'low', 'medium', 'high', and 'xhigh'.

GPT-6 models on Chat Completions take function tools only with reasoning off. An agent with tools and reasoning="low" on gpt-6-luna gets OpenAI's 400: "Function tools with reasoning_effort are not supported for gpt-6-luna in /v1/chat/completions". Use reasoning="none" there; the GPT-5 family takes tools at every level.

Gemini 3 and later cannot turn thinking off. With "none", the SDK sends the lowest level, minimal, and logs a warning once.

What comes back. LLMResponse.reasoning holds the text the provider shows.

Provider Reasoning text
Anthropic summarized thinking
Gemini thought summaries
DeepSeek, Groq, OpenRouter, vLLM reasoning_content / reasoning
OpenAI Chat Completions none (usage["reasoning_tokens"] still counts it)

Reasoning text is never part of content. Before 1.91.0 a Gemini thought summary would have been glued onto the answer. Reasoning tokens are already inside completion_tokens on every provider, so they are billed at the output rate.

Streaming. astream yields a ReasoningDelta(text=...) per chunk of reasoning, alongside the TextDeltas of the answer:

from fastaiagent import ReasoningDelta, TextDelta

async for event in agent.astream("Is it warmer in Paris or Rome?"):
    if isinstance(event, ReasoningDelta):
        show_thinking(event.text)
    elif isinstance(event, TextDelta):
        show_answer(event.text)

Claude's thinking in the tool loop

Claude returns its thinking as thinking blocks carrying a signature. Claude 5.x models think by default, without reasoning=. While a tool-use turn is in progress, Anthropic requires the blocks to be sent back exactly as they came.

Before 1.91.0 the agent loop dropped them. The API does not return an error for this; it just turns thinking off for the rest of the turn, so a Claude agent lost its own reasoning after its first tool call. Now the blocks ride on the assistant message (Message.reasoning_blocks) through run, astream, checkpoints and resume. They go back only to the provider that issued them, because a model cannot read another provider's reasoning. Bedrock's reasoningContent blocks are carried the same way.

Prompt caching

llm = LLMClient(provider="anthropic", model="claude-sonnet-5-5", cache=True)    # 5 minutes
llm = LLMClient(provider="anthropic", model="claude-sonnet-5-5", cache="1h")    # one hour
Provider What cache= does
Anthropic sends the automatic top-level cache_control: one breakpoint on the last cacheable block, moved forward as the conversation grows
Bedrock a cachePoint after the system prompt, after the tools, and on the last turn
OpenAI, Gemini nothing to send: both cache long prompts on their own

Every provider's counts come back in one shape:

response.usage["prompt_tokens"]        # the whole prompt, cached part included
response.usage["cached_tokens"]        # read from the cache
response.usage["cache_write_tokens"]   # written to it

Anthropic and Bedrock report only the uncached part as input, so the SDK adds the cache back: prompt_tokens means the same thing on every provider. A prompt shorter than the model's minimum is simply not cached. On Claude the minimum is 512 tokens for the 5.x models and 1,024 to 4,096 for older ones; on OpenAI it is 1,024.

Cost. Cached and written tokens are priced at their own rates. The provider pricing pages, read 2026-10-18, give:

Provider Cache read Cache write
OpenAI 1/10 of input (1/20 on GPT-6.1 Sol) 1.25× input from GPT-5.6; no charge before
Anthropic 0.1× input (0.05× Opus/Sonnet 5.5, 0.025× Fable/Mythos 5.1) 1.25× input, 2× for one hour

A second identical prompt costs less than the first, and AgentResult.cost, the cost_limit guardrail and the cost pages all see the difference. See cost tracking.

Reasoning, a thinking agent streamed through a tool loop, and a cached prompt on Gemini 3.8 Flash, Claude and OpenAI: examples/112_reasoning_and_prompt_caching.py.

In traces

Every llm.* span records the counts. With FASTAIAGENT_TRACE_PAYLOADS=0 the counts still leave the machine, but the reasoning text stays local.

Attribute What it holds Exported with payloads off
gen_ai.usage.reasoning_tokens tokens spent reasoning yes
gen_ai.usage.cache_read.input_tokens prompt tokens read from the cache (OpenTelemetry GenAI name) yes
gen_ai.usage.cache_creation.input_tokens prompt tokens written to the cache (OpenTelemetry GenAI name) yes
gen_ai.response.reasoning the reasoning text no: kept locally (security)

Saved with the agent

reasoning and cache are part of LLMClient.to_dict(), so a replayed or plane-dispatched agent reasons and caches as the original did. They are written only when set, so a default client's payload is unchanged.