Research note · Semantics
A tool result in the next LLM context does not mean the agent used it
You shipped MCP observability. Your dashboard shows a green checkmark: consumed. The tool returned a result, and mcp_tool.name shows up in the next model request. Chain closed, right?
Here is what that checkmark actually proves — and what it doesn't.
The concrete case
An agent calls a pricing tool. The tool returns price = $38.21. The next model request contains the tool result. The next model output says:
"The current price is $38.21."
Did the agent use the result? Most dashboards would say yes. But look closer — there are at least three different facts hiding in that "yes":
- The result entered the model's context (your trace can show this)
- The output restates the value (you can observe this, sometimes)
- The agent's decision depended on the value (you cannot show this from a single trace)
The third is the one everyone actually wants. It is also the only one your telemetry cannot deliver.
What a single trace can prove
We asked people on both sides of the API where this boundary sits, and two answers converged from opposite directions:
- A provider engineer (the tool side): the server knows the call happened and the result was returned. What the agent did with it afterwards is invisible from the server.
- A reviewer of our spec (the client side): a single trace can certify the result was placed into the next model call's context. Proving it influenced the next decision requires removing or corrupting the result and running again — an ablation, not an observation.
Two sides, same boundary. That boundary is the interesting object:
STATE — AVAILABLE (a fact about the context)
Tool invoked ✅ observable
Result returned ✅ observable
Result available next step ✅ observable — a trace certifies exactly this
STATE — INFLUENTIAL (a fact about the behavior)
Result influential ✗ requires counterfactual evidence (ablation / rerun)
Evidence for influence, ranked:
ablation / rerun strong
result referenced weak estimator — fails both ways:
cite-without-use · use-without-citeTwo states, not three: referenced is not a level in between — it sits on the evidence axis. A model restating $38.21 is compatible with the model ignoring the price entirely; only the counterfactual run rules that out.
Why this matters more than it seems
If your usage metrics collapse these two states into one flag, every downstream number inherits the ambiguity. "Consumption rate" becomes a number nobody can defend: it can't be compared across runtimes, because each runtime means something different by it. And when it's time to bill, or to decide which tools the agent actually needs, the causal question is the only one that matters — and it's the one the trace can't answer.
The honest split is architectural, not cosmetic:
- Trace layer proves availability, and observes reference as weak evidence.
- Experiment layer (ablation / rerun) estimates causal influence.
A measurement system that knows which layer it's in can say exactly what it knows — and that turns out to be the harder, more valuable feature than another dashboard.
What we did with this
This boundary exposed an overclaim in our own spec: our Claude Code profile called mcp_tool.name in the next request a "consumption signal" — the first platform where consumption was "empirically proven." By the two states above, that signal certifies context availability, and we've corrected the claim and renamed the signal (issue #11, DR-005).
Claude Code is still genuinely notable — it's the first client runtime where availability is certifiable at all. That's a real capability. It's one state short of what we claimed, and the state is the whole point.
This is why agent usage needs evidence-aware measurement semantics: metrics that name what they observe, dashboards that mark what they don't.