Runner and tracing

Pass any callable as the evaluate(…) runner. Return a final value; use tool spans only for Trajectory.

The runner is any callable that takes an input and returns a final value. There is no framework allowlist — a plain function works, and nothing else is required to run an eval.

evaluate(...) opens an Eval: <name> span per case; any PromptLayer client used inside the runner nests under it automatically.

What the runner returns

Return the agent’s final answer as a single value. evaluate(...) records it as the case’s Output (stored on the eval span as the eval.output attribute), so it has to be a resolved value, not a live one:

• Prefer a primitive. Strings, numbers, and booleans are stored as-is — that’s why the framework examples wrap the result in str(...). Any other object is JSON-serialized, so return something JSON-friendly.
• No streams. A generator or async iterator is rejected — consume the stream and return its final text.
• No un-awaited coroutines. On the sync path a bare awaitable is rejected; use aevaluate(...) (Python) or an async function (JavaScript) for async runners.

Runner	Python	JavaScript
Sync	evaluate(...)	evaluate(...)
Async	aevaluate(...)	evaluate(...) with an async function (no aevaluate export)

Default concurrency is 1; raise it with max_concurrency / maxConcurrency (see Concurrency). Copy-paste harnesses for common frameworks are in the Quickstart.

Tool spans (only for Trajectory)

Tool spans matter for exactly one thing: the Trajectory scorer, which grades the sequence of tools your agent called. It reads spans named Tool: <name> from the imported Trace and ignores everything else. Scorers that read the Output — Contains, Compare, LLM assertion — need no tool spans at all.

You don’t create those spans by hand. Emit them with either:

• A framework helper — OpenAI Agents, Claude, Vercel AI, and the rest in Telemetry Integrations — which instruments your tools automatically.
• traceTool on a custom agent’s tool handlers. It records the Tool: <name> span for you (on a tracing-enabled client); see Tracing Tools for the full contract.

Only the tools you assert on (the names in accepted_scenarios) need a span — a helper tool you don’t score can stay untraced. How the modes treat what they observe:

mode	Rule	Untraced / extra tools
strict	Observed tools must exactly equal the expected list.	An expected tool that isn’t traced fails; every traced tool must be in your expected list.
non_strict	Expected list must be an ordered subsequence of observed.	Extra traced tools are tolerated; only an untraced expected tool fails.

Common errors

Issue	Fix
Empty Trace / Trajectory fails	Emit Tool: spans for the tools you assert on — a framework helper or traceTool on a tracing-enabled client
Traced tool missing in strict mode	Add it to the expected list, or switch to non_strict
Streamed return	Collect the final answer before returning it
Awaitable on the sync Python path	Use aevaluate