Building an eval

Reference for evaluate(…): dataset, runner, scorers, and every option.

An eval file calls evaluate(...), then you run it:

promptlayer eval run path/to/your.eval.py

Use evaluate for sync runners and aevaluate (Python) for async runners; JavaScript always uses async evaluate. For full harness setups (OpenAI Agents, Claude, LangChain, and more), see the Quickstart.

evaluate(...) options

name is the first positional argument; everything else is keyword-only.

Option (Python / JS)	Required	Default	Purpose
name	✅	—	Experiment name; also the Eval: <name> span.
dataset	✅	—	Cases fed to the runner and scorers. See Dataset.
runner	✅	—	The agent under test. See Runner.
scorers	✅	—	Scorecard checks. See Columns and scorers.
columns	—	None	Processing steps applied before scorers. See Columns and scorers.
passing_score / passingScore	—	None	Fail the run below this bar. See Results and failures.
max_concurrency / maxConcurrency	—	1	Run cases in parallel.
include_failure_examples / includeFailureExamples	—	false	Print failing rows in the CLI output.
table_id / tableId	—	None	Reuse an existing Table instead of creating one.
folder_id / folderId	—	None	Folder for a newly created Table.
experiment_name / experimentName	—	auto	Name of the experiment sheet.
api_key / apiKey	—	env	Auth; prefer PROMPTLAYER_API_KEY for the CLI.
base_url / baseURL	—	env	Override the API base URL.

Do not pass a top-level sheet_id / sheetId. To read cases from a dashboard sheet, use dataset.sheet_id / dataset.sheetId instead.

Dataset

Cases feed the runner and scorers. Datasets covers inline cases and dashboard Tables. Each case supports:

Field	Required	Purpose
input	✅	Passed to the runner (column input).
expected	—	Expected output for Contains, Compare, … (column expected).
expected_trace / expectedTrace	—	Trajectory config — which tools should run, not the answer text.
other keys	—	Custom TEXT columns (exact title). No reserved or supporting-column name collisions.

Runner

The runner is any callable that takes an input and returns a final value (not a stream) — no framework allowlist.

For Trajectory, tools must appear as Tool: <name> spans on the imported Trace; use a framework helper or traceTool. The runner and tool-span rules live on Runner.

Columns and scorers

• Columns are optional processing steps that run before scorers. Declare them with Columns; types match Column Types.
• Scorers become scorecard steps on the experiment sheet. Declare them with Scorers. Rollup matches Scorecards; check types match Column Types.

What you get back

On success, the SDK creates (or reuses) a Table, writes a dedicated experiment sheet, configures your scorecard, and returns an EvalResult with a dashboard URL. For failure handling and passing_score, see Results and failures.