Scorers

Add scorecard checks to evaluate(…) with typed helpers.

Pass scorers into evaluate(...) to grade each case. On the experiment sheet they show up as scorecard evaluators (Contains, Trajectory, and the rest).

Thresholds, weights, and the overall score work the same as Scorecards in Tables. The underlying check types are documented in Column Types.

Scorers at a glance

Scorer	Python	JavaScript	Checks
Compare	compare_scorer	compareScorer	Two columns match (default Output vs Expected)
Contains	contains_scorer	containsScorer	A column contains a literal or another column’s value
Regex	regex_scorer	regexScorer	A column matches a regular expression
Count	count_scorer	countScorer	Character / word / sentence / paragraph count within bounds
Assert Valid	assert_valid_scorer	assertValidScorer	A column is a valid object, number, or SQL
LLM Assertion	llm_assertion_scorer	llmAssertionScorer	An LLM judge grades a column
Trajectory	trajectory_scorer	trajectoryScorer	Tool-call sequence matches accepted scenarios
Custom Code	named fn / code_execution_column	named fn / codeExecutionColumn	Your own scoring logic

Every typed helper also accepts shared scoring options — weight, required, and thresholds.

Shared scoring options

Every scorer produces a score in [0, 1]. These three options — accepted by every typed helper — control how much a scorer counts and where its pass/fail line sits. They behave exactly like Scorecards in Tables.

• Weight — how much this scorer contributes to the overall score. The scorecard is a weighted mean, so weight=2 pulls twice as hard as the default weight=1. Use it to make the checks you care about most dominate the result.
• Required — if true, this scorer failing fails the whole scorecard, no matter how well the weighted average comes out. Use it for must-pass checks (e.g. the agent called the right tools).
• Thresholds — the per-scorer cutoffs that decide pass vs. warn vs. fail, each in [0, 1] (default pass 0.8, warn 0.6). Raise the pass threshold for a stricter bar on that check.

Option	Python	JavaScript	Meaning
Weight	weight	weight	How much this step counts (default 1)
Required	required	required	If true, failing this step can fail the scorecard
Thresholds	thresholds	thresholds	Per-step pass / warn cutoffs in [0, 1]
Pass alias	pass_threshold	passThreshold	Sets thresholds.pass
Warn alias	failure_threshold	failureThreshold	Sets thresholds.warn

This support agent example weights the trajectory check higher, marks it required, and sets a pass threshold:

Python

from agents import Agent, Runner, function_tool
from promptlayer import evaluate, contains_scorer, trajectory_scorer
from promptlayer.integrations.openai_agents import instrument_openai_agents

instrument_openai_agents()

@function_tool
def lookup_user(email: str) -> str:
    """Look up a user account by email."""
    return f"Found user for {email}."

@function_tool
def send_reset(email: str) -> str:
    """Send a password reset email."""
    return f"Reset link sent to {email}."

agent = Agent(
    name="Support agent",
    instructions=(
        "Help with password resets. Call lookup_user, then send_reset, "
        "then apologize briefly and confirm the reset was sent."
    ),
    model="gpt-5.6",
    tools=[lookup_user, send_reset],
)

def run_agent(user_message: str) -> str:
    return str(Runner.run_sync(agent, user_message).final_output)

evaluate(
    "support-bot",
    dataset=[{"input": "Please reset my password for [e-mail]"}],
    runner=run_agent,
    scorers=[
        contains_scorer(
            "Has apology",
            source="Output",
            value="sorry",
            weight=1.0,
        ),
        trajectory_scorer(
            accepted_scenarios=[["lookup_user", "send_reset"]],
            weight=2.5,
            required=True,
            pass_threshold=0.9,
        ),
    ],
)

import { Agent, run, tool } from "@openai/agents";
import { instrumentOpenAIAgents } from "promptlayer/openai-agents";
import { evaluate, containsScorer, trajectoryScorer } from "promptlayer";
import { z } from "zod";

await instrumentOpenAIAgents();

const lookupUser = tool({
  name: "lookup_user",
  description: "Look up a user account by email.",
  parameters: z.object({ email: z.string() }),
  execute: async ({ email }) => `Found user for ${email}.`,
});

const sendReset = tool({
  name: "send_reset",
  description: "Send a password reset email.",
  parameters: z.object({ email: z.string() }),
  execute: async ({ email }) => `Reset link sent to ${email}.`,
});

const agent = new Agent({
  name: "Support agent",
  instructions:
    "Help with password resets. Call lookup_user, then send_reset, then apologize briefly and confirm the reset was sent.",
  model: "gpt-5.6",
  tools: [lookupUser, sendReset],
});

async function runAgent(userMessage) {
  return String((await run(agent, userMessage)).finalOutput);
}

await evaluate("support-bot", {
  dataset: [{ input: "Please reset my password for [e-mail]" }],
  runner: runAgent,
  scorers: [
    containsScorer({
      title: "Has apology",
      source: "Output",
      value: "sorry",
      weight: 1.0,
    }),
    trajectoryScorer({
      acceptedScenarios: [["lookup_user", "send_reset"]],
      weight: 2.5,
      required: true,
      passThreshold: 0.9,
    }),
  ],
});

Scorer types

Each helper below maps to a scorecard check type in Column Types. All of them accept the shared scoring options above.

Compare

Compare two columns. By default that is Output vs Expected as strings. Pass exactly two column titles in sources. Behavior matches the Compare type in Column Types.

Set how they compare with comparison_type / comparisonType:

Mode	What it does	Example
STRING (default)	Exact string match	"STRING"
JSON	Compare JSON values; optional path	{"type": "JSON", "json_path": "$.id"}
NUMBER	Compare numbers with an operator (EQ by default)	{"type": "NUMBER", "operator": "GTE"}

Number operators: EQ, NE, GT, GTE, LT, LTE.

Python

from openai import OpenAI
from promptlayer import evaluate, compare_scorer

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": "Reply with exactly one word: hello",
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "compare-demo",
    dataset=[{"input": "Greet the user.", "expected": "hello"}],
    runner=run_llm,
    scorers=[
        compare_scorer(
            "Exact match",
            sources=["Output", "Expected"],
            comparison_type="STRING",
        )
    ],
)

import OpenAI from "openai";
import { evaluate, compareScorer } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      { role: "system", content: "Reply with exactly one word: hello" },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

await evaluate("compare-demo", {
  dataset: [{ input: "Greet the user.", expected: "hello" }],
  runner: runLlm,
  scorers: [
    compareScorer({
      title: "Exact match",
      sources: ["Output", "Expected"],
      comparisonType: "STRING",
    }),
  ],
});

Contains

Check whether a column contains a literal or another column’s value. Pass exactly one of value or value_source / valueSource.

Key	Required	Default
source	yes	"Output"
value	one of	literal substring
value_source / valueSource	one of	column title

Python

from openai import OpenAI
from promptlayer import evaluate, contains_scorer

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": "You are a support agent. Apologize and help in one short sentence.",
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "contains-demo",
    dataset=[{"input": "I cannot log in to my account."}],
    runner=run_llm,
    scorers=[contains_scorer(source="Output", value="sorry")],
)

import OpenAI from "openai";
import { evaluate, containsScorer } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      {
        role: "system",
        content: "You are a support agent. Apologize and help in one short sentence.",
      },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

await evaluate("contains-demo", {
  dataset: [{ input: "I cannot log in to my account." }],
  runner: runLlm,
  scorers: [containsScorer({ source: "Output", value: "sorry" })],
});

Regex

Match a column against a regular expression.

Key	Required	Default
source	yes	"Output"
regex_pattern / regexPattern	yes	—

Python

from openai import OpenAI
from promptlayer import evaluate, regex_scorer

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are an order lookup bot. Reply with only an order id "
                    "like order-123."
                ),
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "regex-demo",
    dataset=[{"input": "What is the order id for my headphones?"}],
    runner=run_llm,
    scorers=[regex_scorer(source="Output", regex_pattern=r"order-\d+")],
)

import OpenAI from "openai";
import { evaluate, regexScorer } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      {
        role: "system",
        content:
          "You are an order lookup bot. Reply with only an order id like order-123.",
      },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

await evaluate("regex-demo", {
  dataset: [{ input: "What is the order id for my headphones?" }],
  runner: runLlm,
  scorers: [regexScorer({ source: "Output", regexPattern: "order-\\d+" })],
});

Count

Score by character, word, sentence, or paragraph count bounds. Require at least one of min_count / minCount or max_count / maxCount. Backend type values: chars, words, sentences, paragraphs.

Key	Required	Default
source	yes	"Output"
type	no	"chars"
min_count / minCount	one of min/max	—
max_count / maxCount	one of min/max	—

Python

from openai import OpenAI
from promptlayer import evaluate, count_scorer

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": "Answer in one short sentence of at most five words.",
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "count-demo",
    dataset=[{"input": "How do I reset my password?"}],
    runner=run_llm,
    scorers=[count_scorer(source="Output", type="words", min_count=1, max_count=5)],
)

import OpenAI from "openai";
import { evaluate, countScorer } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      {
        role: "system",
        content: "Answer in one short sentence of at most five words.",
      },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

await evaluate("count-demo", {
  dataset: [{ input: "How do I reset my password?" }],
  runner: runLlm,
  scorers: [
    countScorer({ source: "Output", type: "words", minCount: 1, maxCount: 5 }),
  ],
});

Assert Valid

Validate a column as object, number, or SQL. There is no json / boolean / string assert type.

Key	Required	Default	Allowed
source	yes	"Output"	column title
type	yes	"object"	object | number | sql

Python

from openai import OpenAI
from promptlayer import evaluate, assert_valid_scorer

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": (
                    "Reply with a JSON object only, no markdown. "
                    'Example: {"status":"ok","ticket_id":101}'
                ),
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "assert-valid-demo",
    dataset=[{"input": "Create a support ticket for a billing issue."}],
    runner=run_llm,
    scorers=[assert_valid_scorer(source="Output", type="object")],
)

import OpenAI from "openai";
import { evaluate, assertValidScorer } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      {
        role: "system",
        content:
          'Reply with a JSON object only, no markdown. Example: {"status":"ok","ticket_id":101}',
      },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

await evaluate("assert-valid-demo", {
  dataset: [{ input: "Create a support ticket for a billing issue." }],
  runner: runLlm,
  scorers: [assertValidScorer({ source: "Output", type: "object" })],
});

LLM Assertion

Score a column with an LLM judge prompt. Pass exactly one of prompt or prompt_source / promptSource.

Key	Required	Default
source	yes	"Output"
prompt	one of	literal prompt text
prompt_source / promptSource	one of	column title
variable_mappings / variableMappings	no	Prompt variable → column title

Python

from openai import OpenAI
from promptlayer import evaluate, llm_assertion_scorer

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": "Answer the user's question in one short factual sentence.",
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "llm-assert-demo",
    dataset=[{"input": "Why is the sky blue?"}],
    runner=run_llm,
    scorers=[
        llm_assertion_scorer(
            source="Output",
            prompt="Is this factually reasonable? Reply yes or no.",
        )
    ],
)

import OpenAI from "openai";
import { evaluate, llmAssertionScorer } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      {
        role: "system",
        content: "Answer the user's question in one short factual sentence.",
      },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

await evaluate("llm-assert-demo", {
  dataset: [{ input: "Why is the sky blue?" }],
  runner: runLlm,
  scorers: [
    llmAssertionScorer({
      source: "Output",
      prompt: "Is this factually reasonable? Reply yes or no.",
    }),
  ],
});

Trajectory

Scores Tool: <name> spans from the imported Trace against accepted scenarios. Any runner works if those spans exist — emit them with a framework helper or traceTool (see Runner), or copy a harness from the Quickstart.

Pass exactly one of inline accepted_scenarios / acceptedScenarios, or expected_source / expectedSource (usually "Expected Trace").

Key	Required	Default
trace_source / traceSource	yes	"Trace"
accepted_scenarios / acceptedScenarios	one of	list of non-empty tool-name lists
expected_source / expectedSource	one of	column title
mode	no	"strict"

Mode	Behavior
strict (default)	Observed tools must equal the scenario exactly
non_strict	Required tools appear in order as a subsequence

Inline accepted_scenarios:

Python

from agents import Agent, Runner, function_tool
from promptlayer import evaluate, trajectory_scorer, contains_scorer
from promptlayer.integrations.openai_agents import instrument_openai_agents

instrument_openai_agents()

@function_tool
def get_weather(city: str) -> str:
    """Return demo weather for a city."""
    return f"{city} is 72F and sunny."

agent = Agent(
    name="Weather agent",
    instructions="Use get_weather, then answer in one short sentence.",
    model="gpt-5.6",
    tools=[get_weather],
)

evaluate(
    "trajectory-demo",
    dataset=[{"input": "What is the weather in Tokyo?"}],
    runner=lambda message: str(Runner.run_sync(agent, message).final_output),
    scorers=[
        trajectory_scorer(accepted_scenarios=[["get_weather"]], mode="non_strict"),
        contains_scorer(source="Output", value="72"),
    ],
)

import { Agent, run, tool } from "@openai/agents";
import { instrumentOpenAIAgents } from "promptlayer/openai-agents";
import { evaluate, trajectoryScorer, containsScorer } from "promptlayer";
import { z } from "zod";

await instrumentOpenAIAgents();

const getWeather = tool({
  name: "get_weather",
  description: "Return demo weather for a city.",
  parameters: z.object({ city: z.string() }),
  execute: async ({ city }) => `${city} is 72F and sunny.`,
});

const agent = new Agent({
  name: "Weather agent",
  instructions: "Use get_weather, then answer in one short sentence.",
  model: "gpt-5.6",
  tools: [getWeather],
});

await evaluate("trajectory-demo", {
  dataset: [{ input: "What is the weather in Tokyo?" }],
  runner: async (userMessage) => String((await run(agent, userMessage)).finalOutput),
  scorers: [
    trajectoryScorer({ acceptedScenarios: [["get_weather"]], mode: "non_strict" }),
    containsScorer({ source: "Output", value: "72" }),
  ],
});

Or read scenarios from an Expected Trace column. Cell JSON:

{
  "accepted_scenarios": [
    { "required_tools": ["get_weather"] }
  ]
}

Python

trajectory_scorer(
    expected_source="Expected Trace",
    mode="non_strict",
)

trajectoryScorer({
  expectedSource: "Expected Trace",
  mode: "non_strict",
})

Put that JSON on each case as expected_trace / expectedTrace (or on a dashboard dataset sheet).

Local helpers: score_trajectory / scoreTrajectory, diagnose_trajectory_failure / diagnoseTrajectoryFailure, and JS-only extractTrajectoryToolNames.

Custom Code

Prefer passing a named function by reference in scorers. The SDK serializes it to a CODE_EXECUTION scorecard step.

Parameter names map to columns:

Param	Column
input	Input
output	Output
expected	Expected
trace	Trace

Any other param name is looked up as a column title of the same name. Return a number (or {"score": x}, which is normalized to x).

• Use a named function — no lambdas / anonymous functions.
• Sync only — async scorers are rejected.
• Default title is the function name with _ → spaces. Override with scorer_from_function(fn, title=...) / scorerFromFunction(fn, { title }) when needed.

Python

from openai import OpenAI
from promptlayer import evaluate

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": "You are a support agent. Answer in one short sentence.",
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

def exact_match(output, expected):
    return 1 if expected.lower() in (output or "").lower() else 0

evaluate(
    "fn-scorer-demo",
    dataset=[
        {
            "input": "How do I reset my password?",
            "expected": "password",
        }
    ],
    runner=run_llm,
    scorers=[exact_match],
)

import OpenAI from "openai";
import { evaluate } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      {
        role: "system",
        content: "You are a support agent. Answer in one short sentence.",
      },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

function exactMatch(output, expected) {
  return String(output || "")
    .toLowerCase()
    .includes(String(expected).toLowerCase())
    ? 1
    : 0;
}

await evaluate("fn-scorer-demo", {
  dataset: [
    {
      input: "How do I reset my password?",
      expected: "password",
    },
  ],
  runner: runLlm,
  scorers: [exactMatch],
});

code_execution_column / codeExecutionColumn

Use this when you want a raw code string instead of a function. code is required. Default language is PYTHON (Python helper) or JAVASCRIPT (JS helper). Row values are available as data.

Python

from openai import OpenAI
from promptlayer import evaluate, code_execution_column

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": "You are a support agent. Answer in one short sentence.",
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "code-scorer-demo",
    dataset=[
        {
            "input": "How do I reset my password?",
            "expected": "password",
        }
    ],
    runner=run_llm,
    scorers=[
        code_execution_column(
            "exact",
            code=(
                'return 1 if str(data.get("Expected", "")).lower() '
                'in str(data.get("Output", "")).lower() else 0'
            ),
        )
    ],
)

import OpenAI from "openai";
import { evaluate, codeExecutionColumn } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      {
        role: "system",
        content: "You are a support agent. Answer in one short sentence.",
      },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

await evaluate("code-scorer-demo", {
  dataset: [
    {
      input: "How do I reset my password?",
      expected: "password",
    },
  ],
  runner: runLlm,
  scorers: [
    codeExecutionColumn("exact", {
      code: 'return String(data["Output"] || "").toLowerCase().includes(String(data["Expected"] || "").toLowerCase()) ? 1 : 0;',
    }),
  ],
});