Create Evaluation Pipeline

Creates a new evaluation pipeline (report) that can be configured with multiple evaluation steps (columns). An evaluation pipeline processes datasets through a series of transformations and evaluations to measure prompt performance.

Workflow:

1. Create a dataset group and dataset first
2. Create an evaluation pipeline using this endpoint
3. Add evaluation steps using /report-columns
4. Run the evaluation using /reports/{report_id}/run
5. Monitor progress and retrieve scores

Note: While you can create pipelines programmatically, the dashboard provides a more user-friendly interface for initial pipeline design.

This endpoint allows for the creation of Evaluation pipelines. These pipelines are given a name and associated with a dataset.

Headers

X-API-KEY · string · required

API key to authorize the operation. Can also use JWT authentication.

Body

application/json

dataset_group_id · integer · required

The ID of the dataset group containing the dataset versions to evaluate. The dataset group must be within a workspace accessible to the authenticated user.

Required range: x >= 1

name · string

Optional name for the evaluation pipeline. If not provided, a unique name will be auto-generated. Must be between 1 and 255 characters if specified.

Required string length: 1 - 255

folder_id · integer | null

Optional folder ID to organize the pipeline within your workspace. If not specified, the pipeline will be created at the root level.

Required range: x >= 1

dataset_version_number · integer | null

Optional specific dataset version number to use. If not specified, the latest non-draft version will be used. Cannot be -1 (draft version).

Response

201

application/json

Evaluation pipeline created successfully

success · boolean · required

Indicates if the operation was successful

Example:

true

report_id · integer · required

The unique ID of the created evaluation pipeline. Use this ID to add columns and run evaluations.

Example:

456