Execute Eval Dataset
POST/api/v1/pipelines/:pipeline_id/eval-datasets/:eval_dataset_id/execute
Execute a dataset.
Request
Path Parameters
Cookie Parameters
- application/json
Body
required
- SupportedLLMModelNames
- MOD1
params
object
The parameters for the eval execution that will override the ones set in the pipeline.
llm_model
object
The LLM model to use within eval execution.
anyOf
string
Possible values: [GPT_3_5_TURBO
, GPT_4
, GPT_4_TURBO
, GPT_4O
, GPT_4O_MINI
, AZURE_OPENAI
]
qa_prompt_tmpl
object
The template to use for the question answering prompt.
anyOf
string
Responses
- 200
- 422
Successful Response
- application/json
- Schema
- Example (from schema)
Schema
- EvalDatasetJobParams
- MOD1
- MOD1
- MOD1
- MOD1
- MOD1
- MOD1
- MOD1
- MOD1
- MOD1
- Base
Possible values: [eval_dataset_job
]
partitions
object
required
The partitions for this execution. Used for determining where to save job output.
parameters
object
Additional input parameters for the eval execution.
anyOf
The IDs for the EvalQuestions this execution ran against.
eval_execution_params
object
required
Schema for the params for an eval execution.
Possible values: [GPT_3_5_TURBO
, GPT_4
, GPT_4_TURBO
, GPT_4O
, GPT_4O_MINI
, AZURE_OPENAI
]
Default value: GPT_4O
The LLM model to use within eval execution.
Default value: `Context information is below.
{context_str}
Given the context information and not prior knowledge, answer the query. Query: {query_str} Answer: `
The template to use for the question answering prompt.
session_id
object
The upstream request ID that created this job. Used for tracking the job across services.
anyOf
string
correlation_id
object
The correlation ID for this job. Used for tracking the job across services.
anyOf
string
parent_job_execution_id
object
The ID of the parent job execution.
anyOf
string
user_id
object
The ID of the user that created this job
anyOf
string
Creation datetime
Unique identifier
Possible values: [PENDING
, SUCCESS
, ERROR
, PARTIAL_SUCCESS
]
Enum for representing the status of a job
error_code
object
anyOf
string
error_message
object
anyOf
string
attempts
object
The number of times this job has been attempted
anyOf
integer
started_at
object
anyOf
string
ended_at
object
anyOf
string
Update datetime
data
object
Additional metadata for the job execution.
anyOf
{
"job_name": "eval_dataset_job",
"partitions": {},
"parameters": {},
"session_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"correlation_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"parent_job_execution_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"user_id": "string",
"created_at": "2024-10-03T01:48:33.678Z",
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"status": "PENDING",
"error_code": "string",
"error_message": "string",
"attempts": 0,
"started_at": "2024-10-03T01:48:33.678Z",
"ended_at": "2024-10-03T01:48:33.678Z",
"updated_at": "2024-10-03T01:48:33.678Z",
"data": {}
}
Validation Error
- application/json
- Schema
- Example (from schema)
Schema
Array [
Array [
- MOD1
- MOD2
]
]
detail
object[]
loc
object[]
required
anyOf
string
integer
{
"detail": [
{
"loc": [
"string",
0
],
"msg": "string",
"type": "string"
}
]
}