Create Local Eval Set For Project

POST 
/api/v1/projects/:project_id/localevalset

Create a new local eval set.

Request

Path Parameters

project_id anyrequired

Query Parameters

organization_id any

Cookie Parameters

session any

application/json

Body

required

app_name App Name (string)required

The name of the app.

results

object

required

The eval results.

property name*

object[]

Array [

query

object

Query string

anyOf

MOD1

string

contexts

object

Context strings

anyOf

MOD1

Array [

string

]

response

object

Response string

anyOf

MOD1

string

passing

object

Binary evaluation result (passing or not)

anyOf

MOD1

boolean

feedback

object

Feedback or reasoning for the response

anyOf

MOD1

string

score

object

Score for the response

anyOf

MOD1

number

pairwise_source

object

Used only for pairwise and specifies whether it is from original order of presented answers or flipped order

anyOf

MOD1

string

invalid_result Invalid Result (boolean)

Whether the evaluation result is an invalid one.

Default value: false

invalid_reason

object

Reason for invalid evaluation.

anyOf

MOD1

string

]

Responses

200

Successful Response

application/json

Schema

Schema

• Array [

project_id uuidrequired

The ID of the project.

eval_set_id

object

The ID of the local eval result set.

anyOf

MOD1

string

app_name App Name (string)required

The name of the app.

eval_name Eval Name (string)required

The name of the eval.

result

object

required

Evaluation result, EvaluationResult from llama_index.

Output of an BaseEvaluator.

query

object

Query string

anyOf

MOD1

string

contexts

object

Context strings

anyOf

MOD1

Array [

string

]

response

object

Response string

anyOf

MOD1

string

passing

object

Binary evaluation result (passing or not)

anyOf

MOD1

boolean

feedback

object

Feedback or reasoning for the response

anyOf

MOD1

string

score

object

Score for the response

anyOf

MOD1

number

pairwise_source

object

Used only for pairwise and specifies whether it is from original order of presented answers or flipped order

anyOf

MOD1

string

invalid_result Invalid Result (boolean)

Whether the evaluation result is an invalid one.

Default value: false

invalid_reason

object

Reason for invalid evaluation.

anyOf

MOD1

string

• ]

Example (from schema)

[
  {
    "project_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "eval_set_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "app_name": "string",
    "eval_name": "string",
    "result": {
      "query": "string",
      "contexts": [
        null
      ],
      "response": "string",
      "passing": true,
      "feedback": "string",
      "score": 0,
      "pairwise_source": "string",
      "invalid_result": false,
      "invalid_reason": "string"
    }
  }
]

422

Validation Error

application/json

Schema

Schema

detail

object[]

Array [

loc

object[]

required

Array [

anyOf

MOD1

string

MOD2

integer

]

msg Message (string)required

type Error Type (string)required

]

Example (from schema)

{
  "detail": [
    {
      "loc": [
        "string",
        0
      ],
      "msg": "string",
      "type": "string"
    }
  ]
}