Evaluations: test question sets
Evaluations check how well an agent answers reference questions, and follow it over time. Run them again after changing a prompt, a model, a document or a search setting, to see whether answers get better or worse.
Page: Monitoring → Evaluations in the main menu, for workspace administrators.
Create a question set
A question set belongs to the workspace and targets one agent. It holds up to 100 questions. For each one:
| Field | Purpose |
|---|---|
| Question | Asked to the agent as is. Required. |
| Expected answer | Optional. Used for coverage and, when turned on, the AI grade. |
| Expected sources | Optional, up to 10, separated by |. A title or file name, even partial, or a document ID. Case and accents are ignored. |
You can also import a CSV file: one line per question, with the columns question; expected answer; sources. The ; or , separator is detected, a first header line starting with "question" is skipped, and sources in one cell are separated by |.
Start a run
Start a run asks each question to the agent, one after the other, exactly as a real conversation: same settings, same knowledge bases, same guardrails. Test conversations are hidden from history and user statistics. One run at a time per set, and each question gets at most 4 minutes.
Each answer gets up to three measures:
- Sources: were all expected sources found? Only when the question lists some.
- Coverage: share of the important words of the expected answer found in the answer. Simple and free, useful to follow a trend.
- AI grade (1 to 5), off by default: the agent's chat model compares the answer with the expected one and gives a grade with a one-sentence reason. It uses extra tokens on the model already configured; no other model is called.
The run then shows: answered and failed questions, source hit rate, average coverage, average AI grade, average answer time and tokens used.
Read and compare results
Click a run to see each question; click a row to unfold the answer, the sources found and the AI grade comment. Compare with puts two runs side by side, question by question, marked better, worse, same or new.
API
With an administrator account or an API key with the run_evaluations capability: GET/POST /api/v1/eval-suites, GET/PUT/DELETE /api/v1/eval-suites/{id}, GET/POST /api/v1/eval-suites/{id}/runs ({use_judge: false}, 202 response) and GET /api/v1/eval-suites/runs/{run_id}.