Retrieval assistant — an acceptance test pack

A grounded assistant needs tests for retrieval, answer quality and the action a user takes afterwards.

Prepare representative questions

Create questions from real work. Include straightforward requests, ambiguous questions, outdated information, requests outside scope and examples with no answer in the approved corpus. Keep a held-out set that the builders do not use to tune the system.

Score the retrieved evidence

Before judging the generated answer, inspect whether the system found the relevant material. Mark missing evidence, irrelevant passages and inaccessible sources. Separate failures in the source collection from failures in retrieval.

Score the answer

Review factual support, completeness, citation accuracy and handling of uncertainty. Check whether a citation supports the specific statement beside it. A source link by itself does not establish that the answer is grounded.

Test boundaries

Attempt to access information the user should not see. Test instructions embedded in retrieved content, conflicting documents and requests to make unsupported commitments. Confirm that the service can decline, clarify and hand over.

Record the release decision

Report errors by category rather than one average score. State which issues block release, which need monitoring and who accepts the remaining risk. Repeat the relevant tests when the model or corpus changes.

AIStrategy.gr working resource · 6 November 2026. Adapt this template to the actual workflow and responsible teams.