Dossier

Greek-language AI — evaluate the task, not the fluency

A practical framework for assessing Greek answers, retrieval and specialised terminology.

Why fluent output is insufficient

A system may write convincing Greek while misunderstanding a technical term, missing a qualification or citing an irrelevant source. Define what a correct answer does in the intended workflow, then evaluate that behaviour.

Build the evaluation set

Collect ordinary questions, difficult questions, ambiguous wording and requests the system should decline. Include accents, transliteration, abbreviations, mixed Greek and English, and terms used by actual users. Separate development examples from the final acceptance set.

Evaluate retrieval separately

For a knowledge assistant, first ask whether the right evidence was retrieved. Then test whether the answer faithfully represents that evidence. Combining those stages into one score makes it harder to diagnose whether a failure comes from search, generation or the underlying content.

Use Greek research as a starting point

Athena’s Meltemi announcement demonstrates local language-model research capability. Selecting a model for an application still requires checking its terms, version, operating constraints and task performance. A model release is not a ready-made service.

A review rubric

Score factual correctness, completeness, citation support, ambiguity handling and appropriate refusal. Have domain reviewers inspect errors that would affect a user’s next action. Record both average performance and the categories in which the system fails.

Discussion for EventChanger

Can organisations share non-sensitive Greek evaluation examples and common scoring rubrics? A useful output would be a versioned test set and a documented process for keeping it current.