Our paper was presented at ACL 2025 (Industry Track).
- Presenting PROMPTCUE, a straightforward approach for the direct meta-evaluation of LLM-as-judge models in Machine Translation.
- Upgrading the conventional MQM scoring system to assess DA at the span level using a detailed scoring rubric. This approach enhances correlation and is especially apt for assessing top-tier models.
- Pinpointing major weaknesses of GEMBA-MQM and rectifying them by examining fundamental prompt structures.
RUBRIC-MQM : Span-Level LLM-as-judge in Machine Translation For High-End Models
Abstract Paper Code BibTeX
Referred to as LLM-as-judge, a generative large language model (LLM) has demonstrated considerable efficacy as an evaluator in various tasks, including Machine Translation (LAJ-MT) by predicting scores or identifying error types for individual sentences. However, its dependability in practical application has yet to be demonstrated, as there is only an approximated match due to the taskโs open-ended nature. To address this problem, we introduce a straightforward and novel meta-evaluation strategy PromptCUE and evaluate cutting-edge LAJ-MT models such as GEMBA-MQM. We identify their fundamental deficits, including certain label biases and the inability to assess near-perfect translations.To improve reliability, we investigate more trustworthy and less biased models using multidimensional prompt engineering. Our findings indicate that the combination of span-level error quantification and a rubric-style prompt tailored to the characteristics of LLMs has efficiently addressed the majority of the challenges current LAJ-MT models face. Furthermore, it demonstrates a considerably enhanced alignment with human values. Accordingly, we present Rubric-MQM, the LAJ-MT for high-end models and an updated version of GEMBA-MQM.