BLEU: a Method for Automatic Evaluation of Machine Translation

Inside the paper

The problem

Human evaluation of machine translation is valuable but slow and costly, making frequent comparison of systems difficult. An automatic measure needs to tolerate multiple valid translations while discouraging meaningless word repetition.

The idea

Compare a system’s output with one or more human references using short sequences of words, then penalize translations that are too short. The resulting score is intended for a corpus of translations.

How it works

• Count matching word sequences at several lengths, limiting each candidate count by the largest count found in any reference.
• Aggregate those clipped counts across the test corpus and calculate a precision for each sequence length.
• Combine the precisions with a geometric mean, so poor matching at any included length affects the overall score.
• Apply a brevity penalty based on candidate and reference lengths, preventing a system from scoring well by emitting only a few safe words.

What it showed

The authors compare system rankings from the metric with judgments by monolingual and bilingual human evaluators. On their test sets, the automatic scores track the aggregate human rankings, though the paper notes that human judgment remains necessary.

Read it for

Start with the repeated-word example in Section 2.1 and the length discussion in Section 2.2, then read Section 5 for the human comparison. The detailed judgment procedure in Section 4 can wait.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.