The shared evaluation task of the workshop will examine automatic evaluation metrics for machine translation. We will provide all of the translations produced in the translation task along with the reference human translations. You will return rankings for each of the translations at the system-level and/or at the sentence-level. We will calculate the correlation of your rankings with the human judgements once the manual evaluation has been completed.
The goals of the shared evaluation task are:
This task will be run unofficially this year. This means that the results will not be published in the official overview paper but probably in a short paper.
We will provide the output of machine translation systems for five different language pairs (French-English, Spanish-English, German-English, Czech-English, Russian-English), and will give you the reference translations in each of those languages. You will provide scores for each of the outputs at the system-level and the sentence-level. If your automatic metric does not produce sentence-level scores, you can participate in just the system-level ranking. If your automatic metric uses linguistic annotation and cannot work with translations into languages other than English, you are free to assign scores only for translations into English.
We will measure the goodness of automatic evaluation metrics in the following ways:
System-level correlation: We will use Spearman's rank correlation coefficient (rho) to measure the correlation of the automatic metrics with the human judgments of translation quality at the system-level. The human ranking of the systems will be based on the manual evaluation. A system's rank will be assigned based on the percent of time that the sentences it produced were judged to be better than or equal to the translations of any other system. Since automatic metrics generally assign a score rather than a rank, we will convert their raw scores into ranks prior to calculating rho.
Sentence-level correlation: We will use Kendall's tau to measure metrics' correlation with human judgments at the sentence-level. For every pairwise comparison of two systems' output for a single sentence, we will count the automatic metric as being concordunt with the human judgment if it orders the systems' output the same way (i.e. the metric assigned a higher score to the higher ranked system). We will exclude pairs that the human annotators ranked as ties.
Once we receive the system outputs from the translation task and the system combination task we will post all of the system outputs for you to score with your metric. The translations will be distributed as plain text files with one translation per line. (We can also provide the output in the NIST MT Evaluation Workshop's XML format. Please contact us if this format is easier for you.)
The output of your software should produce scores for the translations either at the system-level or the segment-level (or preferably both).
The output files for system-level rankings should be formatted in the following way:
<METRIC NAME> <LANG-PAIR> <TEST SET> <SYSTEM> <SYSTEM LEVEL SCORE>Where:
METRIC NAME
is the name of your automatic evaluation metric.LANG-PAIR
is the language pair using two letter abbreviations for the languages (cz
for Czech, de
for German, en
for English, es
for Spanish, fr
for French). You should use de-en
for German-English, for example.
TEST SET
is the ID of the test set (given by the setid
attribute of of the tstset
tag in the XML file, or by the directory structure in the plain text files).SYSTEM
is the ID of system being scored (given by the sysid
attribute in the XML document, or as part of the filename for the plain text file).SYSTEM LEVEL SCORE
is the overall system level score.
The output files for segment-level rankings should be formatted in the following way:
<METRIC NAME> <LANG-PAIR> <TEST SET> <SYSTEM> <SEGMENT NUMBER> <SEGMENT SCORE>Where:
METRIC NAME
is the name of your automatic evaluation metric.LANG-PAIR
is the language pair using two letter abbreviations for the languages.
TEST SET
is the ID of the test set.
SYSTEM
is the ID of system being scored.
SEGMENT NUMBER
is the line number starting from one of the plain text input files.SEGMENT SCORE
is the score for the particular segment. The system outputs and human judgments from the last three years' workshops is available for download from the following links:
Last year's data contains all of the system's translations, the source documents and reference human translations and the human judgments of the translation quality.
If you participate in the evaluation shared task, we ask you to commit about 8 hours of time to do the manual evaluation. The evaluation will be done with an online tool.
You are invited to submit a short paper (4 to 6 pages) describing your automatic evaluation metric. You are not required to submit a paper if you do not want to. If you don't, we ask that you give an appropriate reference describing your metric that we can cite in the overview paper.
System outputs distributed for metrics task (download tarball) | May 10, 2013 |
Submission deadline for metrics task (email to machacekmatous@gmail.com) | May 31, 2013 |
Paper submission deadline | June 7, 2013 |
Supported by the European Commision
under the
project (grant number 288487)