The recurring translation task of the WMT workshops focuses mainly on European language pairs, but this year we have introduced English-Hindi as an experimental, low resource language pair. Translation quality will be evaluated on a shared, unseen test set of news stories. We provide a parallel corpus as training data, a baseline system, and additional resources for download. Participants may augment the baseline system or use their own system.
The goals of the shared translation task are:
We provide training data for five language pairs, and a common framework (including a baseline system). The task is to improve methods current methods. This can be done in many ways. For instance participants could try to:
You may participate in any or all of the following language pairs:
We also strongly encourage your participation, if you use your own training corpus, your own sentence alignment, your own language model, or your own decoder.
If you use additional training data or existing translation systems, you must flag that your system uses additional data. We will distinguish system submissions that used the provided training data (constrained) from submissions that used significant additional data resources. Note that basic linguistic tools such as taggers, parsers, or morphological analyzers are allowed in the constrained condition.
Your submission report should highlight in which ways your own methods and data differ from the standard task. We may break down submitted results in different tracks, based on what resources were used. We are mostly interested in submission that are constraint to the provided training data, so that the comparison is focused on the methods, not on the data used. You may submit contrastive runs to demonstrate the benefit of additional training data.
The provided data is mainly taken from version 7 of the Europarl corpus, which is freely available. Please click on the links below to download the sentence-aligned data, or go to the Europarl website for the source release. Note that this the same data as last year, since Europarl is not anymore translted across all 23 official European languages.
Additional training data is taken from the new News Commentary corpus. There are about 50 million words of training data per language from the Europarl corpus and 3 million words from the News Commentary corpus.
A new data resource from 2013 is the Common Crawl corpus which was collected from web sources. Each parallel corpus comes with a annotation file that gives the source of each sentence pair.
For English-Hindi the parallel training data will consist of the new
HindEnCorp, collected by Charles University, Prague.
In addition, there will be a new, improved release of the English-Hindi section of the JHU Indic corpus, before
the end of 2013. For development we supply a corpus of news data translated specifically for the
You may also use the following monolingual corpora released by the LDC:
Note that the released data is not tokenized and includes sentences of any length (including empty sentences). All data is in Unicode (UTF-8) format. The following tools allow the processing of the training data into tokenized format:
To evaluate your system during development, we suggest using the 2013 test set. The data is provided in raw text format and in an SGML format that suits the NIST scoring tool. We also release other test sets from previous years.
The news-test2011 set has three additional Czech translations that you may want to use. You can download them from Charles University.
|Europarl v7||628MB||✓||✓||✓||same as previous year, corpus home page|
|876MB||✓||✓||✓||✓||same as previous year|
|UN corpus||2.3GB||✓||same as previous year, corpus home page|
|News Commentary||77MB||✓||✓||✓||✓||updated, data with document boundaries|
|CzEng 1.0||115MB||✓||same as previous year, corpus home page (avoid sections 98 and 99)|
|121MB||✓||corpus home page; v1.3 now in original case|
|7.8MB||✓||✓||Provided by CMU. The ru-en is unchanged from last year.|
|25MB||✓||Collected by Charles University|
|This is fully contained in HindEnCorp, so not made available here.|
|News Crawl: articles from 2007||3.7MB||92MB||198MB||6.0MB||302MB||
Extracted article text from various online news publications.
The data sets from 2007-2012 are the same as last year's.
|News Crawl: articles from 2008||191MB||313MB||672MB||1.2MB||244MB||2.3MB||1.5GB|
|News Crawl: articles from 2009||194MB||296MB||757MB||2.8MB||233MB||5.1MB||1.6GB|
|News Crawl: articles from 2010||107MB||135MB||345MB||99MB||2.5MB||727MB|
|News Crawl: articles from 2011||389MB||746MB||784MB||9.9MB||317MB||564MB||3.1GB|
|News Crawl: articles from 2012||337MB||946MB||751MB||4.0KB||218MB||568MB||3.1GB|
|News Crawl: articles from 2013||395MB||1.6GB||1.1GB||62MB||474MB||730MB||4.3GB|
To submit your results, please first convert into into SGML format as required by the NIST BLEU scorer, and then upload it to the website matrix.statmt.org.
Each submitted file has to be in a format that is used by standard scoring scripts such as NIST BLEU or TER.
This format is similar to the one used in the source test set files that were released, except for:
<tstset trglang="en" setid="newstest2014" srclang="any">, with trglang set to either
ru. Important: srclang is always
The script wrap-xml.perl makes the conversion of a output file in one-segment-per-line format into the required SGML file very easy:
wrap-xml.perl LANGUAGE SRC_SGML_FILE SYSTEM_NAME < IN > OUT
wrap-xml.perl en newstest2014-src.de.sgm Google < decoder-output > decoder-output.sgm
Upload happens in three easy steps:
If you are submitting contrastive runs, please submit your primary system first and mark it clearly as the primary submission.
Evaluation will be done both automatically as well as by human judgement.
Supported by the European Commision
project (grant number 288487)