European Parliament Proceedings Parallel Corpus 1996-2011


For a detailed description of this corpus, please read:

Europarl: A Parallel Corpus for Statistical Machine Translation, Philipp Koehn, MT Summit 2005, pdf.

Please cite the paper, if you use this corpus in your work. See also the extended (but earlier) version of the report (ps, pdf).

The Europarl parallel corpus is extracted from the proceedings of the European Parliament. It includes versions in 21 European languages: Romanic (French, Italian, Spanish, Portuguese, Romanian), Germanic (English, Dutch, German, Danish, Swedish), Slavik (Bulgarian, Czech, Polish, Slovak, Slovene), Finni-Ugric (Finnish, Hungarian, Estonian), Baltic (Latvian, Lithuanian), and Greek.

The goal of the extraction and processing was to generate sentence aligned text for statistical machine translation systems. For this purpose we extracted matching items and labeled them with corresponding document IDs. Using a preprocessor we identified sentence boundaries. We sentence aligned the data using a tool based on the Church and Gale algorithm.


Release v7

On 15 May 2012 we released a further expanded and improved version of the corpus. Previous versions are available here. The corpus is released as a source release with the document files and a sentence aligner, and parallel corpora of language pairs that include English.

Changes since v6

All formats contain document (<CHAPTER id>), speaker (<SPEAKER id name language>), and paragraph (<P>) mark-up on a separate line. The data is stored in one file per day, and in smaller units for newer data.

Some documents have the SPEAKER tag attribute LANGUAGE which indicates what language the original speaker was using.

To use the parallel corpora with tools like GIZA++, you want to:

Download


Size of the Corpus

Sizes for single-language data after removing XML.

LanguageSentencesWords
Bulgarian 411,636-
Czech 668,59513,195,311
Danish 2,323,09947,761,381
German 2,176,53747,236,849
Greek 1,517,141-
English 2,218,20153,974,751
Spanish 2,123,83554,806,927
Estonian 692,21011,358,009
Finnish 2,119,51533,708,706
French 2,190,57954,202,850
Hungarian 658,82412,606,986
Italian 2,081,66950,259,169
Lithuanian 678,66511,512,131
Latvian 666,02612,085,228
Dutch 2,333,81653,487,257
Polish 387,490 7,087,016
Portuguese2,121,88952,300,149
Romanian 402,904 9,663,544
Slovak 674,35913,116,301
Slovene 634,48812,665,974
Swedish 2,241,38645,665,947

Sizes for parallel corpora after sentence aligning and removing XML.

Parallel Corpus (L1-L2)SentencesL1 WordsEnglish Words
Bulgarian-English 406,934 - 9,886,291
Czech-English 646,60512,999,45515,625,264
Danish-English 1,968,80044,654,41748,574,988
German-English 1,920,20944,548,49147,818,827
Greek-English 1,235,976 -31,929,703
Spanish-English 1,965,73451,575,74849,093,806
Estonian-English 651,74611,214,22115,685,733
Finnish-English 1,924,94232,266,34347,460,063
French-English 2,007,72351,388,64350,196,035
Hungarian-English 624,93412,420,27615,096,358
Italian-English 1,909,11547,402,92749,666,692
Lithuanian-English 635,14611,294,69015,341,983
Latvian-English 637,59911,928,71615,411,980
Dutch-English 1,997,77550,602,99449,469,373
Polish-English 632,56512,815,54415,268,824
Portuguese-English1,960,40749,147,82649,216,896
Romanian-English 399,375 9,628,010 9,710,331
Slovak-English 640,71512,942,43415,442,233
Slovene-English 623,49012,525,64415,021,497
Swedish-English 1,862,23441,508,71245,703,795


Test Sets

Several test sets have been released for the Europarl corpus. In general, the Q4/2000 portion of the data (2000-10 to 2000-12) should be reserved for testing. All released test sets have been selected from this quarter. The shared tasks for the 2006 and 2007 ACL Workshops on Statistical Machine Translation provide test sets from the Europarl corpus.

The original common test set from the Koehn/Och/Marcu ACL 2003 Paper is available in the archives.

Extended versions of these test sets are available in the Evaluation Matrix of the EuroMatrix project.

Known Bugs

Terms of Use

We are not aware of any copyright restrictions of the material. If you use this data in your research, please contact phi@jhu.edu. Please let us know if you find problems with the data or if you want the data for other language pairs. We recommend using the last quarter of 2000 for testing (2000-10 until 2000-12) for consistency in reporting research results on this data.

Acknowledgments

The work was in part supported by the EuroMatrixPlus project funded by the European Commission (7th Framework Programme).