European Parliament Proceedings Parallel Corpus 1996-2009
For a detailed description of this corpus, please read:
Europarl: A Parallel Corpus for Statistical Machine Translation, Philipp Koehn, MT Summit 2005, pdf.
Please cite the paper, if you use this corpus in your work.
See also the extended (but earlier) version of the report
(ps,
pdf).
The Europarl parallel corpus is extracted from the proceedings of the European
Parliament. It includes versions in 11 European languages: Romanic (French,
Italian, Spanish, Portuguese), Germanic (English, Dutch, German,
Danish, Swedish), Greek and Finnish.
The goal of the extraction and processing was to generate sentence aligned text
for statistical machine translation systems. For
this purpose we extracted matching items and labeled them with
corresponding document IDs. Using a preprocessor we identified sentence
boundaries. We sentence aligned the data using a tool based on the
Church
and Gale algorithm.
Release v6
On 4 February 2011 we released a further expanded and improved version of the corpus. Previous versions are available here. The corpus is released as a source release with the document files and a sentence aligner, and parallel corpora of language pairs that include English.
Changes since v5
- added 11/2009 - 12/2010 data, now up to around 50 million words per language
- added corpora for 10 more official languages of more recent EU member countries (Bulgarian, Czech, Estonian, Hungarian, Latvian, Lithuanian, Polish, Romanian, Slovak, and Slovene), albeit smaller in size, from 01/2007
- further refined preprocessing, cleaning
All formats contain document (<CHAPTER id>),
speaker (<SPEAKER id name language>), and paragraph (<P>)
mark-up on a separate line. The data is stored in one file per day, and in
smaller units for newer data.
Some documents have the SPEAKER tag attribute LANGUAGE which indicates what language the original speaker was using.
To use the parallel corpora with tools like GIZA++, you want to:
- tokenize the text (required)
- lowercase the text (recommended)
- strip empty lines and their correspondences (required)
- remove lines with XML-Tags (starting with "<") (required)
Download
- source release (text files with preprocessing tools and sentence aligner), 1.3 GB
- tools (preprocessing tools and sentence aligner only), 8.6 KB
- parallel corpus Bulgarian-English, 23 MB, 01/2007-12/2010
- parallel corpus Czech-English, 43 MB, 01/2007-12/2010
- parallel corpus Danish-English, 164 MB, 04/1996-12/2010
- parallel corpus German-English, 172 MB, 04/1996-12/2010
- parallel corpus Greek-English, 125 MB, 04/1996-12/2010
- parallel corpus Spanish-English, 170 MB, 04/1996-12/2010
- parallel corpus Estonian-English, 41 MB, 01/2007-12/2010
- parallel corpus Finnish-English, 163 MB, 01/1997-12/2010
- parallel corpus French-English, 177 MB, 04/1996-12/2010
- parallel corpus Hungarian-English, 43 MB, 01/2007-12/2010
- parallel corpus Italian-English, 172 MB, 04/1996-12/2010
- parallel corpus Lithuanian-English, 41 MB, 01/2007-12/2010
- parallel corpus Latvian-English, 41 MB, 01/2007-12/2010
- parallel corpus Dutch-English, 174 MB, 04/1996-12/2010
- parallel corpus Polish-English, 42 MB, 01/2007-12/2010
- parallel corpus Portuguese-English, 173 MB, 04/1996-12/2010
- parallel corpus Romanian-English, 21 MB, 01/2007-12/2010
- parallel corpus Slovak-English, 43 MB, 01/2007-12/2010
- parallel corpus Slovene-English, 40 MB, 01/2007-12/2010
- parallel corpus Swedish-English, 155 MB, 01/1997-12/2010
Size of the Corpus
Sizes for single-language data after tokenizing and removing XML.
| Language | Sentences | Words |
|---|
| Bulgarian | 229,649 | - |
| Czech | 479,636 | 10,770,230 |
| Danish | 2,117,839 | 49,615,228 |
| German | 1,985,560 | 48,648,697 |
| Greek | 1,344,198 | - |
| English | 2,032,006 | 54,720,731 |
| Spanish | 1,942,761 | 55,105,479 |
| Estonian | 493,198 | 9,455,337 |
| Finnish | 1,929,054 | 35,799,132 |
| French | 2,002,266 | 57,860,307 |
| Hungarian | 479,676 | 10,601,411 |
| Italian | 1,905,555 | 52,306,430 |
| Lithuanian | 493,204 | 9,731,052 |
| Latvian | 473,276 | 10,024,350 |
| Dutch | 2,147,195 | 53,459,456 |
| Polish | 387,537 | 8,142,067 |
| Portuguese | 1,942,700 | 53,799,459 |
| Romanian | 224,805 | 5,891,952 |
| Slovak | 487,416 | 10,783,688 |
| Slovene | 465,985 | 10,616,127 |
| Swedish | 2,037,945 | 45,562,972 |
Sizes for parallel corpora after sentence aligning, tokenizing, and removing XML.
| Parallel Corpus (L1-L2) | Sentences | L1 Words | English Words |
|---|
| Bulgarian-English | 226,768 | - | 6,011,944 |
| Czech-English | 462,351 | 10,573,983 | 12,296,772 |
| Danish-English | 1,785,775 | 46,102,455 | 48,833,481 |
| German-English | 1,739,154 | 45,607,269 | 47,978,832 |
| Greek-English | 1,064,544 | - | 30,325,647 |
| Spanish-English | 1,786,594 | 51,551,485 | 49,411,045 |
| Estonian-English | 469,622 | 9,318,986 | 12,452,336 |
| Finnish-English | 1,742,553 | 34,123,013 | 47,601,416 |
| French-English | 1,825,077 | 54,568,499 | 50,551,047 |
| Hungarian-English | 455,270 | 10,429,935 | 12,111,122 |
| Italian-English | 1,737,081 | 49,065,283 | 49,981,015 |
| Lithuanian-English | 456,796 | 9,489,997 | 12,144,335 |
| Latvian-English | 453,879 | 9,854,124 | 1,2051,769 |
| Dutch-English | 1,822,036 | 50,315,412 | 49,938,127 |
| Polish-English | 448,433 | 10,317,697 | 11,910,117 |
| Portuguese-English | 1,783,437 | 50,267,741 | 49,634,127 |
| Romanian-English | 222,854 | 5,866,203 | 5,908,150 |
| Slovak-English | 460,780 | 10,602,998 | 12,228,702 |
| Slovene-English | 456,818 | 10,475,913 | 12,121,729 |
| Swedish-English | 1,678,333 | 41,031,740 | 45,628,613 |
Test Sets
Several test sets have been released for the Europarl corpus.
In general, the Q4/2000 portion of the data (2000-10 to 2000-12) should be reserved for
testing. All released test sets have been selected from this quarter. The shared tasks for the 2006 and 2007 ACL Workshops on
Statistical Machine Translation provide test sets from the Europarl corpus.
The original common test set from the Koehn/Och/Marcu ACL 2003 Paper is available in the archives.
Extended versions of these test sets are available in the Evaluation Matrix of the EuroMatrix project.
Known Bugs
- Some special HTML entities and noisy characters are not removed from the data.
- Some recent Greek data has only parts of transcripts in the files.
Terms of Use
We are not aware of any copyright restrictions of the material.
If you use this data in your research, please contact
pkoehn@inf.ed.ac.uk.
Please let us know if you find problems with the data
or if you want the data for other language pairs.
We recommend using the last quarter of 2000 for testing
(2000-10 until 2000-12) for consistency in reporting
research results on this data.
Acknowledgments
The work was in part supported by the EuroMatrixPlus project funded by the European Commission (7th Framework Programme).