Raw data

This is all the pure text right after splitting into languages using Google's Compact Language Detector 2. The data is very noisy and contains a lot of boilerplate but it is also very much.

The file format is a very simple plain-text format:

df6fa1abb58549287111ba8d776733e9 1.000000 url
content
...

where df6fa1abb58549287111ba8d776733e9 is a magic number to mark the start of a new block. The second element is a number that gives the index of the extracted block which can be useful if blocks in different languages were found on a single page.

Since at this point no tokenization or sentence splitting is done we report the size in byes:

@inproceedings{Buck-commoncrawl,
 author = {Christian Buck and Kenneth Heafield and Bas van Ooyen},
 title = {N-gram Counts and Language Models from the Common Crawl},
 year = {2014},
 month = {May},
 booktitle = {Proceedings of the Language Resources and Evaluation Conference},
 address = {Reykjavk, Iceland{i}k, Iceland}
}