Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics

Details

A single-author study of CC-MAIN-2021-31-PDF-UNTRUNCATED (7.9M web PDFs, 32.6B tokens) reveals a fundamental measurement problem in training corpus reporting: statistics are published per document, but token distribution is heavily skewed. Just 3.02% of text-bearing documents hold half the tokens, with a Gini coefficient of 0.807. Documents over 50 pages are 5% of the corpus but 53.53% of its text. The clearest casualty is Common Crawl's truncation cap, which affects 23.06% of documents — but 63.08% of the corpus text. Recovery libraries used to reclaim truncated content perform poorly: the two most widely used recover only 11.4% and 1.4% of affected text respectively, with 72-97% of affected documents yielding nothing. Net result: 55-62% of the corpus's text is silently lost. PDFs from TeX toolchains — covering scientific papers and technical documents — are 1.66% of documents but 4.05% of text, making them disproportionately affected by truncation. Even under Common Crawl's newer 5 MiB cap (March 2025), 30.19% of tokens would still be truncated. The paper recommends reporting corpus statistics in both document and token units — a simple change that would make these losses visible.