Note Includes both the raw (`/data`) and tokenised (`/tokenized`) the Pile deduplicated data. See README.md for more details.

Note Includes the tokenised and packed data in the exact order they were seen by the Pythia "dedup" models. Note that some documents are repeated (though in a different order) because the "dedup" models are trained for ~1.5 epochs in order to keep an equal token count with respect to the "non-dedup" version.

Note Includes a simple token count. Note that, since we now have the preshuffled data in a parquet format (`pietrolesci/pile-deduped-pythia-preshuffled`), recreating this dataset only requires a simply duckdb SQL query, which runs in a few minutes locally. See more in the README.md.

Note Includes the validation set for the Pile. This split has not been seen by Pythia models, thus it can be used for evaluation.