Releasing the largest multilingual open pretraining dataset
•
88
Pierre-Carl Langlais
Pclanglais
AI & ML interests
Open data & open LLMs
Articles
Organizations
Posts
6
Post
2466
We release today our first foundation model and experiment with a new category: specialized pre-training.
OCRonos-Vintage is a 124m parameters model trained end-to-end by Pleias on llm.c from 18 billion tokens from cultural heritage archives. Despite its small size it achieve nearly state of the art results for OCR correction of historical English sources. OCRonos-Vintage is also an historical model with an unusual cut-off date: December 29th, 1955…
We look forward to replicate this approach very soon on other "hard" tasks commonly associated with generalist LLMs/SLMs: RAG, function calling, summarization, document segmentation…
OCRonos-Vintage: PleIAs/OCRonos-Vintage
CPU Demo: PleIAs/OCRonos-Vintage-CPU
GPU Demo: PleIAs/OCRonos-Vintage-GPU
Our annoncement and call for specialized pre-training: https://huggingface.co/blog/Pclanglais/specialized-pre-training
OCRonos-Vintage is a 124m parameters model trained end-to-end by Pleias on llm.c from 18 billion tokens from cultural heritage archives. Despite its small size it achieve nearly state of the art results for OCR correction of historical English sources. OCRonos-Vintage is also an historical model with an unusual cut-off date: December 29th, 1955…
We look forward to replicate this approach very soon on other "hard" tasks commonly associated with generalist LLMs/SLMs: RAG, function calling, summarization, document segmentation…
OCRonos-Vintage: PleIAs/OCRonos-Vintage
CPU Demo: PleIAs/OCRonos-Vintage-CPU
GPU Demo: PleIAs/OCRonos-Vintage-GPU
Our annoncement and call for specialized pre-training: https://huggingface.co/blog/Pclanglais/specialized-pre-training
Post
1050
Since it is release season, at PleIAs we announce our first suite of specialized language models for document processing tasks (OCR correction, text segmentation, bibliographic extraction) and the release of the largest multimodal dataset of financial document Finance Commons: https://huggingface.co/blog/Pclanglais/finance-commons-bad-data-toolbox
LLM research is currently focused on quality data. We went on the opposite direction and voluntarily trained models on bad data. Far from degrading models, it made them more resilient to text sources commonly used in production.
Having a wider range of real life data proved critical for this project. A few months after the release of Common Corpus, we expanded our pool of "training data commons" with a major multimodal ressource: document released as open financial data. Finance commons comprises 17 billion tokens and 1.25 PDF corporate documents released by the SEC, WTO, AMF, EU Tenders In a multiple languages with a large variety of document layouts and challenging sources to train more robust models.
With HuggingFace compute support, we release an entire pipeline to process bad data sources and make them usable in production for LLMOps or simply retrieval: PleIAs/PleIAs-Editor
This approach is based on our new series of specialized models for document processing, the "bad data toolbox" comprising:
*OCRonos, the best available model to date for OCR correction. PleIAs/OCRonos
*Segmentext, a pure semantic small model for text segmentation, working without any visual reference. PleIAs/Segmentext
*Bibtexer, a small model for bibliographic data extraction acting as a "reversed-Zotero." PleIAs/BibTexer
LLM research is currently focused on quality data. We went on the opposite direction and voluntarily trained models on bad data. Far from degrading models, it made them more resilient to text sources commonly used in production.
Having a wider range of real life data proved critical for this project. A few months after the release of Common Corpus, we expanded our pool of "training data commons" with a major multimodal ressource: document released as open financial data. Finance commons comprises 17 billion tokens and 1.25 PDF corporate documents released by the SEC, WTO, AMF, EU Tenders In a multiple languages with a large variety of document layouts and challenging sources to train more robust models.
With HuggingFace compute support, we release an entire pipeline to process bad data sources and make them usable in production for LLMOps or simply retrieval: PleIAs/PleIAs-Editor
This approach is based on our new series of specialized models for document processing, the "bad data toolbox" comprising:
*OCRonos, the best available model to date for OCR correction. PleIAs/OCRonos
*Segmentext, a pure semantic small model for text segmentation, working without any visual reference. PleIAs/Segmentext
*Bibtexer, a small model for bibliographic data extraction acting as a "reversed-Zotero." PleIAs/BibTexer
spaces
9
models
35
Pclanglais/Headlines-OCR-Correction
Updated
•
8
Pclanglais/SynthRag3
Updated
•
2
Pclanglais/SynthRag2
Updated
•
2
Pclanglais/SynthRag1
Updated
•
2
Pclanglais/Experiment1
Updated
•
4
Pclanglais/Segmentext-Marianne
Updated
•
2
Pclanglais/OCRonos-Vintage-GGUF
Updated
Pclanglais/French-TV-transcript-NER
Token Classification
•
Updated
•
30
Pclanglais/llama3-instruct-guillaumetell
Text Generation
•
Updated
•
4
Pclanglais/t5-small-common-corpus-topic-simple
Text2Text Generation
•
Updated
•
3
datasets
10
Pclanglais/open-science
Viewer
•
Updated
•
10.8M
•
3
Pclanglais/pdf_sample_10k
Updated
•
12
Pclanglais/LLM-for-DH
Viewer
•
Updated
•
1.62k
•
8
Pclanglais/youtube-commons-metadata
Viewer
•
Updated
•
6.91M
•
19
Pclanglais/OCR-test
Viewer
•
Updated
•
20.1k
•
10
•
1
Pclanglais/AllWikidataCharacters
Viewer
•
Updated
•
180k
•
43
•
7
Pclanglais/wiki-dataset
Viewer
•
Updated
•
282
•
28
Pclanglais/Mickey-1928-dataset
Viewer
•
Updated
•
96
•
34
•
7
Pclanglais/MonadGPT
Viewer
•
Updated
•
10.8k
•
39
•
11
Pclanglais/Brahe-Novels
Viewer
•
Updated
•
8.23k
•
49
•
5