Hugging Face's logo

HPLT

community
https://hplt-project.org/
hplt_eu
hplt-project
Activity Feed Request to join this org

AI & ML interests

Web as a corpus, Large Language Models, Machine Translation, Language Technologies, Natural Language Processing, Internet Archive, CommonCrawl

Recent Activity

MariaFjodorowa  new activity 3 days ago
HPLT/hplt_bert_base_2_0_eng-Latn:Fix AttributeError in _init_weights for LayerNorm
ltgoslo  updated a dataset 9 days ago
HPLT/HPLT2.0_cleaned
ltgoslo  updated a Space 10 days ago
HPLT/README
View all activity

HPLT's activity

MariaFjodorowa 
in HPLT/hplt_bert_base_2_0_eng-Latn 3 days ago

Fix AttributeError in _init_weights for LayerNorm

1
#5 opened 4 days ago by
dschulmeist
davanstrien 
posted an update 6 days ago
view post
Post
2655
Inspired by Hugging Face's official MCP server, I've developed a complementary tool that exposes my semantic search API to enhance discovery across the HF platform.

Key capabilities:

- AI-powered semantic search for models and datasets
- Parameter count analysis via safetensors metadata
- Trending content discovery
- Find similar models/datasets functionality
- 11 tools total for enhanced ecosystem navigation

The semantic search goes beyond simple keyword matching, understanding context and relationships between different models and datasets.

Example query: "Find around 10 reasoning Hugging Face datasets published in 2025 focusing on topics other than maths and science. Show a link and a short summary for each dataset." (results in video!)

https://github.com/davanstrien/hub-semantic-search-mcp
ltgoslo 
updated a dataset 9 days ago
ltgoslo 
updated a Space 10 days ago
vmkhlv 
updated a dataset 10 days ago
vmkhlv 
published 2 datasets 10 days ago
nikolare 
in HPLT/HPLT2.0_cleaned 11 days ago

4-digit Numbers/Charaters in the Doucment Beginning

1
#2 opened 11 days ago by
Liping
Villekom 
updated a collection 11 days ago
Villekom 
updated a model 11 days ago
tiedeman 
authored a paper 12 days ago
jisx 
authored 8 papers 12 days ago
BramVanroy 
posted an update about 1 month ago
view post
Post
3190
📢💾 Introducing the Common Crawl Creative Commons Corpus (C5)!

C5 is a large-scale effort to heavily filter web-crawled data, as collected by the non-profit Common Crawl, to only documents that are Creative Commons-licensed such as cc-by-4.0 or public domain cc0. At this stage 150 billion tokens have been collected.

---
📄 data: BramVanroy/CommonCrawl-CreativeCommons
🧰 software: https://github.com/BramVanroy/CommonCrawl-CreativeCommons
---

</> To build C5, HTML pages are scrutinized and all links (if any) to CC licenses are collected, both in regular hyperlinks as well as in metadata. Additional data fields are included such as "was the license found in the head?" or "if multiple licenses were found, do they contradict each other?", which makes further filtering a breeze.

🌐 In this first version of C5, 8 languages are included (Afrikaans, German, English, French, Frysian, Italian, Dutch and Spanish). The language set was limited for two reasons: computational and storage limitations, and a collaboration with GPT-NL, which requested CC data for these languages to train a Dutch-focused, copyright-conscious LLM. In total, this V1 release contains almost 150 thousand documents and 150 billion tokens. This data was not filtered on quality nor deduplicated so that you can decide for yourself how much data to keep. To give some quality indication, a dataset field is present to describe whether a document is included in the FineWeb(-2) datasets, which are of high quality.

🔍 More work needs to be done! Only 7 out of 100+ Common Crawl crawls have been processed so far. That's encouraging because it means there is a lot more Creative Commons data to be collected! But to get there I need help in terms of compute. The current processing was already heavily sponsored by the Flemish Supercomputer but more is needed. If you have the compute available and which to collaborate in an open and transparent manner, please get in touch!
  • 1 reply
·