ZEBRA: Zero-Shot Example-Based Retrieval Augmentation for Commonsense Question Answering
Paper
•
2410.05077
•
Published
•
2
Occiglot
community
AI & ML interests
Open Source Language Models for Europe
Recent Activity
View all activity
occiglot's activity
s-conia
authored
6
papers
1 day ago
Echoes from Alexandria: A Large Resource for Multilingual Book Summarization
Paper
•
2306.04334
•
Published
•
2
Semantic Role Labeling Meets Definition Modeling: Using Natural Language to Describe Predicate-Argument Structures
Paper
•
2212.01094
•
Published
•
2
Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMs
Paper
•
2410.15956
•
Published
Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities
Paper
•
2307.01870
•
Published
Entity Disambiguation with Entity Definitions
Paper
•
2210.05648
•
Published
barthfab
updated
a
dataset
2 days ago
mbrack
authored
a
paper
26 days ago
PSaiml
authored
a
paper
26 days ago
malteos
updated
a
collection
about 1 month ago
Post
1293
My latest project is the outcome of the last 2+ years working with TPUs from the amazing TPU Research Cloud (TRC) program and training Encoder-only LMs with the TensorFlow Model Garden library.
👉 Link: https://github.com/stefan-it/model-garden-lms
An overview of some features:
- Cheatsheet for setting-up a TPU VM Pod (with all necessary dependencies) to pretrain LMs with TF Model Garden
- Conversion scripts that convert TF Model Garden weights to Hugging Face Transformers-compatible models
- Supported architectures include BERT, BERT with Token Dropping and TEAMS
I also released BERT-based models pretrained on the great Hugging Face FineWeb and FineWeb-Edu datasets (10BT subset). With more to come!
👉 Model Hub Link: https://huggingface.co/model-garden-lms
If you find these resources useful, please give them a like!
Made from Bavarian Oberland with ❤️ and 🥨.
👉 Link: https://github.com/stefan-it/model-garden-lms
An overview of some features:
- Cheatsheet for setting-up a TPU VM Pod (with all necessary dependencies) to pretrain LMs with TF Model Garden
- Conversion scripts that convert TF Model Garden weights to Hugging Face Transformers-compatible models
- Supported architectures include BERT, BERT with Token Dropping and TEAMS
I also released BERT-based models pretrained on the great Hugging Face FineWeb and FineWeb-Edu datasets (10BT subset). With more to come!
👉 Model Hub Link: https://huggingface.co/model-garden-lms
If you find these resources useful, please give them a like!
Made from Bavarian Oberland with ❤️ and 🥨.
mbrack
updated
a
dataset
about 1 month ago
BramVanroy
posted
an
update
about 1 month ago
Post
534
In the spirit of "Better late than never", I've finally written a brief overview paper for GEITje 7B Ultra. Initially released 10 months ago (oops), but still reaching around 1300 monthly downloads across the HF ecosystem (not including ollama).
GEITje 7B Ultra: A Conversational Model for Dutch (2412.04092)
While the paper discusses the model a little bit, I especially wanted to write about the datasets, which to this day seem an important asset for Dutch LLM training (SFT and preference tuning). We have a long way to go for Dutch, but publishing transparent and reproducible artefacts seems an important step to me, alongside having open discussions about data, bias, architectures.
In that spirit, thanks are in order for the creation of GEITje 7B Ultra and all related datasets:
- Michiel Buisman and UWV for providing the means to create the datasets
- Flemish Supercomputer Center (VSC) for the compute
- The Hugging Face Fellows and rest of the team for their discussions and insights
- The Dutch NLP community, notably @Rijgersberg for building the base GEITje model and the fruitful discussions we've had
More to come, step by step!
BramVanroy/geitje-7b-ultra-65c1ee010ad80fd1f6a8f208
GEITje 7B Ultra: A Conversational Model for Dutch (2412.04092)
While the paper discusses the model a little bit, I especially wanted to write about the datasets, which to this day seem an important asset for Dutch LLM training (SFT and preference tuning). We have a long way to go for Dutch, but publishing transparent and reproducible artefacts seems an important step to me, alongside having open discussions about data, bias, architectures.
In that spirit, thanks are in order for the creation of GEITje 7B Ultra and all related datasets:
- Michiel Buisman and UWV for providing the means to create the datasets
- Flemish Supercomputer Center (VSC) for the compute
- The Hugging Face Fellows and rest of the team for their discussions and insights
- The Dutch NLP community, notably @Rijgersberg for building the base GEITje model and the fruitful discussions we've had
More to come, step by step!
BramVanroy/geitje-7b-ultra-65c1ee010ad80fd1f6a8f208
mbrack
authored
a
paper
7 months ago
pjox
authored
a
paper
7 months ago
BramVanroy
posted
an
update
8 months ago
Post
1773
The InstructGPT paper mentions that they insert 10% pretraining data during SFT, which they find improves the effect of PPO (IIUC). Has anyone else done later ablations on this? I've only seen the inverse suggested, mixing in SFT data during pretraining.
- 2 replies
BramVanroy
posted
an
update
8 months ago
Post
2299
All my models seem to be plagued by infinite lists. When you ask a question that requires it to write a list, it most often keeps adding bullet points or enumeration. I am wondering whether this is a result of using chatty GPT-4 as DPO preferences. Any thoughts?
- 1 reply
s-conia
authored
a
paper
9 months ago
PSaiml
authored
a
paper
9 months ago