When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
Paper • 2608.03994 • Published • 5
AI & ML interests
None defined yet.
Recent Activity
View all activity
cschroeder
submitted a
paper to Daily Papers 2 days ago
espejelomar
posted an update 3 months ago
Post
4749
Sharing WorldForge with @abdelstark
It's an open-source Python project for evaluating and replaying robotics and world-model workflows.
The useful part is not only calling a model. WorldForge records the run, validates action shapes, translates outputs into actions, and keeps replay artifacts you can inspect later.
The current demo uses LeRobot + LeWorldModel on PushT through the official loader:
The harness also has replay-only paths for Cosmos-Policy and GR00T-style outputs, so you can inspect the provider contract from saved artifacts without keeping a GPU server online.
Try it:
Repo: https://github.com/AbdelStark/worldforge
Docs: https://abdelstark.github.io/worldforge/
Pre-1.0, MIT, and actively looking for contributors. Good areas:
- robotics provider adapters
- replay artifacts
- eval flows
- docs & first-run demos
Good first issues: https://github.com/AbdelStark/worldforge/contribute
If you're building robot policy evals or model adapters, would love a PR — or an issue describing what's missing.
It's an open-source Python project for evaluating and replaying robotics and world-model workflows.
The useful part is not only calling a model. WorldForge records the run, validates action shapes, translates outputs into actions, and keeps replay artifacts you can inspect later.
The current demo uses LeRobot + LeWorldModel on PushT through the official loader:
stable_worldmodel.policy.AutoCostModel("pusht/lewm")The harness also has replay-only paths for Cosmos-Policy and GR00T-style outputs, so you can inspect the provider contract from saved artifacts without keeping a GPU server online.
Try it:
pip install worldforge-aiuv run --extra harness worldforge-harness --flow robotics-compareRepo: https://github.com/AbdelStark/worldforge
Docs: https://abdelstark.github.io/worldforge/
Pre-1.0, MIT, and actively looking for contributors. Good areas:
- robotics provider adapters
- replay artifacts
- eval flows
- docs & first-run demos
Good first issues: https://github.com/AbdelStark/worldforge/contribute
If you're building robot policy evals or model adapters, would love a PR — or an issue describing what's missing.
christopher
authored a
paper 8 months ago
christopher
authored a
paper 10 months ago
christopher
posted an update 10 months ago
fschlatt
authored 2
papers 10 months ago
adhakimi
authored a
paper over 1 year ago
cschroeder
posted an update over 1 year ago
Post
618
🔥 𝐅𝐢𝐧𝐚𝐥 𝐂𝐚𝐥𝐥 𝐚𝐧𝐝 𝐃𝐞𝐚𝐝𝐥𝐢𝐧𝐞 𝐄𝐱𝐭𝐞𝐧𝐬𝐢𝐨𝐧: Survey on Data Annotation and Active Learning
Short summary: We need your support for a web survey in which we investigate how recent advancements in natural language processing, particularly LLMs, have influenced the need for labeled data in supervised machine learning — with a focus on, but not limited to, active learning. See the original post for details.
➡️ Extended Deadline: January 26th, 2025.
Please consider participating or sharing our survey! (If you have any experience with supervised learning in natural language processing, you are eligible to participate in our survey.)
Survey: https://bildungsportal.sachsen.de/umfragen/limesurvey/index.php/538271
Short summary: We need your support for a web survey in which we investigate how recent advancements in natural language processing, particularly LLMs, have influenced the need for labeled data in supervised machine learning — with a focus on, but not limited to, active learning. See the original post for details.
➡️ Extended Deadline: January 26th, 2025.
Please consider participating or sharing our survey! (If you have any experience with supervised learning in natural language processing, you are eligible to participate in our survey.)
Survey: https://bildungsportal.sachsen.de/umfragen/limesurvey/index.php/538271
cschroeder
posted an update over 1 year ago
Post
505
Here’s just one of the many exciting questions from our survey. If these topics resonate with you and you have experience working on supervised learning with text (i.e., supervised learning in Natural Language Processing), we warmly invite you to participate!
Survey: https://bildungsportal.sachsen.de/umfragen/limesurvey/index.php/538271
Estimated time required: 5–15 minutes
Deadline for participation: January 12, 2025
—
❤️ We’re seeking responses from across the globe! If you know 1–3 people who might qualify for this survey—particularly those in different regions—please share it with them. We’d really appreciate it!
#NLProc #ActiveLearning #ML
Survey: https://bildungsportal.sachsen.de/umfragen/limesurvey/index.php/538271
Estimated time required: 5–15 minutes
Deadline for participation: January 12, 2025
—
❤️ We’re seeking responses from across the globe! If you know 1–3 people who might qualify for this survey—particularly those in different regions—please share it with them. We’d really appreciate it!
#NLProc #ActiveLearning #ML
- 2 replies
cschroeder
posted an update over 1 year ago
Post
379
💡𝗟𝗼𝗼𝗸𝗶𝗻𝗴 𝗳𝗼𝗿 𝘀𝘂𝗽𝗽𝗼𝗿𝘁: 𝗛𝗮𝘃𝗲 𝘆𝗼𝘂 𝗲𝘃𝗲𝗿 𝗵𝗮𝗱 𝘁𝗼 𝗼𝘃𝗲𝗿𝗰𝗼𝗺𝗲 𝗮 𝗹𝗮𝗰𝗸 𝗼𝗳 𝗹𝗮𝗯𝗲𝗹𝗲𝗱 𝗱𝗮𝘁𝗮 𝘁𝗼 𝗱𝗲𝗮𝗹 𝘄𝗶𝘁𝗵 𝗮𝗻 𝗡𝗟𝗣 𝘁𝗮𝘀𝗸?
Are you working on Natural Language Processing tasks and have faced the challenge of a lack of labeled data before? 𝗪𝗲 𝗮𝗿𝗲 𝗰𝘂𝗿𝗿𝗲𝗻𝘁𝗹𝘆 𝗰𝗼𝗻𝗱𝘂𝗰𝘁𝗶𝗻𝗴 𝗮 𝘀𝘂𝗿𝘃𝗲𝘆 to explore the strategies used to address this bottleneck, especially in the context of recent advancements, including but not limited to large language models.
The survey is non-commercial and conducted solely for academic research purposes. The results will contribute to an open-access publication that also benefits the community.
👉 With only 5–15 minutes of your time, you would greatly help to investigate which strategies are used by the #NLP community to overcome a lack of labeled data.
❤️How you can help even more: If you know others working on supervised learning and NLP, please share this survey with them—we’d really appreciate it!
Survey: https://bildungsportal.sachsen.de/umfragen/limesurvey/index.php/538271
Estimated time required: 5–15 minutes
Deadline for participation: January 12, 2025
#NLP #ML
Are you working on Natural Language Processing tasks and have faced the challenge of a lack of labeled data before? 𝗪𝗲 𝗮𝗿𝗲 𝗰𝘂𝗿𝗿𝗲𝗻𝘁𝗹𝘆 𝗰𝗼𝗻𝗱𝘂𝗰𝘁𝗶𝗻𝗴 𝗮 𝘀𝘂𝗿𝘃𝗲𝘆 to explore the strategies used to address this bottleneck, especially in the context of recent advancements, including but not limited to large language models.
The survey is non-commercial and conducted solely for academic research purposes. The results will contribute to an open-access publication that also benefits the community.
👉 With only 5–15 minutes of your time, you would greatly help to investigate which strategies are used by the #NLP community to overcome a lack of labeled data.
❤️How you can help even more: If you know others working on supervised learning and NLP, please share this survey with them—we’d really appreciate it!
Survey: https://bildungsportal.sachsen.de/umfragen/limesurvey/index.php/538271
Estimated time required: 5–15 minutes
Deadline for participation: January 12, 2025
#NLP #ML
christopher
posted an update over 1 year ago
Post
2134
The folks at Foursquare released a dataset of 104.5 million places of interest ( foursquare/fsq-os-places) and here's all of them on a plot
- 4 replies
christopher
posted an update over 1 year ago
Post
2465
The Lichess database of games, puzzles, and engine evaluations is now on the Hub: 
Lichess
Billions of chess data points to download, query, and stream and we're excited to see what you'll build with it! ♟️ 🤗
- https://huggingface.co/collections/Lichess/positions-datasets-66f50837db5cd3287d60d489
- https://huggingface.co/collections/Lichess/games-datasets-66f508df78f4b43e1bb2d353
Billions of chess data points to download, query, and stream and we're excited to see what you'll build with it! ♟️ 🤗
- https://huggingface.co/collections/Lichess/positions-datasets-66f50837db5cd3287d60d489
- https://huggingface.co/collections/Lichess/games-datasets-66f508df78f4b43e1bb2d353
cschroeder
posted an update over 1 year ago
Post
1104
🐣 New release: small-text v2.0.0.dev1
With small language models on the rise, the new version of small-text has been long overdue! Despite the generative AI hype, many real-world tasks still rely on supervised learning—which is reliant on labeled data.
Highlights:
- Four new query strategies: Try even more combinations than before.
- Vector indices integration: HNSW and KNN indices are now available via a unified interface and can easily be used within your code.
- Simplified installation: We dropped the torchtext dependency and cleaned up a lot of interfaces.
Github: https://github.com/webis-de/small-text
👂 Try it out for yourself! We are eager to hear your feedback.
🔧 Share your small-text applications and experiments in the newly added showcase section.
🌟 Support the project by leaving a star on the repo!
#activelearning #nlproc #machinelearning
With small language models on the rise, the new version of small-text has been long overdue! Despite the generative AI hype, many real-world tasks still rely on supervised learning—which is reliant on labeled data.
Highlights:
- Four new query strategies: Try even more combinations than before.
- Vector indices integration: HNSW and KNN indices are now available via a unified interface and can easily be used within your code.
- Simplified installation: We dropped the torchtext dependency and cleaned up a lot of interfaces.
Github: https://github.com/webis-de/small-text
👂 Try it out for yourself! We are eager to hear your feedback.
🔧 Share your small-text applications and experiments in the newly added showcase section.
🌟 Support the project by leaving a star on the repo!
#activelearning #nlproc #machinelearning
cschroeder
posted an update over 1 year ago
Post
710
#EMNLP2024 is happening soon! Unfortunately, I will not be on site, but I will present our poster virtually on Wednesday, Nov 13 (7:45 EST / 13:45 CEST) in Virtual Poster Session 2.
In this work, we leverage self-training in an active learning loop in order to train small language models with even less data. Hope to see you there!
In this work, we leverage self-training in an active learning loop in order to train small language models with even less data. Hope to see you there!
- 1 reply
cschroeder
posted an update almost 2 years ago
Post
409
⚖️ 𝐀𝐈 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐢𝐬 𝐂𝐨𝐩𝐲𝐫𝐢𝐠𝐡𝐭 𝐈𝐧𝐟𝐫𝐢𝐧𝐠𝐞𝐦𝐞𝐧𝐭
This bold claim is not my opinion, but it has been made in a recent "report" of a group, whose stance is recognizable in their name. It is roughly translated as "Authors' Rights Initiative". They published a report which was also presented before the EU Parliament according to the LinkedIn post below.
I am not really interested in politics, but as an EU citizen I am of course somewhat interested in a reasonable and practical version of the EU AI Act. Not saying there should not be rules around data and AI, but this report is obviously very biased towards one side.
While I think the report itself does not deserve attention, I post it in the hope that you find more examples, where they did not address the issue adequately. Feel free to add to my LinkedIn posts (where the original authors will see it) or here.
[en] Executive summary: https://urheber.info/media/pages/diskurs/ai-training-is-copyright-infringement/3b900058e6-1725460935/executive-summary_engl_final_29-08-2024.pdf
[de] Full report: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4946214
LinkedIn: https://www.linkedin.com/posts/activity-7238912869268959232-6cFx
This bold claim is not my opinion, but it has been made in a recent "report" of a group, whose stance is recognizable in their name. It is roughly translated as "Authors' Rights Initiative". They published a report which was also presented before the EU Parliament according to the LinkedIn post below.
I am not really interested in politics, but as an EU citizen I am of course somewhat interested in a reasonable and practical version of the EU AI Act. Not saying there should not be rules around data and AI, but this report is obviously very biased towards one side.
While I think the report itself does not deserve attention, I post it in the hope that you find more examples, where they did not address the issue adequately. Feel free to add to my LinkedIn posts (where the original authors will see it) or here.
[en] Executive summary: https://urheber.info/media/pages/diskurs/ai-training-is-copyright-infringement/3b900058e6-1725460935/executive-summary_engl_final_29-08-2024.pdf
[de] Full report: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4946214
LinkedIn: https://www.linkedin.com/posts/activity-7238912869268959232-6cFx
christopher
posted an update almost 2 years ago
Post
1356
4 million chess puzzles
cschroeder
posted an update almost 2 years ago
Post
910
🌟 Liger Kernel: Efficient Triton Kernels for LLM Training
LIGER "is a [Hugging Face-compatible] collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduces memory usage by 60%."
GitHub: https://github.com/linkedin/Liger-Kernel
LIGER "is a [Hugging Face-compatible] collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduces memory usage by 60%."
GitHub: https://github.com/linkedin/Liger-Kernel
cschroeder
posted an update almost 2 years ago
Post
362
📄 ACL 2024: The Missing Papers
Apparently, some papers from the ACL 2024 are still not listed in the ACL Anthology. While this issue will hopefully be fixed soon, we should give those papers additional spotlight.
Some of my favorites:
1. Dolma is an English corpus that encompasses 3 trillion tokens. Additionally, it is accompanied by an exceptional software package that consdierably advances the state-of-the-art in preparing data for LLM pretraining. (Source: I am currently using Dolma.)
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research (2402.00159)
2. In the paper "Same Task, More Tokens: the Impact of Input Length on
the Reasoning Performance of Large Language Models", the authors show how extending the context length impacts an LLM's reasoning performance. I asked myself a similar question a few months ago, and therefore this paper is highly interesting to me.
Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models (2402.14848)
This was brought to my attention through a Linkedin post by @ShayeghB , who is also affected:
Ensemble-Based Unsupervised Discontinuous Constituency Parsing by Tree Averaging (2403.00143)
View all the missing papers here:
https://theshayegh.github.io/ACL2024MissingPapers/
Apparently, some papers from the ACL 2024 are still not listed in the ACL Anthology. While this issue will hopefully be fixed soon, we should give those papers additional spotlight.
Some of my favorites:
1. Dolma is an English corpus that encompasses 3 trillion tokens. Additionally, it is accompanied by an exceptional software package that consdierably advances the state-of-the-art in preparing data for LLM pretraining. (Source: I am currently using Dolma.)
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research (2402.00159)
2. In the paper "Same Task, More Tokens: the Impact of Input Length on
the Reasoning Performance of Large Language Models", the authors show how extending the context length impacts an LLM's reasoning performance. I asked myself a similar question a few months ago, and therefore this paper is highly interesting to me.
Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models (2402.14848)
This was brought to my attention through a Linkedin post by @ShayeghB , who is also affected:
Ensemble-Based Unsupervised Discontinuous Constituency Parsing by Tree Averaging (2403.00143)
View all the missing papers here:
https://theshayegh.github.io/ACL2024MissingPapers/
cschroeder
posted an update almost 2 years ago
Post
1890
🔔 Release: small-text v1.4.1
The new release contains some smaller bugfixes. Check it out!
Github: https://github.com/webis-de/small-text
Paper: Small-Text: Active Learning for Text Classification in Python (2107.10314)
The new release contains some smaller bugfixes. Check it out!
Github: https://github.com/webis-de/small-text
Paper: Small-Text: Active Learning for Text Classification in Python (2107.10314)