CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs with Controllable Puzzle Generation
Paper
•
2504.00043
•
Published
•
8
Blog-explorers
AI & ML interests
None defined yet.
Recent Activity
View all activity
blog-explorers's activity
yuchenlin
authored
a
paper
6 days ago
eliebak
authored
a
paper
7 days ago
reach-vb
authored
a
paper
7 days ago
Preview dark/light toggle
#9 opened 10 days ago
by
Aurelien-Morgan
osanseviero
authored
a
paper
18 days ago
Razvanip
authored
a
paper
22 days ago
Post
2209
We introduce FAT5 (Flash Attention T5) ⚡
An implementation of T5 in PyTorch with UL2 objective optimized for GPGPU for both training and inference thanks to 13 different optimizations.
The main one is that we have designed a CUDA kernel to expand the Flash Attention by @tridao with RPE biases and supports other PE such as RoPE, ALiBi or FIRE.
The result kernel is 2 times faster than a SPDA implementation.
We also use Triton kernels to optimize certain parts of the architecture, such as the cross-entropy and RMSNorm layer.
The various kernels have been carefully built to be compatible with BF16 and torch.compile to go even faster and achieve efficient pretraining.
All other optimizations are described in a 📝 subsequent blog post available on @huggingface 🤗: CATIE-AQ/FAT5-report.
This methodology enabled us to efficiently pretrain as a proof of concept a FAT5 with 147M parameters in French in a reasonable time (1,461H for 419B tokens), with limited resources (1 A100 i.e. a computational budget of ~ €1,900) and a low carbon footprint (13.5kg eq CO2).
The model's weights are also available on Hugging Face: CATIE-AQ/FAT5-small.
Not very useful in practice, it's a PoC and not an instructed model (it's planned for later).
All the code is available on GitHub if you want to pretrain your own model in your own language or for a specific domain: https://github.com/catie-aq/flashT5 ⭐
Ending by indicating that was a joint project with @BorisAlbar at hf.co/CATIE-AQ.
An implementation of T5 in PyTorch with UL2 objective optimized for GPGPU for both training and inference thanks to 13 different optimizations.
The main one is that we have designed a CUDA kernel to expand the Flash Attention by @tridao with RPE biases and supports other PE such as RoPE, ALiBi or FIRE.
The result kernel is 2 times faster than a SPDA implementation.
We also use Triton kernels to optimize certain parts of the architecture, such as the cross-entropy and RMSNorm layer.
The various kernels have been carefully built to be compatible with BF16 and torch.compile to go even faster and achieve efficient pretraining.
All other optimizations are described in a 📝 subsequent blog post available on @huggingface 🤗: CATIE-AQ/FAT5-report.
This methodology enabled us to efficiently pretrain as a proof of concept a FAT5 with 147M parameters in French in a reasonable time (1,461H for 419B tokens), with limited resources (1 A100 i.e. a computational budget of ~ €1,900) and a low carbon footprint (13.5kg eq CO2).
The model's weights are also available on Hugging Face: CATIE-AQ/FAT5-small.
Not very useful in practice, it's a PoC and not an instructed model (it's planned for later).
All the code is available on GitHub if you want to pretrain your own model in your own language or for a specific domain: https://github.com/catie-aq/flashT5 ⭐
Ending by indicating that was a joint project with @BorisAlbar at hf.co/CATIE-AQ.
Post
1650
Google just dropped an exciting technical report for the brand-new Gemma3 model! 🚀 Here are my personal notes highlighting the most intriguing architectural innovations, design choices, and insights from this release:
1) Architecture choices:
> No more softcaping, replace by QK-Norm
> Both Pre AND Post Norm
> Wider MLP than Qwen2.5, ~ same depth
> SWA with 5:1 and 1024 (very small and cool ablation on the paper!)
> No MLA to save KV cache, SWA do the job!
2) Long context
> Only increase the rope in the global layer (to 1M)
> Confirmation that it's harder to do long context for smol models, no 128k for the 1B
> Pretrained with 32k context? seems very high
> No yarn nor llama3 like rope extension
3) Distillation
> Only keep te first 256 logits for the teacher
> Ablation on the teacher gap (tl;dr you need some "patience" to see that using a small teacher is better)
> On policy distillation yeahh (by
@agarwl_
et al), not sure if the teacher gap behave the same here, curious if someone have more info?
4) Others
> Checkpoint with QAT, that's very cool
> RL using improve version of BOND, WARM/WARP good excuse to look at
@ramealexandre
papers
> Only use Zero3, no TP/PP if i understand correctly ?
> Training budget relatively similar than gemma2
1) Architecture choices:
> No more softcaping, replace by QK-Norm
> Both Pre AND Post Norm
> Wider MLP than Qwen2.5, ~ same depth
> SWA with 5:1 and 1024 (very small and cool ablation on the paper!)
> No MLA to save KV cache, SWA do the job!
2) Long context
> Only increase the rope in the global layer (to 1M)
> Confirmation that it's harder to do long context for smol models, no 128k for the 1B
> Pretrained with 32k context? seems very high
> No yarn nor llama3 like rope extension
3) Distillation
> Only keep te first 256 logits for the teacher
> Ablation on the teacher gap (tl;dr you need some "patience" to see that using a small teacher is better)
> On policy distillation yeahh (by
@agarwl_
et al), not sure if the teacher gap behave the same here, curious if someone have more info?
4) Others
> Checkpoint with QAT, that's very cool
> RL using improve version of BOND, WARM/WARP good excuse to look at
@ramealexandre
papers
> Only use Zero3, no TP/PP if i understand correctly ?
> Training budget relatively similar than gemma2
- 1 reply
[Support] Community Articles
83
#5 opened about 1 year ago
by
victor
alvarobartt
posted
an
update
about 2 months ago
Post
3024
🔥 Agents can do anything!
@microsoft
Research just announced the release of Magma 8B!
Magma is a new Visual Language Model (VLM) with 8B parameters for multi-modal agents designed to handle complex interactions across virtual and real environments; and it's MIT licensed!
Magma comes with exciting new features such as:
- Introduces the Set-of-Mark and Trace-of-Mark techniques for fine-tuning
- Leverages a large amount of unlabeled video data to learn the spatial-temporal grounding and planning
- A strong generalization and ability to be fine-tuned for other agentic tasks
- SOTA in different multi-modal benchmarks spanning across UI navigation, robotics manipulation, image / video understanding and spatial understanding and reasoning
- Generates goal-driven visual plans and actions for agentic use cases
Model: microsoft/Magma-8B
Technical Report: Magma: A Foundation Model for Multimodal AI Agents (2502.13130)
Magma is a new Visual Language Model (VLM) with 8B parameters for multi-modal agents designed to handle complex interactions across virtual and real environments; and it's MIT licensed!
Magma comes with exciting new features such as:
- Introduces the Set-of-Mark and Trace-of-Mark techniques for fine-tuning
- Leverages a large amount of unlabeled video data to learn the spatial-temporal grounding and planning
- A strong generalization and ability to be fine-tuned for other agentic tasks
- SOTA in different multi-modal benchmarks spanning across UI navigation, robotics manipulation, image / video understanding and spatial understanding and reasoning
- Generates goal-driven visual plans and actions for agentic use cases
Model: microsoft/Magma-8B
Technical Report: Magma: A Foundation Model for Multimodal AI Agents (2502.13130)
suayptalha
posted
an
update
about 2 months ago
Post
1083
I've just distilled Llama-3.2-3B-Instruct with
deepseek-ai/DeepSeek-R1 on
ServiceNow-AI/R1-Distill-SFT dataset. 🐋🦙
Here is the model:
suayptalha/DeepSeek-R1-Distill-Llama-3B
Here is the model:
suayptalha/DeepSeek-R1-Distill-Llama-3B
DmitryRyumin
posted
an
update
about 2 months ago
Post
3785
🚀🎭🌟 New Research Alert - WACV 2025 (Avatars Collection)! 🌟🎭🚀
📄 Title: EmoVOCA: Speech-Driven Emotional 3D Talking Heads 🔝
📝 Description: EmoVOCA is a data-driven method for generating emotional 3D talking heads by combining speech-driven lip movements with expressive facial dynamics. This method has been developed to overcome the limitations of corpora and to achieve state-of-the-art animation quality.
👥 Authors: @FedeNoce , Claudio Ferrari, and Stefano Berretti
📅 Conference: WACV, 28 Feb – 4 Mar, 2025 | Arizona, USA 🇺🇸
📄 Paper: https://arxiv.org/abs/2403.12886
🌐 Github Page: https://fedenoce.github.io/emovoca/
📁 Repository: https://github.com/miccunifi/EmoVOCA
🚀 CVPR-2023-24-Papers: https://github.com/DmitryRyumin/CVPR-2023-24-Papers
🚀 WACV-2024-Papers: https://github.com/DmitryRyumin/WACV-2024-Papers
🚀 ICCV-2023-Papers: https://github.com/DmitryRyumin/ICCV-2023-Papers
📚 More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
🚀 Added to the Avatars Collection: DmitryRyumin/avatars-65df37cdf81fec13d4dbac36
🔍 Keywords: #EmoVOCA #3DAnimation #TalkingHeads #SpeechDriven #FacialExpressions #MachineLearning #ComputerVision #ComputerGraphics #DeepLearning #AI #WACV2024
📄 Title: EmoVOCA: Speech-Driven Emotional 3D Talking Heads 🔝
📝 Description: EmoVOCA is a data-driven method for generating emotional 3D talking heads by combining speech-driven lip movements with expressive facial dynamics. This method has been developed to overcome the limitations of corpora and to achieve state-of-the-art animation quality.
👥 Authors: @FedeNoce , Claudio Ferrari, and Stefano Berretti
📅 Conference: WACV, 28 Feb – 4 Mar, 2025 | Arizona, USA 🇺🇸
📄 Paper: https://arxiv.org/abs/2403.12886
🌐 Github Page: https://fedenoce.github.io/emovoca/
📁 Repository: https://github.com/miccunifi/EmoVOCA
🚀 CVPR-2023-24-Papers: https://github.com/DmitryRyumin/CVPR-2023-24-Papers
🚀 WACV-2024-Papers: https://github.com/DmitryRyumin/WACV-2024-Papers
🚀 ICCV-2023-Papers: https://github.com/DmitryRyumin/ICCV-2023-Papers
📚 More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
🚀 Added to the Avatars Collection: DmitryRyumin/avatars-65df37cdf81fec13d4dbac36
🔍 Keywords: #EmoVOCA #3DAnimation #TalkingHeads #SpeechDriven #FacialExpressions #MachineLearning #ComputerVision #ComputerGraphics #DeepLearning #AI #WACV2024
- 1 reply
yuchenlin
authored
a
paper
about 2 months ago
cxdu
authored
a
paper
2 months ago
eliebak
authored
a
paper
2 months ago
reach-vb
authored
a
paper
2 months ago
eliebak
authored
a
paper
2 months ago
gabrielmbmb
authored
a
paper
2 months ago
[Support] Community Articles
83
#5 opened about 1 year ago
by
victor
Post
5431
Hey everyone, we've given https://hf.co/spaces page a fresh update!
Smart Search: Now just type what you want to do—like "make a viral meme" or "generate music"—and our search gets it.
New Categories: Check out the cool new filter bar with icons to help you pick a category fast.
Redesigned Space Cards: Reworked a bit to really show off the app descriptions, so you know what each Space does at a glance.
Random Prompt: Need ideas? Hit the dice button for a burst of inspiration.
We’d love to hear what you think—drop us some feedback plz!
Smart Search: Now just type what you want to do—like "make a viral meme" or "generate music"—and our search gets it.
New Categories: Check out the cool new filter bar with icons to help you pick a category fast.
Redesigned Space Cards: Reworked a bit to really show off the app descriptions, so you know what each Space does at a glance.
Random Prompt: Need ideas? Hit the dice button for a burst of inspiration.
We’d love to hear what you think—drop us some feedback plz!
- 6 replies