Hugging Face's logo
Mariusz Kurman's picture
19 3 20

Mariusz Kurman PRO

mkurman
maxrubin629's profile picture Mindweller's profile picture longevist's profile picture
·

AI & ML interests

AI Tech Lead | MD

Recent Activity

reacted to Kseniase's post with 🔥 6 days ago
15 types of attention mechanisms Attention mechanisms allow models to dynamically focus on specific parts of their input when performing tasks. In our recent article, we discussed Multi-Head Latent Attention (MLA) in detail and now it's time to summarize other existing types of attention. Here is a list of 15 types of attention mechanisms used in AI models: 1. Soft attention (Deterministic attention) -> https://huggingface.co/papers/1409.0473 Assigns a continuous weight distribution over all parts of the input. It produces a weighted sum of the input using attention weights that sum to 1. 2. Hard attention (Stochastic attention) -> https://huggingface.co/papers/1508.04025 Makes a discrete selection of some part of the input to focus on at each step, rather than attending to everything. 3. Self-attention -> https://huggingface.co/papers/1706.03762 Each element in the sequence "looks" at other elements and "decides" how much to borrow from each of them for its new representation. 4. Cross-Attention (Encoder-Decoder attention) -> https://huggingface.co/papers/2104.08771 The queries come from one sequence and the keys/values come from another sequence. It allows a model to combine information from two different sources. 5. Multi-Head Attention (MHA) -> https://huggingface.co/papers/1706.03762 Multiple attention “heads” are run in parallel.​ The model computes several attention distributions (heads), each with its own set of learned projections of queries, keys, and values. 6. Multi-Head Latent Attention (MLA) -> https://huggingface.co/papers/2405.04434 Extends MHA by incorporating a latent space where attention heads can dynamically learn different latent factors or representations. 7. Memory-Based attention -> https://huggingface.co/papers/1503.08895 Involves an external memory and uses attention to read from and write to this memory. See other types in the comments 👇
reacted to BrigitteTousi's post with ❤️ 11 days ago
LeRobot goes to driving school! 🚗🚗🚗 Hugging Face just announced a new collab with Yaak to bring the largest open-source self-driving dataset to LeRobot! Major kudos to HF's @cadene, as well as @sandhawalia, @Shnissen and the Yaak team! Check out the blog post here: https://huggingface.co/blog/lerobot-goes-to-driving-school
reacted to BrigitteTousi's post with 🔥 11 days ago
LeRobot goes to driving school! 🚗🚗🚗 Hugging Face just announced a new collab with Yaak to bring the largest open-source self-driving dataset to LeRobot! Major kudos to HF's @cadene, as well as @sandhawalia, @Shnissen and the Yaak team! Check out the blog post here: https://huggingface.co/blog/lerobot-goes-to-driving-school
View all activity

Organizations

MedIT Solutions's profile picture BigScience Biomedical Datasets's profile picture SOWA Project's profile picture

Posts 17

view post
Post
822
Just released NVAMP Loss!

✔️ modification of the cross-entropy loss function designed specifically for training LLMs.
✔️ twist on the standard cross-entropy loss by emphasizing the importance of outlier prediction errors and dynamically normalizing token-level variance.
✔️ more stable and efficient training, leading to models that generalize better.

Check it out, give it a spin, and let me know what you think!

Licensed under the Apache 2.0 license and ready to use. Happy training! 🔥🤖

https://github.com/mkurman/nvamp-loss
View all Posts