Universal Assisted Generation: Faster Decoding with Any Assistant Model
โข
44
Joao Gante
joaogante
AI & ML interests
None yet
Articles
Organizations
joaogante's activity
reacted to
m-ric's
post with โค๏ธ
4 months ago
Post
3211
๐๐๐ฐ ๐๐๐๐จ๐๐ข๐ง๐ ๐ญ๐๐๐ก๐ง๐ข๐ช๐ฎ๐ ๐ข๐ง ๐ญ๐ซ๐๐ง๐ฌ๐๐จ๐ซ๐ฆ๐๐ซ๐ฌ ๐ฌ๐ข๐ ๐ง๐ข๐๐ข๐๐๐ง๐ญ๐ฅ๐ฒ ๐ซ๐๐๐ฎ๐๐๐ฌ ๐ก๐๐ฅ๐ฅ๐ฎ๐๐ข๐ง๐๐ญ๐ข๐จ๐ง๐ฌ ๐
DoLa decoding, which made a conference paper at ICLR '24, has just been merged in Transformers by @joaogante and Yung-Sung Chuang.
This new decoding method is simple yet extremely impressive!
Reminder: Decoder LLMs (the GPT kind of LLM, the most common one) generate their outputs one token at a time: at each step, given a current text, they compute a logit for each token in their vocabulary that should represent the probability of this token coming next.
Then they either pick the highest logit token (greedy decoding) or sample one with a probability defined by the logits (sampling).
The authors of DoLa wanted to improve that simple method.
They knew this established fact that transformer LMs encode low-level info (like base syntax) in early layers and more high-level info like knowledge in the later layers.
๐ก This gave them their key idea: During decoding, rather than picking the token with the highest logit, ๐๐ต๐ ๐ป๐ผ๐ ๐ฝ๐ถ๐ฐ๐ธ ๐๐ต๐ฒ ๐๐ผ๐ธ๐ฒ๐ป ๐๐ถ๐๐ต ๐๐ต๐ฒ ๐บ๐ผ๐๐ ๐ถ๐บ๐ฝ๐ฟ๐ฒ๐๐๐ถ๐๐ฒ ๐ถ๐ป๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ถ๐ป ๐น๐ผ๐ด๐ถ๐ ๐ฎ๐ฐ๐ฟ๐ผ๐๐ ๐น๐ฎ๐๐ฒ๐ฟ๐?
This gives impressive results:
๐ ๐ฑ% - ๐ฎ๐ฌ% ๐ฏ๐ฎ๐๐ฒ ๐ฝ๐ผ๐ถ๐ป๐๐ ๐ถ๐ป๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ฎ๐ฐ๐ฟ๐ผ๐๐ ๐๐ต๐ฒ ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ๐
๐ For instance on TruthfulQA / Open-ended, across all model sizes the increase in truthfulness is 14 base points, which is ๐ฎ๐ฟ๐ผ๐๐ป๐ฑ ๐ฐ๐ฌ% ๐ถ๐บ๐ฝ๐ฟ๐ผ๐๐ฒ๐บ๐ฒ๐ป๐ ๐ฐ๐ผ๐บ๐ฝ๐ฎ๐ฟ๐ฒ๐ฑ ๐๐ผ ๐๐๐ฎ๐ป๐ฑ๐ฎ๐ฟ๐ฑ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด!
๐ค Wouldn't decoding take longer because of this added contrasting step? ๐ ๐ง๐ต๐ฒ ๐ฟ๐๐ป๐๐ถ๐บ๐ฒ ๐ถ๐ป๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ถ๐ ๐ป๐ฒ๐ด๐น๐ถ๐ด๐ถ๐ฏ๐น๐ฒ, ๐ญ ๐๐ผ ๐ด% ๐ผ๐ป๐น๐.
Paper added to my collection ๐ m-ric/optimization-mechanics-661d543a5fc6ca1dc84284a0
DoLa decoding, which made a conference paper at ICLR '24, has just been merged in Transformers by @joaogante and Yung-Sung Chuang.
This new decoding method is simple yet extremely impressive!
Reminder: Decoder LLMs (the GPT kind of LLM, the most common one) generate their outputs one token at a time: at each step, given a current text, they compute a logit for each token in their vocabulary that should represent the probability of this token coming next.
Then they either pick the highest logit token (greedy decoding) or sample one with a probability defined by the logits (sampling).
The authors of DoLa wanted to improve that simple method.
They knew this established fact that transformer LMs encode low-level info (like base syntax) in early layers and more high-level info like knowledge in the later layers.
๐ก This gave them their key idea: During decoding, rather than picking the token with the highest logit, ๐๐ต๐ ๐ป๐ผ๐ ๐ฝ๐ถ๐ฐ๐ธ ๐๐ต๐ฒ ๐๐ผ๐ธ๐ฒ๐ป ๐๐ถ๐๐ต ๐๐ต๐ฒ ๐บ๐ผ๐๐ ๐ถ๐บ๐ฝ๐ฟ๐ฒ๐๐๐ถ๐๐ฒ ๐ถ๐ป๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ถ๐ป ๐น๐ผ๐ด๐ถ๐ ๐ฎ๐ฐ๐ฟ๐ผ๐๐ ๐น๐ฎ๐๐ฒ๐ฟ๐?
This gives impressive results:
๐ ๐ฑ% - ๐ฎ๐ฌ% ๐ฏ๐ฎ๐๐ฒ ๐ฝ๐ผ๐ถ๐ป๐๐ ๐ถ๐ป๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ฎ๐ฐ๐ฟ๐ผ๐๐ ๐๐ต๐ฒ ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ๐
๐ For instance on TruthfulQA / Open-ended, across all model sizes the increase in truthfulness is 14 base points, which is ๐ฎ๐ฟ๐ผ๐๐ป๐ฑ ๐ฐ๐ฌ% ๐ถ๐บ๐ฝ๐ฟ๐ผ๐๐ฒ๐บ๐ฒ๐ป๐ ๐ฐ๐ผ๐บ๐ฝ๐ฎ๐ฟ๐ฒ๐ฑ ๐๐ผ ๐๐๐ฎ๐ป๐ฑ๐ฎ๐ฟ๐ฑ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด!
๐ค Wouldn't decoding take longer because of this added contrasting step? ๐ ๐ง๐ต๐ฒ ๐ฟ๐๐ป๐๐ถ๐บ๐ฒ ๐ถ๐ป๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ถ๐ ๐ป๐ฒ๐ด๐น๐ถ๐ด๐ถ๐ฏ๐น๐ฒ, ๐ญ ๐๐ผ ๐ด% ๐ผ๐ป๐น๐.
Paper added to my collection ๐ m-ric/optimization-mechanics-661d543a5fc6ca1dc84284a0
- 2 replies
reacted to
alex-abb's
post with ๐ฅ
5 months ago
Post
4760
Hi everyone!
I'm Alex, I'm 16, I've been an internship at Hugging Face for a little over a week and I've already learned a lot about using and prompting LLM models. With @victor as tutor I've just finished a space that analyzes your feelings by prompting an LLM chat model. The aim is to extend it so that it can categorize hugging face posts.
alex-abb/LLM_Feeling_Analyzer
I'm Alex, I'm 16, I've been an internship at Hugging Face for a little over a week and I've already learned a lot about using and prompting LLM models. With @victor as tutor I've just finished a space that analyzes your feelings by prompting an LLM chat model. The aim is to extend it so that it can categorize hugging face posts.
alex-abb/LLM_Feeling_Analyzer
- 4 replies
posted
an
update
6 months ago
Post
2751
New sampling strategy dropped in ๐ค transformers -- Min P sampling ๐ฅ
Are you tired of having
Min P consists of a dynamic token filter -- as opposed to Top K, which keeps the K most likely tokens, and Top P, which keeps the most likely tokens up to a fixed cumulative probability, both static filters. Min P takes a base probability (defined in the
๐ High probability token present -> aggressive filter (we don't want to miss on that high-probability case and risk derailing generation)
๐ No high probability token present -> relaxed filter (there are many continuation possibilities that the model finds plausible)
You should set
Kudos to @kalomaze and @menhguin for creating this technique ๐ฅ Read their discussion in the original issue for benchmarks (https://github.com/huggingface/transformers/issues/27670)
Copy-pasteable version of the example in the image below here: https://pastebin.com/VqXNtuxd
Have fun experimenting! ๐
Are you tired of having
top_k arbitrarily discarding high-quality continuations? Or top_p forgetting to exclude low-probability tokens, derailing your generation? Try out the new min_p flag in generate, fresh from a PR merged today! ๐ฅฌMin P consists of a dynamic token filter -- as opposed to Top K, which keeps the K most likely tokens, and Top P, which keeps the most likely tokens up to a fixed cumulative probability, both static filters. Min P takes a base probability (defined in the
min_p flag) and multiplies it by the probability of the most likely token in the distribution for the next token. All tokens less likely than the resulting value are filtered. What happens with this strategy?๐ High probability token present -> aggressive filter (we don't want to miss on that high-probability case and risk derailing generation)
๐ No high probability token present -> relaxed filter (there are many continuation possibilities that the model finds plausible)
You should set
min_p to a low value, between 0.05 and 0.1. It behaves particularly well for creative text generation when paired up with temperature > 1.Kudos to @kalomaze and @menhguin for creating this technique ๐ฅ Read their discussion in the original issue for benchmarks (https://github.com/huggingface/transformers/issues/27670)
Copy-pasteable version of the example in the image below here: https://pastebin.com/VqXNtuxd
Have fun experimenting! ๐
posted
an
update
7 months ago
Post
2553
Adding a long prompt can help you fight LLM hallucinations. However, if you know exactly how you want your LLM output constrained, there are much better strategies! ๐ช
Did you know you can force your LLM to ALWAYS generate a valid JSON file? Or to follow a well-defined answer template? You can do that and more with the ๐ค transformers-compatible
It doesn't only allow you to master your LLM -- your text generation application will also become faster! ๐ฅ The more constrained your text generation is, the bigger speedups you'll see!
Follow @remi and other
Did you know you can force your LLM to ALWAYS generate a valid JSON file? Or to follow a well-defined answer template? You can do that and more with the ๐ค transformers-compatible
outlines library.It doesn't only allow you to master your LLM -- your text generation application will also become faster! ๐ฅ The more constrained your text generation is, the bigger speedups you'll see!
Follow @remi and other
outlines folks to stay on top of the constrained generation game ๐ง
reacted to
m-ric's
post with โค๏ธ๐ฅ
8 months ago
Post
1702
๐๐ผ๐ ๐ฑ๐ผ๐ฒ๐ ๐ฏ๐ฒ๐ฎ๐บ ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด ๐๐ผ๐ฟ๐ธ? โก๏ธ ๐๐๐ฌ ๐ซ๐๐จ๐ช๐๐ก๐๐ฏ๐๐ฉ๐๐ค๐ฃ ๐ฉ๐ค๐ค๐ก! ๐
In Decoder-type LLMs like GPT4 or Mistral-Large, the output is generated one token (=word part) at a time. That's why they're nicknamed "stochastic parrots": the "thinking" process only happens one step at a time, so it can seem really myopic.
๐๐จ ๐ก๐จ๐ฐ ๐ข๐ฌ ๐ญ๐ก๐ ๐ง๐๐ฑ๐ญ ๐ญ๐จ๐ค๐๐ง ๐ฌ๐๐ฅ๐๐๐ญ๐๐?
๐ Given its input sentence like "๐๐ฉ๐ข๐ต ๐ช๐ด ๐ต๐ฉ๐ฆ 7๐ต๐ฉ ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ? ๐๐ฉ๐ฆ 7๐ต๐ฉ ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ", the Decoder LLM generates, for each token in its vocabulary, a score that represents this token's probability of coming next.
For instance: "๐๐จ" gets score 0.56, and "๐๐๐ฃ" gets score 0.35.
๐ค ๐๐ซ๐๐๐๐ฒ ๐๐๐๐จ๐๐ข๐ง๐ is the naive option where you simply take the next most probable token at each step. But this creates paths that maximize very short-term rewards, thus may overlook better paths for the long term (like this time when you played FIFA all evening and arrived unprepared to your school exam on the next day).
In our example, the next highest score token might be "๐๐จ", but this will strongly bias the LLM towards giving an hasty response. On the opposite, starting with "๐๐๐ฃ" could have been completed with "๐ฃ๐ฆ ๐ฐ๐ฃ๐ต๐ข๐ช๐ฏ๐ฆ๐ฅ ๐ง๐ณ๐ฐ๐ฎ ๐ค๐ฐ๐ฎ๐ฑ๐ถ๐ต๐ช๐ฏ๐จ ๐ฑ๐ณ๐ฆ๐ท๐ช๐ฐ๐ถ๐ด ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ๐ด ๐ง๐ช๐ณ๐ด๐ต", which steers the LLM towards a correct reasoning!
๐บ๏ธ ๐๐๐๐ฆ ๐ฌ๐๐๐ซ๐๐ก improves on greedy decoding by generating at each step several paths - called beams - instead of one. This allows the generation to explore a much larger space, thus find better completions. In our example, both the "๐๐จ" and the "๐๐๐ฃ" completion could be tested. โ
๐ I've created a tool to let you visualize it, thank you @joaogante for your great help!
๐๐ง๐ฎ ๐๐ฉ ๐๐๐ง๐: m-ric/beam_search_visualizer
In Decoder-type LLMs like GPT4 or Mistral-Large, the output is generated one token (=word part) at a time. That's why they're nicknamed "stochastic parrots": the "thinking" process only happens one step at a time, so it can seem really myopic.
๐๐จ ๐ก๐จ๐ฐ ๐ข๐ฌ ๐ญ๐ก๐ ๐ง๐๐ฑ๐ญ ๐ญ๐จ๐ค๐๐ง ๐ฌ๐๐ฅ๐๐๐ญ๐๐?
๐ Given its input sentence like "๐๐ฉ๐ข๐ต ๐ช๐ด ๐ต๐ฉ๐ฆ 7๐ต๐ฉ ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ? ๐๐ฉ๐ฆ 7๐ต๐ฉ ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ", the Decoder LLM generates, for each token in its vocabulary, a score that represents this token's probability of coming next.
For instance: "๐๐จ" gets score 0.56, and "๐๐๐ฃ" gets score 0.35.
๐ค ๐๐ซ๐๐๐๐ฒ ๐๐๐๐จ๐๐ข๐ง๐ is the naive option where you simply take the next most probable token at each step. But this creates paths that maximize very short-term rewards, thus may overlook better paths for the long term (like this time when you played FIFA all evening and arrived unprepared to your school exam on the next day).
In our example, the next highest score token might be "๐๐จ", but this will strongly bias the LLM towards giving an hasty response. On the opposite, starting with "๐๐๐ฃ" could have been completed with "๐ฃ๐ฆ ๐ฐ๐ฃ๐ต๐ข๐ช๐ฏ๐ฆ๐ฅ ๐ง๐ณ๐ฐ๐ฎ ๐ค๐ฐ๐ฎ๐ฑ๐ถ๐ต๐ช๐ฏ๐จ ๐ฑ๐ณ๐ฆ๐ท๐ช๐ฐ๐ถ๐ด ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ๐ด ๐ง๐ช๐ณ๐ด๐ต", which steers the LLM towards a correct reasoning!
๐บ๏ธ ๐๐๐๐ฆ ๐ฌ๐๐๐ซ๐๐ก improves on greedy decoding by generating at each step several paths - called beams - instead of one. This allows the generation to explore a much larger space, thus find better completions. In our example, both the "๐๐จ" and the "๐๐๐ฃ" completion could be tested. โ
๐ I've created a tool to let you visualize it, thank you @joaogante for your great help!
๐๐ง๐ฎ ๐๐ฉ ๐๐๐ง๐: m-ric/beam_search_visualizer
reacted to
chiphuyen's
post with โค๏ธ๐
8 months ago
Post
Huggingface is carrying the AI open source ecosystem https://huyenchip.com/2024/03/14/ai-oss.html
- 4 replies
replied to
mayank-mishra's
post
8 months ago
In transformers the main blocker is backward compatibility -- we assume in many places that batched inputs come with fixed input length. Once we lift this requirement without breaking backward compatibility, it should be a nice addition! ๐
(Perhaps nested tensors will help)
reacted to
trisfromgoogle's
post with ๐คโค๏ธ
9 months ago
Post
I am thrilled to announce Gemma, new 2B and 7B models from Google, based on the same research and technology used to train the Gemini models! These models achieve state-of-the-art performance for their size, and are launched across Transformers, Google Cloud, and many other surfaces worldwide starting today.
Get started using and adapting Gemma in the model Collection: google/gemma-release-65d5efbccdbb8c4202ec078b
These launches are the product of an outstanding collaboration between the Google DeepMind and Hugging Face teams over the last few months -- very proud of the work both teams have done, from integration with Vertex AI to optimization across the stack. Read more about the partnership in the main launch by @philschmid @osanseviero @pcuenq on the launch blog: https://huggingface.co/blog/gemma
More information below if you are curious about training details, eval results, and safety characteristics!
Gemma Tech Report: https://goo.gle/GemmaReport
Launch announcement: https://blog.google/technology/developers/gemma-open-models/
Get started using and adapting Gemma in the model Collection: google/gemma-release-65d5efbccdbb8c4202ec078b
These launches are the product of an outstanding collaboration between the Google DeepMind and Hugging Face teams over the last few months -- very proud of the work both teams have done, from integration with Vertex AI to optimization across the stack. Read more about the partnership in the main launch by @philschmid @osanseviero @pcuenq on the launch blog: https://huggingface.co/blog/gemma
More information below if you are curious about training details, eval results, and safety characteristics!
Gemma Tech Report: https://goo.gle/GemmaReport
Launch announcement: https://blog.google/technology/developers/gemma-open-models/
-
- 6 replies
reacted to
JustinLin610's
post with โค๏ธ
9 months ago
Post
Yesterday we just released Qwen1.5. Maybe someday I can tell more about the experience. But this is is at least a good release even if it is not yet SOTA. There is not so many SOTA by the way. This time, we actually fixed a lot of problems.
1. Context lengths are finally unified for all sizes. Previously, a lot of users kept telling us that 14B only supports 2K (Yeah even dynamic NTK does not work that well and it can only be extended to around 4-5K. Let alone those know nothing about how to use dynamic NTK).
2. If you carefully use our base language models, you will find that they understand special tokens of ChatML, which means that you can directly use LoRA to train on data with ChatML format. Why you can't do this before? This is because if the base language model does not understand the special tokens, you need to make them trained, which means that you should turn on the training of embedding. This is disgusting and it often leads to problems when you use ZeRO3.
3. We did strengthen our base language models except for 72. You should find better base language models, especially for 7 and 14. Why not 72? Nah, hard to say, but will make it better.
4. About the multilingual capabilities. Yes we finally build up our multilingual evaluation system and find out that our new base language models have nice performance in multilingual evaluation for base language models. This tells us that we should pay more attention to the post-training with multilingual data. And we did that too. This is why this time we tell you something about multilingual performance. It is for sure much much better than our models before this release.
5. Chat models are the most promising stuff. Before this release, we gave you the SFT models. But this time, we had very nice SFT+DPO models. Yeah not only annotators like them but also users like them. I am sure you developers will feel that way too.
1. Context lengths are finally unified for all sizes. Previously, a lot of users kept telling us that 14B only supports 2K (Yeah even dynamic NTK does not work that well and it can only be extended to around 4-5K. Let alone those know nothing about how to use dynamic NTK).
2. If you carefully use our base language models, you will find that they understand special tokens of ChatML, which means that you can directly use LoRA to train on data with ChatML format. Why you can't do this before? This is because if the base language model does not understand the special tokens, you need to make them trained, which means that you should turn on the training of embedding. This is disgusting and it often leads to problems when you use ZeRO3.
3. We did strengthen our base language models except for 72. You should find better base language models, especially for 7 and 14. Why not 72? Nah, hard to say, but will make it better.
4. About the multilingual capabilities. Yes we finally build up our multilingual evaluation system and find out that our new base language models have nice performance in multilingual evaluation for base language models. This tells us that we should pay more attention to the post-training with multilingual data. And we did that too. This is why this time we tell you something about multilingual performance. It is for sure much much better than our models before this release.
5. Chat models are the most promising stuff. Before this release, we gave you the SFT models. But this time, we had very nice SFT+DPO models. Yeah not only annotators like them but also users like them. I am sure you developers will feel that way too.
- 5 replies
reacted to
clefourrier's
post with ๐ค
9 months ago
Post
๐ฅ New LLM leaderboard on the hub: NPHardEval!
It uses questions of logic, of different mathematical complexities, as a proxy for reasoning abilities. It notably removes questions relying on arithmetic, to really focus on logical abilities.
What's interesting imo is the potential to really study a model performance at different levels of complexity.
Bonus: Since the questions can be generated automatically, it's going to be dynamic, updated monthly! ๐
NPHardEval/NPHardEval-leaderboard
Read more about how their questions are generated in the intro blog: https://huggingface.co/blog/leaderboards-on-the-hub-nphardeval
Congrats to @lizhouf , @wenyueH , @hyfrankl and their teams!
It uses questions of logic, of different mathematical complexities, as a proxy for reasoning abilities. It notably removes questions relying on arithmetic, to really focus on logical abilities.
What's interesting imo is the potential to really study a model performance at different levels of complexity.
Bonus: Since the questions can be generated automatically, it's going to be dynamic, updated monthly! ๐
NPHardEval/NPHardEval-leaderboard
Read more about how their questions are generated in the intro blog: https://huggingface.co/blog/leaderboards-on-the-hub-nphardeval
Congrats to @lizhouf , @wenyueH , @hyfrankl and their teams!
reacted to
abidlabs's
post with โค๏ธ
10 months ago
Post
The next version of Gradio will be significantly more efficient (as well as a bit faster) for anyone who uses Gradio's streaming features. Looking at you chatbot developers
@oobabooga
@pseudotensor
:)
The major change that we're making is that when you stream data, Gradio used to send the entire payload at each token. This is generally the most robust way to ensure all the data is correctly transmitted. We've now switched to sending "diffs" --> so at each time step, we automatically compute the diff between the most recent updates and then only send the latest token (or whatever the diff may be). Coupled with the fact that we are now using SSE, which is a more robust communication protocol than WS (SSE will resend packets if there's any drops), we should have the best of both worlds: efficient *and* robust streaming.
Very cool stuff @aliabid94 ! PR: https://github.com/gradio-app/gradio/pull/7102
The major change that we're making is that when you stream data, Gradio used to send the entire payload at each token. This is generally the most robust way to ensure all the data is correctly transmitted. We've now switched to sending "diffs" --> so at each time step, we automatically compute the diff between the most recent updates and then only send the latest token (or whatever the diff may be). Coupled with the fact that we are now using SSE, which is a more robust communication protocol than WS (SSE will resend packets if there's any drops), we should have the best of both worlds: efficient *and* robust streaming.
Very cool stuff @aliabid94 ! PR: https://github.com/gradio-app/gradio/pull/7102
reacted to
clem's
post with โค๏ธ
10 months ago
Post
With the Google announcement last week, I think we're now officially the only AI startup out there who has commercial collaborations with all the major cloud providers (AWS, GCP, Azure) and hardware providers (Nvidia, AMD, Intel, Qualcomm,...), making our vision of being the independent and agnostic platform for all AI builders truer than ever!
Let's go!
Let's go!
replied to
their
post
10 months ago
@MaziyarPanahi
no accuracy penalty at all :) The only catch on the transformers side is that you are limited to a batch size of one (and even that is not a technical limitation of the technique -- we simply haven't built that code path yet)
posted
an
update
10 months ago
Post
Up to 3x faster LLM generation with no extra resources/requirements - ngram speculation has landed in ๐ค transformers! ๐๏ธ๐จ
All you need to do is to add
How does it work? ๐ค
Start with assisted generation, where a smaller model generates candidate sequences. The net result is a significant speedup if the model agrees with the candidate sequences! However, we do require a smaller model trained similarly ๐
The idea introduced (and implemented) by Apoorv Saxena consists of gathering the candidate sequences from the input text itself. If the latest generated ngram is in the input, use the continuation therein as a candidate! No smaller model is required while still achieving significant speedups ๐ฅ
In fact, the penalty of gathering and testing the candidates is so small that you should use this technique whenever possible!
Here is the code example that produces the outputs shown in the video: https://pastebin.com/bms6XtR4
Have fun ๐ค
All you need to do is to add
prompt_lookup_num_tokens=10 to your generate call, and you'll get faster LLMs ๐ฅHow does it work? ๐ค
Start with assisted generation, where a smaller model generates candidate sequences. The net result is a significant speedup if the model agrees with the candidate sequences! However, we do require a smaller model trained similarly ๐
The idea introduced (and implemented) by Apoorv Saxena consists of gathering the candidate sequences from the input text itself. If the latest generated ngram is in the input, use the continuation therein as a candidate! No smaller model is required while still achieving significant speedups ๐ฅ
In fact, the penalty of gathering and testing the candidates is so small that you should use this technique whenever possible!
Here is the code example that produces the outputs shown in the video: https://pastebin.com/bms6XtR4
Have fun ๐ค
-
- 3 replies