Hopefully something open source. They have a long history of open sourcing stuff in the past. Hopefully they open source Gato. That would be a legendary model to be released in the public. Maybe hugginface should train their own Gato? I could contribute to code and data... oss community is awesome.
Hrishbh Dalal
HriDal
AI & ML interests
Large Language Models, Vision Language Models, Synthetic Data Generation and Reinforcement Learning
Organizations
posted
an
update
about 1 year ago
Post
776
I just saw that openai is using an updated tokenizer and it greatly increases the speed of the model and maybe even performance as if we increase the size of vocabulary, it could predict a single token which might be equivalent to two or three tokens in current tokenizers with 50-60k tokens or even 100k. I was thinking of scaling this to a million vocabulary size and then pre training llama3 8b using lora. I know that the model might go to shit, but we can increase the speed of the tokens generation greatly imo. And as one of meta papers said that predicting multiple tokens at the same time can actually increase the performance of a model, so I can imagine increasing the vocabulary in this way means multiple token generation in a way. Yann Lecunn also says that we don’t think in tokens but more like representations or abstractions of situations or problem to be solved. Can scaling to a million vocab size or even 10 million vocab size lead to better and more robust models? Please give me your thoughts on what can go wrong, what can go right etc…
- 1 reply