yuchenxie/arlowgpt-dummy-weights

DUMMY WEIGHTS ONLY !! NOT REAL MODEL !!

Specifications (Will be carried out to the main fully released model)

Architecture: arlow (Not supported by transformers right now, but beta version is out -> here)

Exact location: here

Will feature:

• GQA
• Silu
• Flash Attention VarLen + manual QKV proj
• cross attention (untrained, there for easy vision encoder incorporation)
• model is decoder only, however, cross attention weights is there.
• Custom RoPE
• and more!

Arlow architecture isn't officially supported by transformers but my implementation will be there if you want to try it.