Base Model for TransMLA
mengfanxu
fxmeng
AI & ML interests
None yet
Recent Activity
updated
a dataset
20 days ago
fxmeng/transmla_pretrain_100m_tokens
updated
a dataset
20 days ago
fxmeng/transmla_pretrain_1B_tokens
updated
a dataset
20 days ago
fxmeng/transmla_pretrain_6B_tokens
Organizations
None yet
Principal Singular Values and Singular Vectors Adaptation
Principal Singular Values and Singular Vectors Adaptation
Substructure of mistralai/Mixtral-8x7B-Instruct-v0.1
Base Model for TransMLA
Principal Singular Values and Singular Vectors Adaptation
Principal Singular Values and Singular Vectors Adaptation
Principal Singular Values and Singular Vectors Adaptation
Principal Singular Values and Singular Vectors Adaptation
https://arxiv.org/abs/2404.02948
Substructure of mistralai/Mixtral-8x7B-Instruct-v0.1