Safetensors
English
gated_deltanet

Model of the paper MoM: Linear Sequence Modeling with Mixture-of-Memories and Gated Delta Networks: Improving Mamba2 with Delta Rule.

The model was trained on a sample of SlimPajama with 100B tokens.

Downloads last month
45
Safetensors
Model size
1.21B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train linear-moe-hub/Gated-Deltanet-1.3B

Collection including linear-moe-hub/Gated-Deltanet-1.3B