Hugging Face's logo Hugging Face
  • Models
  • Datasets
  • Spaces
  • Docs
  • Enterprise
  • Pricing

  • Log In
  • Sign Up
ReactiveAI 's Collections
RxT-Alpha-Mini
RxT-Alpha Micro
Interaction SFT Datasets
Sparse Query Attention (SQA) Research

Sparse Query Attention (SQA) Research

updated 22 days ago

Experimental models with Sparse Query Attention layers. Reducing training time/cost by ~3-10% compared to GQA & MQA, with the same level performance

Upvote
1

  • ReactiveAI/sSQAT-mm

    Text Generation • Updated about 1 month ago

  • ReactiveAI/SQAT-mm

    Text Generation • Updated about 1 month ago

  • ReactiveAI/xSQAT-mm

    Text Generation • Updated about 1 month ago

  • ReactiveAI/SQAT-m

    Text Generation • Updated May 1

  • ReactiveAI/sSQAT-m

    Text Generation • Updated May 1

  • ReactiveAI/xSQAT-m

    Text Generation • Updated May 1

  • ReactiveAI/xSMQAT-m

    Text Generation • Updated May 1

  • ReactiveAI/GQA-Ref-Micro

    Text Generation • Updated about 1 month ago

  • ReactiveAI/MQA-Ref-Micro

    Text Generation • Updated about 1 month ago
Upvote
1
  • Collection guide
  • Browse collections
Company
TOS Privacy About Jobs
Website
Models Datasets Spaces Pricing Docs