Muse Spark Open
Collection
Muse Spark Open model. A 816B parameter model with 42B Active parameters. BF16 and NVFP4 Weights. • 2 items • Updated • 169
👋 Open to Work
Van os
Vanos007
AI & ML interests
None yet
Recent Activity
upvoted a collection 8 days ago
Muse Spark Open liked a Space about 1 month ago
multimodalart/jev-decision-index liked a model about 1 month ago
convaiinnovations/layaOrganizations
upvoted a collection 8 days ago
liked a
Space about 1 month ago
liked a
model about 1 month ago
updated a
model 6 months ago
published a
model 6 months ago
liked 3
Spaces 6 months ago
liked a
model 6 months ago
liked a
model 7 months ago
updated a
model 7 months ago
published a
model 7 months ago
liked 2
models 8 months ago
liked a
Space about 1 year ago
published a
Space about 1 year ago
liked a
model about 1 year ago
liked a
model over 1 year ago
upvoted a collection over 1 year ago
reacted to lianghsun's post with 👍 over 1 year ago
Post
4,811
With the arrival of Twinkle April — Twinkle AI’s annual open-source celebration held every April — our community is excited to unveil its very first project:
📊 Twinkle Eval (https://github.com/ai-twinkle/Eval), a next-generation evaluation tool led by our contributor @tedslin .
Unlike traditional evaluation tools like iKala’s ievals (https://github.com/ikala-ai/ievals), which can only evaluate language models (LMs) one sample at a time, Twinkle Eval is designed with Large Reasoning Models (LRMs) in mind. As reasoning time increases with more complex models, traditional tools become increasingly inefficient 😲 — for example, evaluating LRMs on the ikala/tmmluplus benchmark could take *
half a day without finishing.
One question we were especially curious about:
Does shuffling multiple-choice answer order impact model accuracy? 🤔
→ See: "Change Answer Order Can Decrease MMLU Accuracy" – arXiv:2406.19470v1
To address these challenges, Twinkle Eval brings three key innovations to the table:
1️⃣ Parallelized evaluation of samples
2️⃣ Multi-round testing for stability
3️⃣ Randomized answer order to test robustness
After running experiments, we observed that Twinkle Eval can speed up evaluation by up to 15× 🚀🚀. Interestingly, most models scored slightly lower under the 2️⃣3️⃣ test settings compared to their claimed performance — suggesting further benchmarking is needed.
This framework also comes with additional tunable parameters and detailed logging of LM behavior per question — perfect for those who want to dive deeper. 😆
If you find Twinkle Eval useful, please ⭐ the project and help spread the word 🤗
- 5 replies