🧪 MegaSciMoE‑1.2B

A lightweight open MoE language model focused on science reasoning and instruction-following tasks. Fine-tuned using Unsloth on top of LiquidAI’s LFM2‑1.2B architecture.

🔬 Model Overview

Model Type: Sparse Mixture-of-Experts (MoE)
Base: LiquidAI LFM2‑1.2B
Fine-tuned with: Unsloth LoRA
Focus Areas: Physics, Chemistry, Biology, Scientific QA
Parameter Efficiency: Only 2 experts active (~250M active params)
Design Goal: Run efficiently on consumer GPUs and edge devices

🧠 Capabilities

Respond to structured science-related instructions
Generate concise, factual explanations in STEM topics
Follow multiturn question-answer patterns
Summarize technical descriptions in plain language

🚀 Example Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained("yasserrmd/MegaSciMoE-1.2B", torch_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained("yasserrmd/MegaSciMoE-1.2B")

prompt = "What is the difference between an ionic and covalent bond?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

📚 Training Data

MegaSciMoE-1.2B was fine-tuned on a blend of:

Multiturn scientific dialogues
Instruction-based question answering
Simulated domain-specific prompts in physics, chemistry, and biology
Scientific corpus extracted from open datasets

📎 Sample Prompts

Q: What is a neutron made of?
A: A neutron is composed of three quarks (two down quarks and one up quark)...

---

Q: Define osmosis in simple terms.
A: Osmosis is the movement of water across a membrane...

---

Q: What is the role of mitochondria?
A: Mitochondria are the powerhouses of the cell...

🧪 Benchmark Intent

Planned evaluations on:

ARC-Challenge
PubMedQA
ScienceQA (Note: No math-based datasets included in this build)

⚙️ Technical Details

LoRA fine-tuning with Unsloth on 1.2B MoE model
Memory-efficient inference (fits on T4, L4, RTX 3060, even CPU)
GGUF and FP16 variants to be released

📄 License

Apache 2.0 — free to use and modify with attribution. Ensure upstream license compatibility before commercial use.

🤝 Acknowledgements

Thanks to Unsloth, LiquidAI, and Hugging Face for enabling this build
Visuals rendered with scientific prompts and surreal-futuristic style

yasserrmd
/

MegaSciMoE-1.2B