Neo-X1-3B-Preview 🚀

Model Description

Neo-X1-3B-Preview is a 3B parameter model trained from Qwen2.5-3B-Instruct. The model focuses on enhanced text generation, code understanding, and conversational capabilities.

Developed by: Open Neo Team

Model Details

• Base Model: Qwen/Qwen2.5-3B
• Parameters: 3.07B
• Context Length: Full 32,768 tokens and generation 8192 tokens
• Architecture: Transformer-based

Training Details

Training Data

Our model was fine-tuned on a carefully curated combination of high-quality datasets:

• BAAI/TACO - For technical and code understanding
• codeparrot/apps - For programming challenges
• rubenroy/GammaCorpus-chat-70k - For conversational abilities

Training Procedure

• Supervised fine-tuning approach
• Training framework: Transformers + PyTorch
• Training infrastructure: PyTorch with DeepSpeed optimization

Evaluation

Performance metrics on key benchmarks (to be updated):

• Code Generation
• Mathematical Reasoning
• General Text Generation
• Conversational Tasks

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "odyssey-labs/Neo-X1-3B-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Example usage
input_text = "Write a function to calculate fibonacci numbers"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=200)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

Limitations

• Preview release - may contain inconsistencies
• Performance variations across different domains
• Limited testing in production environments
• Model responses should be validated for critical applications

Acknowledgements

We want to thank:

• The Qwen team for the base model
• Lambda Labs for their powerful compute power
• Unsloth for an efficient fine-tuning framework

@misc{sky_t1_2025,
  author       = {Open-Neo},
  title        = {Neo-X1: Fully open-source model that provides performance with low-end compute},
  howpublished = {https://huggingface.co/open-neo},
  note         = {Accessed: 2025-01-015},
  year         = {2025}
}