|
|
--- |
|
|
datasets: |
|
|
- bigcode/the-stack |
|
|
- bigcode/the-stack-v2 |
|
|
- bigcode/starcoderdata |
|
|
- bigcode/commitpack |
|
|
- nvidia/OpenCodeReasoning |
|
|
library_name: transformers |
|
|
tags: |
|
|
- code |
|
|
license: mit |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# Spec Coder V1 |
|
|
**Spec Coder** is a cutting-edge, open-source AI model designed to assist with fundamental coding tasks. It is built on the **Llama architecture**, allowing seamless access via tools like **llama.cpp** and **Ollama**. This makes **Spec Coder** highly compatible with a variety of systems, enabling flexible deployment both locally and in the cloud. |
|
|
|
|
|
Trained on vast datasets, **Spec Coder** excels in generating code, completing code snippets, and understanding programming tasks across multiple languages. It can be used for code completion, debugging, and automated code generation, acting as a versatile assistant for developers. |
|
|
|
|
|
**Spec Coder** is optimized for integration into developer tools, providing intelligent coding assistance and facilitating research in programming languages. Its advanced transformer-based architecture, with 4 billion parameters, allows it to perform tasks across different environments efficiently. |
|
|
|
|
|
The model supports various downstream tasks including supervised fine-tuning (SFT) and reinforcement learning (RL) to improve its performance for specific programming tasks. |
|
|
|
|
|
# Training Data |
|
|
- Total Training Tokens: ~4.3 trillion tokens |
|
|
- Corpus: The Stack, StarCoder Training Dataset, The Stack v2, CommitPack, OpenCodeReasoning, English Wikipedia |
|
|
|
|
|
# Training Details |
|
|
- Context Window: 8,192 tokens |
|
|
- Optimization: Standard language modeling objective |
|
|
- Hardware: Cluster of 5 x RTX 4090 GPUs |
|
|
- Training Duration: ~140 days (approximately 6 months) |
|
|
|
|
|
# Benchmarks |
|
|
## RepoBench 1.1 (Python) |
|
|
| Model | 2k | 4k | 8k | 12k | 16k | Avg | Avg ≤ 8k | |
|
|
|--------------------|-------|-------|-------|-------|-------|-------|----------| |
|
|
| Spec-Coder-4b-V1 | 30.42%| 38.55%| 36.91%| 32.75%| 30.34%| 34.59%| 36.23% | |
|
|
|
|
|
## Syntax-Aware Fill-in-the-Middle (SAFIM) |
|
|
| Model | Algorithmic | Control | API | Average | |
|
|
|----------------------|-------------|---------|--------|---------| |
|
|
| Spec-Coder-4b-V1 | 38.22% | 41.18% | 60.45% | 46.28% | |
|
|
|
|
|
## HumanEval Infilling |
|
|
| Model | Single-Line | Multi-Line | Random Span | |
|
|
|----------------------|-------------|------------|-------------| |
|
|
| Spec-Coder-4b-V1 | 72.34% | 45.65% | 39.12% | |
|
|
|
|
|
# Limitations |
|
|
- **Biases**: The model may reflect biases present in the public codebases. |
|
|
- **Security**: Code generated by the model may contain security vulnerabilities. It is essential to verify and audit the code generated by the model for any potential risks. |
|
|
|
|
|
# Sample Usage |
|
|
Here are examples of how to run and interact with **Spec Coder**: |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
model_name = "SVECTOR-CORPORATION/Spec-Coder-4b-V1" |
|
|
model = AutoModelForCausalLM.from_pretrained(model_name) |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
|
|
|
input_code = "def factorial(n):\n if n == 0:" |
|
|
|
|
|
inputs = tokenizer(input_code, return_tensors="pt") |
|
|
|
|
|
outputs = model.generate(inputs['input_ids'], max_length=50, num_return_sequences=1) |
|
|
|
|
|
generated_code = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
|
|
|
print("Generated Python code:\n", generated_code) |
|
|
``` |
