CodeLLaMa7B-FineTuned-byMoomen

This model is a fine-tuned version of codellama/CodeLlama-7b-Instruct-hf using LoRA (Low-Rank Adaptation) for educational Q&A tasks.

Model Details

• Base Model: codellama/CodeLlama-7b-Instruct-hf
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• LoRA Rank: 32
• LoRA Alpha: 64
• Target Modules: ['gate_proj', 'lm_head', 'k_proj', 'q_proj', 'up_proj', 'down_proj', 'v_proj', 'o_proj']
• Training Focus: Educational programming Q&A
• Model Type: Causal Language Model

Usage

Quick Start

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

# Load model and tokenizer
model = AutoPeftModelForCausalLM.from_pretrained("Moomen123Msaadi/CodeLLaMa7B-FineTuned-byMoomen")
tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-Instruct-hf")

# Generate response
prompt = "Explain recursion in programming"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=300, temperature=0.7)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Chat Format Usage

# For educational Q&A conversations
messages = [
    {"role": "system", "content": "You are a helpful educational assistant."},
    {"role": "user", "content": "What is the difference between lists and tuples in Python?"}
]

formatted_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(formatted_prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=300)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

Memory-Efficient Loading

# For systems with limited VRAM
from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)

model = AutoPeftModelForCausalLM.from_pretrained(
    "Moomen123Msaadi/CodeLLaMa7B-FineTuned-byMoomen",
    quantization_config=quantization_config,
    device_map="auto"
)

Training Details

This model was fine-tuned using:

• Parameter-Efficient Fine-Tuning (PEFT) with LoRA
• Educational conversation dataset focused on programming concepts
• Optimized for Q&A format with system/user/assistant roles

Intended Use

This model is designed for:

• 📚 Educational programming Q&A
• 💡 Concept explanations in computer science
• 🔧 Code debugging assistance
• 🎓 Technical tutoring and learning support

Limitations

• Based on codellama/CodeLlama-7b-Instruct-hf, inherits its limitations
• Optimized for educational content, may not perform well on other tasks
• Requires base model for inference (LoRA adapters only)
• Performance depends on the quality of training data

Model Architecture

This is a LoRA adapter that needs to be loaded with the base model. The adapter files are:

• adapter_config.json: LoRA configuration
• adapter_model.safetensors: Trained LoRA weights

License

This model follows the same license as the base model: Llama 2 Custom License.

Citation

If you use this model, please cite:

@misc{CodeLLaMa7B_FineTuned_byMoomen,
  title={CodeLLaMa7B-FineTuned-byMoomen},
  author={Moomen123Msaadi},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/Moomen123Msaadi/CodeLLaMa7B-FineTuned-byMoomen}
}