0xadityam/llama-3.2-8b-indas-lora

Llama-3.2-11B-Vision-Instruct LoRA Fine-tuned on Indian Accounting Standards (Ind AS)

This model is a LoRA (Low-Rank Adaptation) fine-tuned version of meta-llama/Llama-3.2-11B-Vision-Instruct, specifically trained on Indian Accounting Standards (Ind AS) data.

Model Details

• Base Model: meta-llama/Llama-3.2-11B-Vision-Instruct (latest Meta vision model)
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Training Data: 269 high-quality examples covering Indian Accounting Standards
• Domain: Finance and Accounting (India-specific)
• Language: English
• Model Size: 11B parameters

Key Improvements over 8B Model

• Enhanced Reasoning: Better performance on complex accounting scenarios
• Vision Capabilities: Built-in vision understanding (though trained on text data)
• Improved Instruction Following: Better adherence to financial reporting guidelines
• Larger Context Window: Can handle longer financial documents

Training Details

• Training Epochs: 2 (reduced for larger model)
• LoRA Rank: 32 (optimized for 11B model)
• LoRA Alpha: 16
• Learning Rate: 1e-4 (reduced for stability)
• Batch Size: 8 (effective, 1 per device × 8 accumulation)
• Max Sequence Length: 1024 (optimized for memory)
• Quantization: 4-bit (nf4)

Use Cases

This model is designed to answer questions about:

• Indian Accounting Standards (Ind AS)
• Financial statement presentation
• Balance sheet preparation
• Profit and loss statements
• Cash flow statements
• Accounting policies and disclosures
• Going concern assessments
• Asset and liability classification
• Complex financial scenarios requiring deeper reasoning

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model (requires Meta approval)
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-11B-Vision-Instruct",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-11B-Vision-Instruct")

# Load LoRA weights
model = PeftModel.from_pretrained(base_model, "your-username/llama-3.2-11b-vision-indas-lora")

# Use the model for inference
prompt = "Define the objective of Ind AS 1."
inputs = tokenizer.encode(prompt, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=512)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

Base Model Notes

• Uses official meta-llama/Llama-3.2-11B-Vision-Instruct
• Latest generation with vision capabilities and improved reasoning
• Better performance on complex financial analysis tasks
• Requires Meta approval but usually granted quickly
• Significantly enhanced capabilities compared to previous versions

Memory Requirements

• Minimum VRAM: 16GB (with 4-bit quantization)
• Recommended VRAM: 24GB+ for optimal performance
• Training: Requires high-memory GPU (A100 40GB recommended)
• Inference: Can run on RTX 4090 24GB with quantization

Limitations

• This model is specifically trained on Indian Accounting Standards and may not perform well on other accounting standards (IFRS, US GAAP, etc.)
• The training data focuses on Ind AS 1 and related standards
• Responses should be verified with official accounting standards documents
• Not suitable for providing legal or investment advice
• Vision capabilities are inherited from base model but not specifically trained

Training Infrastructure

• Platform: Google Colab Pro+ / High-memory GPU environment
• GPU: A100 40GB (recommended for 11B model)
• Memory Optimization: 4-bit quantization with LoRA
• Training Time: ~2-3x longer than 8B model

Performance Notes

• Better reasoning on complex accounting scenarios compared to 8B model
• Higher quality responses for detailed financial questions
• Improved consistency in technical terminology
• Better understanding of regulatory context

Citation

If you use this model, please cite:

@misc{llama3.2-11b-vision-indas-lora,
  title={Llama-3.2-11B-Vision-Instruct LoRA Fine-tuned on Indian Accounting Standards},
  author={Your Name},
  year={2024},
  url={https://huggingface.co/your-username/llama-3.2-11b-vision-indas-lora}
}