Cisco Network Configuration Model (4-bit Quantized)

Usage with 4-bit Quantization

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16
)

model = AutoModelForCausalLM.from_pretrained(
    "Renugadevi82/cisco-nx-ai-4bit",
    quantization_config=bnb_config,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Renugadevi82/cisco-nx-ai-4bit")

Memory Requirements

4-bit: ~0.8GB VRAM
16-bit: ~2.5GB VRAM