LlaMa 2 13b 4-bit Chain of Thought Reasoning 👩‍💻

LlaMa-2 13b fine-tuned on the kaist-ai/CoT-Collection dataset by using the method QLoRA in 4-bit with PEFT library.

Pretrained description

Llama-2

Meta developed and publicly released the Llama 2 family of large language models (LLMs), a collection of pretrained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters.

Model Architecture Llama 2 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align to human preferences for helpfulness and safety

Training data

kaist-ai/CoT-Collection

Qunatization Configuration

The following bitsandbytes quantization config was used during training:

bits: 4
group_size: 128
dataset: "c4"
desc_act: False
tokenizer:tokeniaer
device_map: "auto"

Framework versions

PEFT 0.4.0

Training

Downloading (…)okenizer_config.json: 100%
725/725 [00:00<00:00, 118kB/s]
Downloading (…)/main/tokenizer.json: 100%
1.84M/1.84M [00:01<00:00, 1.45MB/s]
Downloading (…)cial_tokens_map.json: 100%
437/437 [00:00<00:00, 35.9kB/s]
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py:472: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
Downloading (…)lve/main/config.json: 100%
631/631 [00:00<00:00, 53.8kB/s]
Downloading (…)/adapter_config.json: 100%
452/452 [00:00<00:00, 39.1kB/s]
Downloading (…)lve/main/config.json: 100%
587/587 [00:00<00:00, 50.1kB/s]
Downloading (…)fetensors.index.json: 100%
33.4k/33.4k [00:00<00:00, 2.97MB/s]
Downloading shards: 100%
3/3 [28:42<00:00, 546.82s/it]
Downloading (…)of-00003.safetensors: 100%
9.95G/9.95G [10:35<00:00, 15.3MB/s]
Downloading (…)of-00003.safetensors: 100%
9.90G/9.90G [11:04<00:00, 15.9MB/s]
Downloading (…)of-00003.safetensors: 100%
6.18G/6.18G [06:56<00:00, 14.5MB/s]
Loading checkpoint shards: 100%
3/3 [00:03<00:00, 1.01s/it]
/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py:374: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
Downloading (…)neration_config.json: 100%
188/188 [00:00<00:00, 16.5kB/s]
Downloading readme: 100%
2.38k/2.38k [00:00<00:00, 162kB/s]
Repo card metadata block was not found. Setting CardData to empty.
Downloading data files: 100%
1/1 [00:42<00:00, 42.25s/it]
Downloading data: 100%
319M/319M [00:42<00:00, 6.57MB/s]
Extracting data files: 100%
1/1 [00:04<00:00, 4.02s/it]
Generating train split:
356317/0 [00:01<00:00, 224199.18 examples/s]
Quantizing model.layers blocks : 100%
40/40 [32:31<00:00, 50.21s/it]
CUDA extension not installed.
Downloading adapter_model.bin: 100%
26.3M/26.3M [00:03<00:00, 8.64MB/s]

Example of usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "aiplanet/effi-13b-int4-GPTQ"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
tst = """Read the Instruction below and provide an answer the question asked.Stick to to theinstruction .Do not repeat the answers.

### INSTRUCTION:
Virgin Australia, the trading name of Virgin Australia Airlines Pty Ltd, is an Australian-based airline. It is the largest airline by fleet size to use the Virgin brand. It commenced services on 31 August 2000 as Virgin Blue, with two aircraft on a single route. It suddenly found itself as a major airline in Australia's domestic market after the collapse of Ansett Australia in September 2001. The airline has since grown to directly serve 32 cities in Australia, from hubs in Brisbane, Melbourne and Sydney.Is Virgin Australia and Virgin Blue the same airlines?

"""
#
prompt = f"[INST] <<SYS>>\n{system_message}\n<</SYS>>\n\n{tst}. [/INST]"
#

# Tokenize the input
input_ids = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.cuda()
# Run the model to infere an output
outputs = model.generate(input_ids=input_ids, max_new_tokens=100, do_sample=True, top_p=0.9,temperature=0.1)

# Print the result
print(f"Prompt:\n{prompt}\n")
print(f"Generated instruction:\n{tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0][len(prompt):].split(' [/INST]')[0]}")

Citation

@misc {Plaban81,
    author       = { {Plaban Nayak} },
    title        = { Quantized version of effi-13b by AI Planet},
    year         = 2023,
    url          = { https://huggingface.co/aiplanet/effi-13b },
    publisher    = { Hugging Face }
}

aiplanet
/

effi-13b-quant