Update README.md

README.md

@@ -2,39 +2,77 @@
 base_model: meta-llama/Llama-2-13b-chat-hf
 tags:
 - generated_from_trainer
+- trl
 metrics:
 - accuracy
 model-index:
 - name: llama-2-13b-reward-oasst1
   results: []
+datasets:
+- tasksource/oasst1_pairwise_rlhf_reward
+library_name: peft
+pipeline_tag: text-classification
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
 
 # llama-2-13b-reward-oasst1
 
-This model is a fine-tuned version of [meta-llama/Llama-2-13b-chat-hf](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) on the None dataset.
+This model is a fine-tuned version of [meta-llama/Llama-2-13b-chat-hf](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) on the [tasksource/oasst1_pairwise_rlhf_reward](https://huggingface.co/datasets/tasksource/oasst1_pairwise_rlhf_reward) dataset.
 It achieves the following results on the evaluation set:
 - Loss: 0.4810
 - Accuracy: 0.7869
 
 ## Model description
 
-More information needed
+This is a reward model trained with QLoRA in 4bit precision. The base model is [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) for which you need to have accepted the license in order to be able use it. Once you've been given permission, you can load the reward model as follows:
+```
+import torch
+from peft import PeftModel, PeftConfig
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+peft_model_id = "vincentmin/llama-2-7b-reward-oasst1"
+config = PeftConfig.from_pretrained(peft_model_id)
+model = AutoModelForSequenceClassification.from_pretrained(
+    config.base_model_name_or_path,
+    num_labels=1,
+    load_in_4bit=True,
+    torch_dtype=torch.float16,
+)
+model = PeftModel.from_pretrained(model, peft_model_id)
+tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path, use_auth_token=True)
+model.eval()
+with torch.no_grad():
+  reward = model(**tokenizer("prompter: hello world. assistant: foo bar", return_tensors='pt')).logits
+reward
+```
+For best results, one should use the prompt format used during training:
+```
+prompt = "prompter: <prompt_1> assistant: <response_1> prompter: <prompt_2> ..."
+```
 
 ## Intended uses & limitations
 
-More information needed
+Since the model was trained on oasst1 data, the reward will reflect any biases present in the oasst1 data.
 
 ## Training and evaluation data
 
-More information needed
+The model was trained using QLoRA and the `trl` library's `RewardTrainer` on the [tasksource/oasst1_pairwise_rlhf_reward](https://huggingface.co/datasets/tasksource/oasst1_pairwise_rlhf_reward) dataset where examples with more than 512 tokens were filtered out from both the training and eval data.
 
 ## Training procedure
 
 ### Training hyperparameters
 
+The following `bitsandbytes` quantization config was used during training:
+- load_in_8bit: False
+- load_in_4bit: True
+- llm_int8_threshold: 6.0
+- llm_int8_skip_modules: None
+- llm_int8_enable_fp32_cpu_offload: False
+- llm_int8_has_fp16_weight: False
+- bnb_4bit_quant_type: nf4
+- bnb_4bit_use_double_quant: False
+- bnb_4bit_compute_dtype: float16
+
 The following hyperparameters were used during training:
 - learning_rate: 2e-05
 - train_batch_size: 1
@@ -45,6 +83,7 @@ The following hyperparameters were used during training:
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: linear
 - num_epochs: 1
+- max_seq_length: 512
 
 ### Training results
 
@@ -66,7 +105,9 @@ The following hyperparameters were used during training:
 
 ### Framework versions
 
+
+- PEFT 0.5.0.dev0 (with https://github.com/huggingface/peft/pull/755)
 - Transformers 4.32.0.dev0
 - Pytorch 2.0.1+cu118
 - Datasets 2.14.0
-- Tokenizers 0.13.3
+- Tokenizers 0.13.3