metadata

datasets:
  - PKU-Alignment/PKU-SafeRLHF
language:
  - en
tags:
  - reinforcement-learning-from-human-feedback
  - reinforcement-learning
  - beaver
  - safety
  - llama
  - ai-safety
  - deepspeed
  - rlhf
  - alpaca
library_name: safe-rlhf

🦫 Beaver's Reward Model

Model Details

The Beaver reward model is a preference model trained using the PKU-SafeRLHF dataset. It can play a role in the safe RLHF algorithm, helping the Beaver model become more helpful.

Developed by: the PKU-Alignment Team.
Model Type: An auto-regressive language model based on the transformer architecture.
License: Non-commercial license.
Fine-tuned from model: LLaMA, Alpaca.

Model Sources

Repository: https://github.com/PKU-Alignment/safe-rlhf
Beaver: https://huggingface.co/PKU-Alignment/beaver-7b-v3.0
Dataset: https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF
Reward Model: https://huggingface.co/PKU-Alignment/beaver-7b-v3.0-reward
Cost Model: https://huggingface.co/PKU-Alignment/beaver-7b-v3.0-cost
Dataset Paper: https://arxiv.org/abs/2307.04657
Paper: https://arxiv.org/abs/2310.12773

How to Use the Reward Model

import torch
from transformers import AutoTokenizer
from safe_rlhf.models import AutoModelForScore

model = AutoModelForScore.from_pretrained('PKU-Alignment/beaver-7b-v3.0-reward', torch_dtype=torch.bfloat16, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained('PKU-Alignment/beaver-7b-v3.0-reward')

input = 'BEGINNING OF CONVERSATION: USER: hello ASSISTANT:Hello! How can I help you today?'

input_ids = tokenizer(input, return_tensors='pt')
output = model(**input_ids)
print(output)

# ScoreModelOutput(
#     scores=tensor([[[-14.0000],
#          [ -2.6094],
#          [ -2.6562],
#          [ -2.0312],
#          [ -1.2188],
#          [ -1.6250],
#          [ -2.4688],
#          [ -2.7500],
#          [ -3.0000],
#          [ -6.0000],
#          [ -5.0625],
#          [ -7.0938],
#          [ -6.9688],
#          [ -4.3125],
#          [ -4.2188],
#          [ -3.7969],
#          [ -3.6875],
#          [ -3.3750],
#          [ -2.8906],
#          [ -3.9219],
#          [ -2.1406],
#          [ -1.7578],
#          [  0.4629],
#          [  2.1719],
#          [  4.4062],
#          [  7.1562],
#          [  7.7188],
#          [ 10.7500]]], grad_fn=<ToCopyBackward0>),
#     end_scores=tensor([[10.7500]], grad_fn=<ToCopyBackward0>),
#     last_hidden_state=tensor([[[ 0.4805, -0.4863, -0.9258,  ..., -0.0718,  0.8555,  0.6641],
#          [ 0.2021,  2.0156,  3.5156,  ..., -0.9844, -1.1484,  1.3203],
#          [ 1.0938,  1.4609,  1.7891,  ..., -3.2031, -0.8555, -1.2969],
#          ...,
#          [ 1.5859,  0.1348,  0.0322,  ..., -1.3672, -1.5234,  1.5156],
#          [ 0.9102,  0.6367, -0.8555,  ..., -1.2109, -0.6953,  1.5312],
#          [ 1.7188,  0.4434, -0.5586,  ..., -1.1484, -0.7461,  2.2031]]],
#        dtype=torch.bfloat16, grad_fn=<ToCopyBackward0>),
#     end_last_hidden_state=tensor([[ 1.7188,  0.4434, -0.5586,  ..., -1.1484, -0.7461,  2.2031]],
#        dtype=torch.bfloat16, grad_fn=<ToCopyBackward0>),
#     end_index=tensor([27])
# )