license: cc-by-nc-4.0
Model Details
RefuelLLM-2-small, aka Llama-3-Refueled, is a Llama3-8B base model further instruction tuned on a corpus of 2750+ datasets, spanning tasks such as classification, reading comprehension, structured attribute extraction and entity resolution. We're excited to open-source the model for the community to build on top of.
- More details about RefuelLLM-2 family of models
- You can also try out the models in Refuel's LLM playground
Model developers Refuel AI
Input Models input text only.
Output Models generate text only.
Model Architecture RefuelLLM-2-small is built on top of Llama-3-8B-instruct which is an auto-regressive language model that uses an optimized transformer architecture.
Model Release Date May 8, 2024.
How to use
This repository contains weights for RefuelLLM-2-small that are compatible for use with HuggingFace.
Use with transformers
See the snippet below for usage with Transformers:
>>> import torch
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> model_id = "refuelai/Llama-3-Refueled"
>>> tokenizer = AutoTokenizer.from_pretrained(model_id)
>>> model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
>>> messages = [{"role": "user", "content": "Is this comment toxic or non-toxic: RefuelLLM is the new way to label text data!"}]
>>> inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to("cuda")
>>> outputs = model.generate(inputs, max_new_tokens=20)
>>> print(tokenizer.decode(outputs[0]))
Training Data
RefuelLLM-2 and RefuelLLM-2-small were both trained on over 4 Billion tokens, spanning 2750+ NLP tasks. Our training collection consists majorly of:
- Human annotated datasets like Flan, Task Source, and the Aya collection
- Synthetic datasets like OpenOrca, OpenHermes and WizardLM
- Proprietary datasets developed or licensed by Refuel
Benchmarks
In this section, we report the results for Refuel models on our benchmark of labeling tasks. For details on the methodology see here.
| Provider | Model | LLM Output Quality (by task type) | |||||
|---|---|---|---|---|---|---|---|
| Overall | Classification | Reading Comprehension | Structure Extraction | Entity Matching | |||
| Refuel | RefuelLLM-2 | 83.82% | 84.94% | 76.03% | 88.16% | 92.00% | |
| OpenAI | GPT-4-Turbo | 80.88% | 81.77% | 72.08% | 84.79% | 97.20% | |
| Refuel | RefuelLLM-2-small | 79.67% | 81.72% | 70.04% | 84.28% | 92.00% | |
| Anthropic | Claude-3-Opus | 79.19% | 82.49% | 67.30% | 88.25% | 94.96% | |
| Meta | Llama3-70B-Instruct | 78.20% | 79.38% | 66.03% | 85.96% | 94.13% | |
| Gemini-1.5-Pro | 74.59% | 73.52% | 60.67% | 84.27% | 98.48% | ||
| Mistral | Mixtral-8x7B-Instruct | 62.87% | 79.11% | 45.56% | 47.08% | 86.52% | |
| Anthropic | Claude-3-Sonnet | 70.99% | 79.91% | 45.44% | 78.10% | 96.34% | |
| Anthropic | Claude-3-Haiku | 69.23% | 77.27% | 50.19% | 84.97% | 54.08% | |
| OpenAI | GPT-3.5-Turbo | 68.13% | 74.39% | 53.21% | 69.40% | 80.41% | |
| Meta | Llama3-8B-Instruct | 62.30% | 68.52% | 49.16% | 65.09% | 63.61% | |
Limitations
The RefuelLLM-v2-small does not have any moderation mechanisms. We're looking forward to engaging with the community on ways to make the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs.