README.md

---
license: other
tags:
- generated_from_trainer
model-index:
- name: OPT-Alpaca-125M
  results: []
widget:
- text: |-
    ### Instruction:
    Write a poem about pancakes.

    ### Response:
datasets:
- tatsu-lab/alpaca
---


# OPT-Alpaca-125M

This model is a fine-tuned version of [facebook/opt-125m](https://huggingface.co/facebook/opt-125m) on the [tatsu-lab/alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca) dataset. This is a test to check if the concept of Databrick's [databricks/dolly-v1-6b](https://huggingface.co/databricks/dolly-v1-6b) can be applied to smaller models.

## Model description

This is an LLM trained to follow instructions, similar to ChatGPT, except much much smaller.

## Sample Use

```python
from transformers import pipeline

generator = pipeline('text-generation', model='linkanjarad/OPT-Alpaca-125M')

prompt = '''### Instruction:
What is the capital of Australia?

### Response:'''

output = generator(prompt, do_sample=True, min_length=10, max_length=50, temperature=0.5)
print(output[0]['generated_text'])
```

## Intended uses & limitations

The model works best when given input in the format of the dataset it was trained on. Here is an example of the format of the [tatsu-lab/alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca) dataset:

```
### Instruction:
What are the three primary colors?

### Response:
The three primary colors are red, blue, and yellow.
```

Here are a few sample model outputs (`temperature=0.7`):

```
### Instruction:
What is the capital of Australia?

### Response:
The capital of Australia is Canberra.
```

```
### Instruction:
How do rainbows form?

### Response:
Rainbows form when the light of a sun enters the earth's atmosphere. The sun's rays are absorbed by the atmosphere, creating rainbows. The sun's light is then used to create a rainbow, which is a result of the Earth's rotation.
```

```
### Instruction:
Write me a poem about pancakes in the style of Dr. Seuss.

### Response:
Pancakes are like a wonderful miracle. They give us a beautiful and delicious breakfast.
Pancakes go from day to day,
But they also bring us pleasure.

I'm so glad I made the time to make my own
Pancakes are a delight for me and my family.
I'm going to make a special treat for you
```

```
### Instruction:
If you put an egg on the tip of a pencil, what will happen?

### Response:
An egg will start to move around the pencil without a trace.
```

```
### Instruction:
Write me code in Python that prints out the first ten numbers in the Fibonacci sequence.

### Response:
def number_primes(fibonacci):
    for i in range(10):
       for j in range(10):
          print(i)
    return i
```

```
### Instruction:
Write an article about card games with SEO.

### Response:
Card games are a popular game that allow players to play the game of cards with friends. The game is a puzzle game with players finding the correct cards that will be used to win a game. Players can use a variety of strategies to reach the right goal and win the game. Card games provide a lot of fun, and are a great way of providing a fun and engaging experience.
```

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 64
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.15
- num_epochs: 3

### Framework versions

- Transformers 4.27.4
- Pytorch 1.13.0
- Datasets 2.1.0
- Tokenizers 0.13.2