distilgpt2-finetuned-finance
This model is a fine-tuned version of distilgpt2 on the the combination of 4 different finance datasets:
- causal-lm/finance
- gbharti/finance-alpaca
- PaulAdversarial/all_news_finance_sm_1h2023
- winddude/reddit_finance_43_250k
Training and evaluation data
One can reproduce the dataset using the following code:
# load dataset
dataset_1 = load_dataset("gbharti/finance-alpaca")
dataset_2 = load_dataset("PaulAdversarial/all_news_finance_sm_1h2023")
dataset_3 = load_dataset("winddude/reddit_finance_43_250k")
dataset_4 = load_dataset("causal-lm/finance")
# create a column called text
dataset_1 = dataset_1.map(
lambda example: {"text": example["instruction"] + " " + example["output"]},
num_proc=4,
)
dataset_1 = dataset_1.remove_columns(["input", "instruction", "output"])
dataset_2 = dataset_2.map(
lambda example: {"text": example["title"] + " " + example["description"]},
num_proc=4,
)
dataset_2 = dataset_2.remove_columns(
["_id", "main_domain", "title", "description", "created_at"]
)
dataset_3 = dataset_3.map(
lambda example: {
"text": example["title"] + " " + example["selftext"] + " " + example["body"]
},
num_proc=4,
)
dataset_3 = dataset_3.remove_columns(
[
"id",
"title",
"selftext",
"z_score",
"normalized_score",
"subreddit",
"body",
"comment_normalized_score",
"combined_score",
]
)
dataset_4 = dataset_4.map(
lambda example: {"text": example["instruction"] + " " + example["output"]},
num_proc=4,
)
dataset_4 = dataset_4.remove_columns(["input", "instruction", "output"])
# combine and split train test sets
combined_dataset = concatenate_datasets(
[
dataset_1["train"],
dataset_2["train"],
dataset_3["train"],
dataset_4["train"],
dataset_4["validation"],
]
)
datasets = combined_dataset.train_test_split(test_size=0.2)
Inference example
from transformers import pipeline
generator = pipeline(model="lxyuan/distilgpt2-finetuned-finance")
generator("Tesla is",
pad_token_id=generator.tokenizer.eos_token_id,
max_new_tokens=200,
num_return_sequences=2
)
>>>
{'generated_text':
'Tesla is likely going to have a "market crash" over 20 years - I believe I\'m just not
sure how this is going to affect the world. \n\nHowever, I would like to see this play out
as a global financial crisis. With US interest rates already high, a crash in global real
estate prices means that people are likely to feel pressure on assets that are less well
served by the assets the US government gives them. \n\nWould these things help you in your
retirement? I\'m fairly new to Wall Street, and it makes me think that you should have a
bit more control over your assets (I’m not super involved in stock picking, but I’ve heard
many times that governments can help their citizens), right? As another commenter has put
it: there\'s something called a market crash that could occur in the second world country
for most markets (I don\'t know how that would fit under US laws if I had done all of the
above. \n\n'
},
{'generated_text':
"Tesla is on track to go from 1.46 to 1.79 per cent growth in Q3 (the fastest pace so far
in the US), which will push down the share price.\n\nWhile the dividend could benefit Amazon’s
growth, earnings also aren’t expected to be high at all, the company's annual earnings could
be an indication that investors have a strong plan to boost sales by the end of the year if
earnings season continues.\n\nThe latest financials showed earnings as of the end of July,
followed by the earnings guidance from analysts at the Canadian Real Estate Association, which
showed that Amazon’s revenues were up over $1.8 Trillion, which is a far cry from what was
expected in early Q1.\n\nAmazon has grown the share price by as much as 1.6 percent since June
2020. Analysts had predicted that earnings growth in the stock would drop to 0.36 per cent for
2020, which would lead to Amazon’"
}
Training procedure
Notebook link: here
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- gradient_accumulation_steps: 64
- total_train_batch_size: 256
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 50
Framework versions
- Transformers 4.30.2
- Pytorch 2.0.1+cu117
- Datasets 2.13.1
- Tokenizers 0.13.3
- Downloads last month
- 817
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.