Model Card for nano-phi-115M-v0.1

Inspired by Phi2, and open source small language model attempts like smol_llama-101M-GQA.
Pre-trained with training 7B token from scratch, with application of quality filter to datasets resulting in 0.26B token.
The control is kenhktsui/nano-phi-115M-control-v0.1, where full dataset (0.6B) is used.
Not much degradation in performance despite only using 42% of the data due to the effective quality filter ("quality_score_v1" > 0.5). In fact, upon inspection, the 6000 steps chkpt achieves similar performance as this model, signaling underlying effective training due to high quality data. It just took 1d to train in Colab with a A100 40GB (<USD$ 50).
It achieves quite competitive results in evaluation given its training token, and training data size.
Yet, there are still large gaps (particularly in ARC, HellaSwag, MMLU and GSM8K) between nano-phi-115M-v0.1 and phi-2, where author will attempt to narrow down the gap in the future. No alignment has been done yet.

How to use

To use the model, you will need transformer version >= 4.37.2

pip install transformers>=4.37.2

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kenhktsui/nano-phi-115M-v0.1")
pipe("I am a machine learning researcher. I work on", max_new_tokens=50, repetition_penalty=10.0)
# [{'generated_text': 'I am a machine learning researcher. I work on the problem of finding patterns in data, and it is not easy to find them all at once!\nThe first step was searching for pattern matching algorithms that are used by many people who have never seen an algorithm before (or even if they do).'}]

Some metrics

• model
  • hidden_size: 768
  • num_key_value_heads: 8 (grouped query attention)
  • num_attention_heads: 24
  • num_hidden_layers: 6
  • context length: 1024
  • total params: 115M
• training:
  • global steps: 14,000

Open LLM Leaderboard Evaluation Results

Metric	kenhktsui/nano-phi-115M-v0.1	kenhktsui/nano-phi-115M-v0.1 (6000 steps)	kenhktsui/nano-phi-115M-control-v0.1	microsoft/phi-2
Model Para	115M	115M	115M	2.7B
Dataset Size	0.26B	0.26B	0.6B	250B
Training Token	7B	3B	7B	1.4T
Context Length	1024	1024	1024	2048
Device	1xA100-40G	1xA100-40G	1xA100-40G	96xA100-80G
Training Time	2d4h	1d	2d4h	14d

Metric	kenhktsui/nano-phi-115M-v0.1	kenhktsui/nano-phi-115M-v0.1 (6000 steps)	kenhktsui/nano-phi-115M-control-v0.1	microsoft/phi-2 (Reproduced)
Avg.	28.68	29.03	28.75	61.53
ARC (25-shot)	21.93	22.27	21.67	61.52
HellaSwag (10-shot)	27.87	26.88	26.89	75.13
MMLU (5-shot)	25.30	25.01	24.76	58.23
TruthfulQA (0-shot)	46.01	48.03	47.69	44.46
Winogrande (5-shot)	50.99	52.01	51.46	74.51
GSM8K (5-shot)	0.0	0.0	0.0	55.34

Details:

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/checkpoint-pegfss6f:v13,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: 16

Task	Version	Metric	Value		Stderr
arc_easy	0	acc	0.4263	±	0.0101
		acc_norm	0.3864	±	0.0100

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/checkpoint-pegfss6f:v13,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 25, batch_size: 16

Task	Version	Metric	Value		Stderr
arc_challenge	0	acc	0.1826	±	0.0113
		acc_norm	0.2193	±	0.0121

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/checkpoint-pegfss6f:v13,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 10, batch_size: 16

Task	Version	Metric	Value		Stderr
hellaswag	0	acc	0.2733	±	0.0044
		acc_norm	0.2787	±	0.0045

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/checkpoint-pegfss6f:v13,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: 16

Task	Version	Metric	Value		Stderr
truthfulqa_mc	1	mc1	0.2521	±	0.0152
		mc2	0.4601	±	0.0154

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/checkpoint-pegfss6f:v13,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 5, batch_size: 16

Task	Version	Metric	Value		Stderr
hendrycksTest-abstract_algebra	1	acc	0.2300	±	0.0423
		acc_norm	0.2300	±	0.0423
hendrycksTest-anatomy	1	acc	0.3111	±	0.0400
		acc_norm	0.3111	±	0.0400
hendrycksTest-astronomy	1	acc	0.2171	±	0.0336
		acc_norm	0.2171	±	0.0336
hendrycksTest-business_ethics	1	acc	0.2500	±	0.0435
		acc_norm	0.2500	±	0.0435
hendrycksTest-clinical_knowledge	1	acc	0.2226	±	0.0256
		acc_norm	0.2226	±	0.0256
hendrycksTest-college_biology	1	acc	0.2292	±	0.0351
		acc_norm	0.2292	±	0.0351
hendrycksTest-college_chemistry	1	acc	0.1700	±	0.0378
		acc_norm	0.1700	±	0.0378
hendrycksTest-college_computer_science	1	acc	0.2500	±	0.0435
		acc_norm	0.2500	±	0.0435
hendrycksTest-college_mathematics	1	acc	0.2500	±	0.0435
		acc_norm	0.2500	±	0.0435
hendrycksTest-college_medicine	1	acc	0.2023	±	0.0306
		acc_norm	0.2023	±	0.0306
hendrycksTest-college_physics	1	acc	0.3235	±	0.0466
		acc_norm	0.3235	±	0.0466
hendrycksTest-computer_security	1	acc	0.2600	±	0.0441
		acc_norm	0.2600	±	0.0441
hendrycksTest-conceptual_physics	1	acc	0.2511	±	0.0283
		acc_norm	0.2511	±	0.0283
hendrycksTest-econometrics	1	acc	0.2281	±	0.0395
		acc_norm	0.2281	±	0.0395
hendrycksTest-electrical_engineering	1	acc	0.2276	±	0.0349
		acc_norm	0.2276	±	0.0349
hendrycksTest-elementary_mathematics	1	acc	0.2460	±	0.0222
		acc_norm	0.2460	±	0.0222
hendrycksTest-formal_logic	1	acc	0.1508	±	0.0320
		acc_norm	0.1508	±	0.0320
hendrycksTest-global_facts	1	acc	0.3000	±	0.0461
		acc_norm	0.3000	±	0.0461
hendrycksTest-high_school_biology	1	acc	0.3387	±	0.0269
		acc_norm	0.3387	±	0.0269
hendrycksTest-high_school_chemistry	1	acc	0.2906	±	0.0319
		acc_norm	0.2906	±	0.0319
hendrycksTest-high_school_computer_science	1	acc	0.3100	±	0.0465
		acc_norm	0.3100	±	0.0465
hendrycksTest-high_school_european_history	1	acc	0.2182	±	0.0323
		acc_norm	0.2182	±	0.0323
hendrycksTest-high_school_geography	1	acc	0.3232	±	0.0333
		acc_norm	0.3232	±	0.0333
hendrycksTest-high_school_government_and_politics	1	acc	0.2021	±	0.0290
		acc_norm	0.2021	±	0.0290
hendrycksTest-high_school_macroeconomics	1	acc	0.2487	±	0.0219
		acc_norm	0.2487	±	0.0219
hendrycksTest-high_school_mathematics	1	acc	0.2741	±	0.0272
		acc_norm	0.2741	±	0.0272
hendrycksTest-high_school_microeconomics	1	acc	0.3319	±	0.0306
		acc_norm	0.3319	±	0.0306
hendrycksTest-high_school_physics	1	acc	0.3179	±	0.0380
		acc_norm	0.3179	±	0.0380
hendrycksTest-high_school_psychology	1	acc	0.2477	±	0.0185
		acc_norm	0.2477	±	0.0185
hendrycksTest-high_school_statistics	1	acc	0.4722	±	0.0340
		acc_norm	0.4722	±	0.0340
hendrycksTest-high_school_us_history	1	acc	0.2696	±	0.0311
		acc_norm	0.2696	±	0.0311
hendrycksTest-high_school_world_history	1	acc	0.2152	±	0.0268
		acc_norm	0.2152	±	0.0268
hendrycksTest-human_aging	1	acc	0.1973	±	0.0267
		acc_norm	0.1973	±	0.0267
hendrycksTest-human_sexuality	1	acc	0.2824	±	0.0395
		acc_norm	0.2824	±	0.0395
hendrycksTest-international_law	1	acc	0.2231	±	0.0380
		acc_norm	0.2231	±	0.0380
hendrycksTest-jurisprudence	1	acc	0.2222	±	0.0402
		acc_norm	0.2222	±	0.0402
hendrycksTest-logical_fallacies	1	acc	0.2822	±	0.0354
		acc_norm	0.2822	±	0.0354
hendrycksTest-machine_learning	1	acc	0.2768	±	0.0425
		acc_norm	0.2768	±	0.0425
hendrycksTest-management	1	acc	0.2039	±	0.0399
		acc_norm	0.2039	±	0.0399
hendrycksTest-marketing	1	acc	0.1966	±	0.0260
		acc_norm	0.1966	±	0.0260
hendrycksTest-medical_genetics	1	acc	0.2800	±	0.0451
		acc_norm	0.2800	±	0.0451
hendrycksTest-miscellaneous	1	acc	0.2746	±	0.0160
		acc_norm	0.2746	±	0.0160
hendrycksTest-moral_disputes	1	acc	0.2081	±	0.0219
		acc_norm	0.2081	±	0.0219
hendrycksTest-moral_scenarios	1	acc	0.2469	±	0.0144
		acc_norm	0.2469	±	0.0144
hendrycksTest-nutrition	1	acc	0.2647	±	0.0253
		acc_norm	0.2647	±	0.0253
hendrycksTest-philosophy	1	acc	0.1897	±	0.0223
		acc_norm	0.1897	±	0.0223
hendrycksTest-prehistory	1	acc	0.2377	±	0.0237
		acc_norm	0.2377	±	0.0237
hendrycksTest-professional_accounting	1	acc	0.2482	±	0.0258
		acc_norm	0.2482	±	0.0258
hendrycksTest-professional_law	1	acc	0.2464	±	0.0110
		acc_norm	0.2464	±	0.0110
hendrycksTest-professional_medicine	1	acc	0.4265	±	0.0300
		acc_norm	0.4265	±	0.0300
hendrycksTest-professional_psychology	1	acc	0.2614	±	0.0178
		acc_norm	0.2614	±	0.0178
hendrycksTest-public_relations	1	acc	0.1818	±	0.0369
		acc_norm	0.1818	±	0.0369
hendrycksTest-security_studies	1	acc	0.1959	±	0.0254
		acc_norm	0.1959	±	0.0254
hendrycksTest-sociology	1	acc	0.2289	±	0.0297
		acc_norm	0.2289	±	0.0297
hendrycksTest-us_foreign_policy	1	acc	0.2400	±	0.0429
		acc_norm	0.2400	±	0.0429
hendrycksTest-virology	1	acc	0.2048	±	0.0314
		acc_norm	0.2048	±	0.0314
hendrycksTest-world_religions	1	acc	0.2222	±	0.0319
		acc_norm	0.2222	±	0.0319

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/checkpoint-pegfss6f:v13,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 5, batch_size: 16

Task	Version	Metric	Value		Stderr
winogrande	0	acc	0.5099	±	0.014

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/checkpoint-pegfss6f:v13,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 5, batch_size: 16

Task	Version	Metric	Value		Stderr
gsm8k	0	acc	0.0	±	0.0

Model Details

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: [More Information Needed]
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: [More Information Needed]
• Language(s) (NLP): [More Information Needed]
• License: [More Information Needed]
• Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

Direct Use

[More Information Needed]

Downstream Use [optional]

[More Information Needed]

Out-of-Scope Use

[More Information Needed]

Bias, Risks, and Limitations

[More Information Needed]

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

Training Details

Training Data

[More Information Needed]

Training Procedure

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

• Training regime: [More Information Needed]

Speeds, Sizes, Times [optional]

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics

Testing Data

[More Information Needed]

Factors

[More Information Needed]

Metrics

[More Information Needed]

Results

[More Information Needed]

Summary

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	28.66
AI2 Reasoning Challenge (25-Shot)	21.93
HellaSwag (10-Shot)	27.86
MMLU (5-Shot)	25.34
TruthfulQA (0-shot)	46.00
Winogrande (5-shot)	50.83
GSM8k (5-shot)	0.00