kz919/llama3_1b_cautious_chinchilla_8252025

Model Details

This model is a 1B llama3 model pretrained from scratch with torchtitan on fineweb-edu with C_AdamW optimizer. 20B tokens seen.

How to use

import torch
from transformers import pipeline


pipe = pipeline(
    "text-generation",
    model="kz919/llama3_1b_cautious_chinchilla_8252025",
)

print(pipe("The key to life is"))

Downstream Eval

ARC, Hellaswag, Lambda_OpenAI, OpenbookQA, PIQA

lm_eval --model hf --model_args pretrained=kz919/llama3_1b_cautious_chinchilla_8252025,dtype="bfloat16",add_bos_token=True --tasks lambada_openai,hellaswag,piqa,arc_easy,arc_challenge,openbookqa --device cuda:7 --batch_size 8

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
arc_challenge	1	none	0	acc	↑	0.2713	±	0.0130
		none	0	acc_norm	↑	0.3123	±	0.0135
arc_easy	1	none	0	acc	↑	0.6309	±	0.0099
		none	0	acc_norm	↑	0.5556	±	0.0102
hellaswag	1	none	0	acc	↑	0.3598	±	0.0048
		none	0	acc_norm	↑	0.4397	±	0.0050
lambada_openai	1	none	0	acc	↑	0.3375	±	0.0066
		none	0	perplexity	↓	35.5328	±	1.3883
openbookqa	1	none	0	acc	↑	0.2260	±	0.0187
		none	0	acc_norm	↑	0.3360	±	0.0211
piqa	1	none	0	acc	↑	0.6839	±	0.0108
		none	0	acc_norm	↑	0.6839	±	0.0108

MMLU

Groups	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc	↑	0.2465	±	0.0036
- humanities	2	none		acc	↑	0.2555	±	0.0064
- other	2	none		acc	↑	0.2182	±	0.0074
- social sciences	2	none		acc	↑	0.2577	±	0.0079
- stem	2	none		acc	↑	0.2502	±	0.0077