This is a SmolLM2-135M-Instruct model fine-tuned on the Faroese portion of Fineweb-2. It is intended for my research and has not been evaluated more broadly yet.

Training:

5 Epochs
Learning rate: 5e-4
LR scheduler: Cosine
Warmup ratio: 0.05
Batch size: 1
4 A100 (40GB) GPUs
Gradient accumulation steps: 64
Effective batch size: 256
Max. context length: 8192 tokens

Downloads last month: 13

Safetensors

Model size

135M params

Tensor type

F32

Inference Providers NEW

Text Generation

This model is not currently available via any of the supported third-party Inference Providers, and HF Inference API was unable to determine this model's library.

Model tree for jekunz/smollm-135m-cpt-fineweb-faroese

Base model

HuggingFaceTB/SmolLM2-135M

Quantized

HuggingFaceTB/SmolLM2-135M-Instruct

Finetuned

(80)

this model

Dataset used to train jekunz/smollm-135m-cpt-fineweb-faroese

Collection including jekunz/smollm-135m-cpt-fineweb-faroese

SmolLM CPT

Collection

Continued Pre-Training of SmolLM models on the Fineweb-2 portions of Scandinavian languages. • 7 items • Updated 2 days ago