tFINE-900m-e16-d32-1024ctx

Pretrained T5 model with nanoT5:

~900m parameters, 16 layers in encoder, 32 layers in decoder
sentencepiece tokenizer with 48k vocab & byte-pair fallback
- handles whitespaces etc correctly (unlike original T5 tokenizer)
1024 ctx during pretrain
relative_attention_num_buckets increased to 48 from 32 for context length upscaling

Experiment logs

Training consisted of two phases:

phase one - ~30k steps at context length 512
phase two - 20k steps at context length 1024

Downloads last month: 6

Safetensors

Model size

887M params

Tensor type

F32

Inference Providers NEW

Text2Text Generation

This model is not currently available via any of the supported third-party Inference Providers, and the model is not deployed on the HF Inference API.

Model tree for pszemraj/tFINE-900m-e16-d32-1024ctx

Finetunes

1 model

Dataset used to train pszemraj/tFINE-900m-e16-d32-1024ctx

Collection including pszemraj/tFINE-900m-e16-d32-1024ctx

tFINE

Collection

pretrained t5 models on high quality data(e.g. fineweb) • 6 items • Updated 14 days ago