local block causal when cuda avail

xformers when cuda available

more finishing touches

Working locally, TBD HF space

Visualisation working on CPU via CUDA_VISIBLE_DEVICE=-1 python demo_patcher.py 'Daenerys Targaryen is in Game of Thrones, a fantasy epic by George R.R. Martin.'

Improve HF integration (#98)

Open source weights! (#97)

Cast int sample id to str (#96)

Init distributed when loading model (#94)

Fix eval mask (#93)

remove selective activation checkpointing (#92)

Get generation working for BLT (#86)

Fix in-place addition of patch_embds (#85)

Some fixes for entropy model predictions (#83)

Update ppl evals to work with blt model, in addition to entropy model (#82)

Update iterate_data (#81)

Add way to call consolidate (#80)

When merging configs, do not merge data sources (#79)

Get evals working again. (#46)

Reduce per file resources arrow uses (#77)

Let process start before yielding preloaded prefetch buffer, avoid needlessly losing buffer in edge cases (#75)

Add approximate state persistence (#73)

Fix rsync to not preserve original permissions, instead use destination (#76)

Correctly reset batch iterator at each arrow create_iter call. (#74)

Pass mask in packing_iterator, correctly handle last batch, fix masking (#65)

Initialize rope embeddings properly for the entropy model (#72)

Remove byte tokenizer and add config args to switch between byte/patch packing (#68)

Add vocab and seq len abstract fields (#66)

Fix: Correct model_args usage in parallelize_model call (#69)

Update iterator inheritance, pass file format args, limit iterator (#63)

Make apex logs less noisy (#60)

Make it possible to specify multiple config files (#54)

using apex rmsnorm (#57)

Fix multiprocessing dataloader checkpointing and use it in the train script (#50)

Test first batch matches (#53)

disable reshard after forward (#56)

make sure max_encoder_seq_length matches (#55)

fix save and reload model state (#49)

Add bpb and n_bytes to metric logging (#41)

Fix init and repro (#48)

Allow ArrowIterator to read from json (#45)

Update checkpointing to use fsspec (#39)

Add rope fp32 (#43)

fix stool (#44)

Fix wandb logging (#42)

Fix distributed all reduce grad norm (#40)

This includes fixes that make checkpointing and reloading work correctly. (#35)

Initial codes and scripts for training entropy model (#34)

Use load_async flag to not start MP iterator (#33)

Update file check script to check sizes (#32)