PJMixers-Dev/LLaMa-3.2-Text-Cleaner-v0.1-1B-GGUF

PJMixers-Dev/LLaMa-3.2-Text-Cleaner-v0.1-1B

Model was trained at 16,384 max length, so potentially 8K input 8K output. Model will likely heavily reformat text, but hopefully end up with a cleaner result. 36,603,909 tokens, 18,552,131 of them supervised.

Probably not good for cleaning something you need to be 100% accurate to the original, like educational texts, but probably fine for cleaning creative writing datasets.

Quants

• GGUF

Prompt format

<|begin_of_text|><|unclean_text|>Put your uncleaned text here.<|unclean_text|>The model will respond with a cleaned version here.<|end_of_text|>

Example using a sample from PJMixers/RyokoAI_Honeyfeed3600, which the model has not been trained on. do_sample=True, max_new_tokens=4096, temperature=0.7, min_p=0.05

<|begin_of_text|><|unclean_text|>As the first ray of sun dawns through the window and illuminates the bedroom and its every corner, the lingering shadow and blackness vanish as if it was never there. As the sun slowly starts to rise to its peak, the rays of sunlight change their position and fall on the face of the fairy. She twitches her face as it was obvious that the sunlight was waking her up from her deep slumber. Opening her eyes, she squinted them as after a long sleep her eyes were not yet adapted to the strong light released by the sun. She rubbed her eyes with the back of her hand to block the sunlight and let them adapt to the light. Stretching her hand in a big yawn and release the stiffness in her body which was accumulated from the long sleep and she got out of bed so she could get ready and check every room of the place which she decided to call her home.


The very first room she decided to check was the exact same room she was in right now and in which she decided to sleep in after a few moments of her arrival. Right in front of the bed was a big closet, it was with a mixture of applewood and rosewood. Applewood had a light tone and texture compared to rose wood which was dark in tone and texture. So most part of cupboard was made of apple wood and rode wood was used to add decoration as well as handles. It was made keeping its preimmunises in mind, so the closet was very luxurious. The dark and light toning of wood color made the closet stand out more in the room. The fairy walked towards the closet and put her both hands on the closet’s each handles and pulled the door towards her, inside the closet on the left-hand side of it on the door there was a very tall oval shaped mirror which was made for looking at the whole body not just face but outfit as well and inside of the closet was dresses of various sizes. Looking at it, it was like the closet was either filled with the clothes of various different people or it was filled with the clothes of same person being of various ages in time. So, there were small dressed and large dresses, some was of small girl somewhat of 7 to 9 years old and some dresses had a very loose areas around the chest, looking at them, it was obvious that this dress was made for grown woman. The closet was filled with dresses from a childs size to till adult grown woman size who seems to be of an age of somewhat 23 to 25.


The fairy browsed between those dresses which were hanged inside of closet and looking through each dress her eyes fell on a certain dress. Fairy was confused, her cheeks were bright red and a little hot and there was a moment of little flutter in her heart seeing that dress. She was confused, she didn’t know what were those emotions and what was the reason she was feeling. She was holding that dress in her hand and thinking about the reason why was she feeling those things, she spends quite a time mull-over it, and in the end, she came to the conclusion that the human emotion and heart are out of her understanding there was lot that she didn’t understand and as she has those too so she couldn’t forget about it and those thought stuck in the back of her mind, with that she change her train of thought, she saw a drawer right below the place where the dresses were hanging, similarly, like the out side of the closet the inside of the closet was made of mixture of two wood and the drawer which was inside the closet was as well. The compartment of the drawer was made of rosewood which hold the sliding part of the drawer which was made of apple wood and the handle is made of rose wood. She pulled the drawer and saw there were clothes inside of it as well, but she didn’t know what the clothes were intended for, they covered much less area and some had elastic and some were transparent, she left those clothes inside the drawer quite confused. There was a lot to check in this room but she decided it was enough for today and it was time to check other rooms. The next room on her list was the library which situated on the second floor of the tower.


‡-_-_-_-‡<|clean_text|>The first ray of sun dawns through the window, illuminating the bedroom and its every corner. The lingering shadow and blackness vanish as if it was never there. As the sun slowly starts to rise, the rays of sunlight change their position, falling on the fairy's face. She twitches as if waking from a deep slumber. Opening her eyes, she squints, for after a long sleep, her eyes are still not yet adapted to the strong light. She rubs them with the back of her hand, blocking the light, letting them adapt. Stretching her hand wide and releasing the stiffness accumulated from the long sleep, she gets out of bed to get ready and check every room she decided to call her home.

The very first room she decides to check is the one she slept in, the one right in front of her bed. Before the bed, a large closet stands, with a mixture of applewood and rosewood. Applewood has a light tone and texture compared to rosewood, which is dark in tone and texture. Most of the cupboard is applewood, and rosewood is used for decoration and handles. It is made keeping its preimmunises in mind, so the closet is very luxurious. The dark and light-toned wood colour makes the closet stand out more in the room.

The fairy walks towards the closet and puts her both hands on the closet's handles and pulls the door towards her. Inside, on the left-hand side of the door, there is a very tall, oval-shaped mirror, designed for looking at the whole body, not just the face, but also the outfit. Inside the closet are dresses of various sizes. Looking at it, it is like the closet is filled with clothes of different people, or it is filled with clothes of the same person in different ages. There are small dresses and large dresses, some for small girls, somewhat between seven and nine years old, and some with loose areas around the chest. It is obvious that this dress is made for grown women. The closet is filled with dresses from a child's size to adult sizes, who seem to be between twenty-three and twenty-five.

The fairy browses between the dresses hung inside the closet, and her eyes fall on a certain dress. She is confused. Her cheeks are bright red, and there is a moment of flutter in her heart, seeing that dress. She is confused; she doesn't know what those emotions are, and what is the reason she is feeling. She holds that dress in her hand, thinking about the reason why she is feeling these things. She spends quite a time mulling it over, and in the end, she comes to the conclusion that human emotions and hearts are beyond her understanding. There is a lot that she doesn't understand, and so, she can't forget about it, and the thought sticks in the back of her mind. With that, she changes her train of thought. She sees a drawer below where the dresses are hanging. Just like outside the closet, the inside of the closet is made of a mixture of two woods, and the drawer is inside the closet as well. The compartment of the drawer is made of rosewood, which holds the sliding part, which is made of applewood, and the handle is made of rosewood. She pulls the drawer, and she sees clothes inside, but she doesn't know what they are intended for. They cover much less area, and some have elastic, and some are transparent. She leaves those clothes inside the drawer, quite confused. There is a lot to check in this room, but she decides it is enough for today, and it is time to check other rooms.

The next room on her list is the library, situated on the second floor of the tower.<|end_of_text|>

Axolotl Config

# Requirements before running
#   - Get latest commit of axolotl (currently c0a0c75)
#   - Download these to axolotl/src/axolotl/prompt_formatters
#     - https://github.com/xzuyn/axolotl/blob/came-plus-formatters/src/axolotl/prompt_strategies/text-cleaner.py
#   - pip install git+https://github.com/xzuyn/CAME.git@sr-grams-cautious-8bit

# Weights and Biases logging config
wandb_project: LLaMa-3.2-1B
wandb_entity:
wandb_watch:
wandb_name: LLaMa-3.2-Text-Cleaner-v0.1-1B-FFT-run4
wandb_log_model:

# Model checkpointing config
output_dir: ./Outputs/LLaMa-3.2-Text-Cleaner-v0.1-1B-FFT-run4
save_steps: 10
save_safetensors: true
save_total_limit: 2
save_only_model: true

# Model architecture config
base_model: meta-llama/Llama-3.2-1B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
chat_template_jinja: "{{- bos_token }}{% for message in messages %}{% if message['role'] == 'system' %}{{ raise_exception('Model does not support system turns.') }}{% elif message['role'] == 'user' %}{{ '<|unclean_text|>' + message['content'] | trim }}{% elif message['role'] == 'assistant' %}{{ '<|clean_text|>' + message['content'] | trim + eos_token }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|clean_text|>' }}{% endif %}"

# Mixed precision training config
bf16: true
fp16: false
tf32: false

# Model loading config
load_in_8bit: false
load_in_4bit: false
strict: false

# Sequence config
sequence_len: 16384
min_sample_len: 256
sample_packing: false
eval_sample_packing: false
pad_to_sequence_len: false
train_on_inputs: false
group_by_length: false

# Dataset config
datasets:
  - path: PJMixers-Dev/Nelathan_synthetic-sugar-quill-cleaner
    type: text-cleaner
val_set_size: 128
eval_strategy: steps
eval_steps: 10
dataset_prepared_path: ./00-Tokenized-Datasets/LLaMa-3.2-Text-Cleaner-v0.1-1B-seed42
shuffle_merged_datasets: true
dataset_exact_deduplication: true

# Training hyperparameters
num_epochs: 1
gradient_accumulation_steps: 1
micro_batch_size: 8
eval_batch_size: 8
warmup_steps: 0
optimizer: came_pytorch
optim_args:
  enable_stochastic_rounding: true
  enable_cautious: true
  enable_8bit: true
lr_scheduler: rex
learning_rate: 1e-6
cosine_min_lr_ratio: 0.05
weight_decay: 0.01
max_grad_norm: 0.5
logging_steps: 1

# Model optimization
embeddings_skip_upcast: true
gradient_checkpointing: offload
sdp_attention: true
plugins:
  - axolotl.integrations.liger.LigerPlugin
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
cut_cross_entropy: true
liger_rope: true
liger_rms_norm: true
liger_layer_norm: true
liger_glu_activation: true
liger_cross_entropy: false
liger_fused_linear_cross_entropy: false

# Garbage Collection
gc_steps: 1

# Debug config
debug: true
seed: 42

# Token config
added_tokens_overrides:
  128011: "<|unclean_text|>"
  128012: "<|clean_text|>"
special_tokens:
  bos_token: "<|begin_of_text|>"
  eos_token: "<|end_of_text|>"
  pad_token: "<|finetune_right_pad_id|>"
tokens: