[2025-04-27 14:34:27,301] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The cache directory for DeepSpeed Triton autotune, /home/jg9904/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
[2025-04-27 14:34:29,565] [WARNING] [runner.py:215:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
Detected VISIBLE_DEVICES=0,1,2,3,4,5: setting --include=localhost:0,1,2,3,4,5
[2025-04-27 14:34:29,565] [INFO] [runner.py:605:main] cmd = /home/jg9904/.conda/envs/rlenv/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgMywgNCwgNV19 --master_addr=127.0.0.1 --master_port=29650 --enable_each_rank_log=None train.py --deepspeed scripts/newzero3.json --seed 26 --model_name_or_path /scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL --train_tokenized_file /scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/data/14K_reward-r2.jsonl --output_dir /scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1 --per_device_train_batch_size 1 --gradient_accumulation_steps 2 --evaluation_strategy no --save_strategy no --learning_rate 9e-7 --lr_scheduler_type cosine --save_only_model True --remove_unused_columns False --warmup_ratio 0.03 --num_train_epochs 2 --logging_steps 1 --report_to tensorboard --gradient_checkpointing True --overwrite_output_dir --bf16 True
[2025-04-27 14:34:31,012] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The cache directory for DeepSpeed Triton autotune, /home/jg9904/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
[2025-04-27 14:34:33,028] [INFO] [launch.py:146:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3, 4, 5]}
[2025-04-27 14:34:33,028] [INFO] [launch.py:152:main] nnodes=1, num_local_procs=6, node_rank=0
[2025-04-27 14:34:33,028] [INFO] [launch.py:163:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3, 4, 5]})
[2025-04-27 14:34:33,028] [INFO] [launch.py:164:main] dist_world_size=6
[2025-04-27 14:34:33,028] [INFO] [launch.py:168:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3,4,5
[2025-04-27 14:34:33,028] [INFO] [launch.py:256:main] process 2201280 spawned with command: ['/home/jg9904/.conda/envs/rlenv/bin/python', '-u', 'train.py', '--local_rank=0', '--deepspeed', 'scripts/newzero3.json', '--seed', '26', '--model_name_or_path', '/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL', '--train_tokenized_file', '/scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/data/14K_reward-r2.jsonl', '--output_dir', '/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1', '--per_device_train_batch_size', '1', '--gradient_accumulation_steps', '2', '--evaluation_strategy', 'no', '--save_strategy', 'no', '--learning_rate', '9e-7', '--lr_scheduler_type', 'cosine', '--save_only_model', 'True', '--remove_unused_columns', 'False', '--warmup_ratio', '0.03', '--num_train_epochs', '2', '--logging_steps', '1', '--report_to', 'tensorboard', '--gradient_checkpointing', 'True', '--overwrite_output_dir', '--bf16', 'True']
[2025-04-27 14:34:33,034] [INFO] [launch.py:256:main] process 2201281 spawned with command: ['/home/jg9904/.conda/envs/rlenv/bin/python', '-u', 'train.py', '--local_rank=1', '--deepspeed', 'scripts/newzero3.json', '--seed', '26', '--model_name_or_path', '/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL', '--train_tokenized_file', '/scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/data/14K_reward-r2.jsonl', '--output_dir', '/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1', '--per_device_train_batch_size', '1', '--gradient_accumulation_steps', '2', '--evaluation_strategy', 'no', '--save_strategy', 'no', '--learning_rate', '9e-7', '--lr_scheduler_type', 'cosine', '--save_only_model', 'True', '--remove_unused_columns', 'False', '--warmup_ratio', '0.03', '--num_train_epochs', '2', '--logging_steps', '1', '--report_to', 'tensorboard', '--gradient_checkpointing', 'True', '--overwrite_output_dir', '--bf16', 'True']
[2025-04-27 14:34:33,034] [INFO] [launch.py:256:main] process 2201282 spawned with command: ['/home/jg9904/.conda/envs/rlenv/bin/python', '-u', 'train.py', '--local_rank=2', '--deepspeed', 'scripts/newzero3.json', '--seed', '26', '--model_name_or_path', '/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL', '--train_tokenized_file', '/scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/data/14K_reward-r2.jsonl', '--output_dir', '/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1', '--per_device_train_batch_size', '1', '--gradient_accumulation_steps', '2', '--evaluation_strategy', 'no', '--save_strategy', 'no', '--learning_rate', '9e-7', '--lr_scheduler_type', 'cosine', '--save_only_model', 'True', '--remove_unused_columns', 'False', '--warmup_ratio', '0.03', '--num_train_epochs', '2', '--logging_steps', '1', '--report_to', 'tensorboard', '--gradient_checkpointing', 'True', '--overwrite_output_dir', '--bf16', 'True']
[2025-04-27 14:34:33,035] [INFO] [launch.py:256:main] process 2201283 spawned with command: ['/home/jg9904/.conda/envs/rlenv/bin/python', '-u', 'train.py', '--local_rank=3', '--deepspeed', 'scripts/newzero3.json', '--seed', '26', '--model_name_or_path', '/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL', '--train_tokenized_file', '/scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/data/14K_reward-r2.jsonl', '--output_dir', '/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1', '--per_device_train_batch_size', '1', '--gradient_accumulation_steps', '2', '--evaluation_strategy', 'no', '--save_strategy', 'no', '--learning_rate', '9e-7', '--lr_scheduler_type', 'cosine', '--save_only_model', 'True', '--remove_unused_columns', 'False', '--warmup_ratio', '0.03', '--num_train_epochs', '2', '--logging_steps', '1', '--report_to', 'tensorboard', '--gradient_checkpointing', 'True', '--overwrite_output_dir', '--bf16', 'True']
[2025-04-27 14:34:33,035] [INFO] [launch.py:256:main] process 2201284 spawned with command: ['/home/jg9904/.conda/envs/rlenv/bin/python', '-u', 'train.py', '--local_rank=4', '--deepspeed', 'scripts/newzero3.json', '--seed', '26', '--model_name_or_path', '/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL', '--train_tokenized_file', '/scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/data/14K_reward-r2.jsonl', '--output_dir', '/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1', '--per_device_train_batch_size', '1', '--gradient_accumulation_steps', '2', '--evaluation_strategy', 'no', '--save_strategy', 'no', '--learning_rate', '9e-7', '--lr_scheduler_type', 'cosine', '--save_only_model', 'True', '--remove_unused_columns', 'False', '--warmup_ratio', '0.03', '--num_train_epochs', '2', '--logging_steps', '1', '--report_to', 'tensorboard', '--gradient_checkpointing', 'True', '--overwrite_output_dir', '--bf16', 'True']
[2025-04-27 14:34:33,035] [INFO] [launch.py:256:main] process 2201285 spawned with command: ['/home/jg9904/.conda/envs/rlenv/bin/python', '-u', 'train.py', '--local_rank=5', '--deepspeed', 'scripts/newzero3.json', '--seed', '26', '--model_name_or_path', '/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL', '--train_tokenized_file', '/scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/data/14K_reward-r2.jsonl', '--output_dir', '/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1', '--per_device_train_batch_size', '1', '--gradient_accumulation_steps', '2', '--evaluation_strategy', 'no', '--save_strategy', 'no', '--learning_rate', '9e-7', '--lr_scheduler_type', 'cosine', '--save_only_model', 'True', '--remove_unused_columns', 'False', '--warmup_ratio', '0.03', '--num_train_epochs', '2', '--logging_steps', '1', '--report_to', 'tensorboard', '--gradient_checkpointing', 'True', '--overwrite_output_dir', '--bf16', 'True']
[2025-04-27 14:34:36,467] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The cache directory for DeepSpeed Triton autotune, /home/jg9904/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
[2025-04-27 14:34:36,548] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-27 14:34:36,562] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-27 14:34:36,587] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The cache directory for DeepSpeed Triton autotune, /home/jg9904/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /home/jg9904/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /home/jg9904/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
[2025-04-27 14:34:36,668] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-27 14:34:36,669] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The cache directory for DeepSpeed Triton autotune, /home/jg9904/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /home/jg9904/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2025-04-27 14:34:38,707] [INFO] [comm.py:658:init_distributed] cdb=None
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2025-04-27 14:34:38,936] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-27 14:34:38,936] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2025-04-27 14:34:38,948] [INFO] [comm.py:658:init_distributed] cdb=None
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2025-04-27 14:34:38,978] [INFO] [comm.py:658:init_distributed] cdb=None
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2025-04-27 14:34:39,046] [INFO] [comm.py:658:init_distributed] cdb=None
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2025-04-27 14:34:39,081] [INFO] [comm.py:658:init_distributed] cdb=None
WARNING:__main__:Process rank: 4, device: cuda:4, n_gpu: 1
WARNING:__main__:Process rank: 2, device: cuda:2, n_gpu: 1
WARNING:__main__:Process rank: 3, device: cuda:3, n_gpu: 1
WARNING:__main__:Process rank: 1, device: cuda:1, n_gpu: 1
WARNING:__main__:Process rank: 0, device: cuda:0, n_gpu: 1
INFO:__main__:Training parameters CustomTrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=scripts/newzero3.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_predict=False,
do_train=False,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=None,
eval_strategy=no,
eval_use_gather_object=False,
evaluation_strategy=no,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=2,
gradient_checkpointing=True,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
kl_coeff=0.0,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=9e-07,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1/runs/Apr27_14-34-38_della-j16g2,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=2.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=False,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1,
save_on_each_node=False,
save_only_model=True,
save_safetensors=True,
save_steps=500,
save_strategy=no,
save_total_limit=None,
seed=26,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tp_size=0,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.0,
)
[INFO|tokenization_utils_base.py:2058] 2025-04-27 14:34:39,869 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2058] 2025-04-27 14:34:39,869 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2058] 2025-04-27 14:34:39,869 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2058] 2025-04-27 14:34:39,869 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2058] 2025-04-27 14:34:39,869 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2058] 2025-04-27 14:34:39,869 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2058] 2025-04-27 14:34:39,869 >> loading file chat_template.jinja
[2025-04-27 14:34:39,910] [INFO] [config.py:734:__init__] Config mesh_device None world_size = 6
[WARNING|logging.py:329] 2025-04-27 14:34:39,913 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[2025-04-27 14:34:39,917] [INFO] [config.py:734:__init__] Config mesh_device None world_size = 6
[WARNING|logging.py:329] 2025-04-27 14:34:39,919 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[2025-04-27 14:34:39,927] [INFO] [config.py:734:__init__] Config mesh_device None world_size = 6
[2025-04-27 14:34:39,928] [INFO] [config.py:734:__init__] Config mesh_device None world_size = 6
[WARNING|logging.py:329] 2025-04-27 14:34:39,929 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[WARNING|logging.py:329] 2025-04-27 14:34:39,930 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
WARNING:__main__:Process rank: 5, device: cuda:5, n_gpu: 1
[INFO|tokenization_utils_base.py:2323] 2025-04-27 14:34:40,171 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[INFO|configuration_utils.py:697] 2025-04-27 14:34:40,171 >> loading configuration file /scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL/config.json
[INFO|configuration_utils.py:771] 2025-04-27 14:34:40,173 >> Model config Qwen2Config {
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "hidden_act": "silu",
  "hidden_size": 5120,
  "initializer_range": 0.02,
  "intermediate_size": 13824,
  "max_position_embeddings": 32768,
  "max_window_layers": 70,
  "model_type": "qwen2",
  "num_attention_heads": 40,
  "num_hidden_layers": 48,
  "num_key_value_heads": 8,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.50.3",
  "use_cache": false,
  "use_sliding_window": false,
  "vocab_size": 152064
}

[INFO|modeling_utils.py:1151] 2025-04-27 14:34:40,207 >> loading weights file /scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL/model.safetensors.index.json
[INFO|modeling_utils.py:1225] 2025-04-27 14:34:40,208 >> Will use torch_dtype=torch.bfloat16 as defined in model's config object
[INFO|modeling_utils.py:2170] 2025-04-27 14:34:40,208 >> Instantiating Qwen2ForCausalLM model under default dtype torch.bfloat16.
[INFO|modeling_utils.py:3747] 2025-04-27 14:34:40,208 >> Detected DeepSpeed ZeRO-3: activating zero.init() for this model
[2025-04-27 14:34:40,208] [INFO] [config.py:734:__init__] Config mesh_device None world_size = 6
[WARNING|logging.py:329] 2025-04-27 14:34:40,211 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[INFO|configuration_utils.py:1139] 2025-04-27 14:34:40,216 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "use_cache": false
}

[2025-04-27 14:34:40,291] [INFO] [config.py:734:__init__] Config mesh_device None world_size = 6
[WARNING|logging.py:329] 2025-04-27 14:34:40,294 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[2025-04-27 14:34:59,946] [INFO] [partition_parameters.py:348:__exit__] finished initializing model - num_params = 579, num_elems = 14.77B

Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:02<00:10,  2.02s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:02<00:10,  2.07s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:02<00:10,  2.01s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:02<00:10,  2.02s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:02<00:10,  2.04s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:02<00:10,  2.18s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:04<00:08,  2.06s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:04<00:08,  2.08s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:04<00:08,  2.06s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:04<00:08,  2.06s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:04<00:08,  2.07s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:04<00:08,  2.12s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:06<00:06,  2.07s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:06<00:06,  2.06s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:06<00:06,  2.07s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:06<00:06,  2.08s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:06<00:06,  2.06s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:06<00:06,  2.09s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:08<00:04,  2.06s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:08<00:04,  2.06s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:08<00:04,  2.06s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:08<00:04,  2.06s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:08<00:04,  2.06s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:08<00:04,  2.07s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:10<00:02,  2.04s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:10<00:02,  2.05s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:10<00:02,  2.05s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:10<00:02,  2.05s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:10<00:02,  2.05s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:10<00:02,  2.05s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:12<00:00,  2.00s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:12<00:00,  2.03s/it]

Loading checkpoint shards: 100%|██████████| 6/6 [00:12<00:00,  2.00s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:12<00:00,  2.03s/it]

Loading checkpoint shards: 100%|██████████| 6/6 [00:12<00:00,  2.00s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:12<00:00,  2.03s/it]

Loading checkpoint shards: 100%|██████████| 6/6 [00:12<00:00,  2.00s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:12<00:00,  2.04s/it]

Loading checkpoint shards: 100%|██████████| 6/6 [00:12<00:00,  2.00s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:12<00:00,  2.03s/it]
/scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/train.py:274: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `OfflineREINFORCETrainer.__init__`. Use `processing_class` instead.
  trainer = OfflineREINFORCETrainer(
/scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/train.py:274: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `OfflineREINFORCETrainer.__init__`. Use `processing_class` instead.
  trainer = OfflineREINFORCETrainer(

Loading checkpoint shards: 100%|██████████| 6/6 [00:12<00:00,  2.01s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:12<00:00,  2.05s/it]
[INFO|modeling_utils.py:4987] 2025-04-27 14:35:12,270 >> All model checkpoint weights were used when initializing Qwen2ForCausalLM.

[INFO|modeling_utils.py:4995] 2025-04-27 14:35:12,270 >> All the weights of Qwen2ForCausalLM were initialized from the model checkpoint at /scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL.
If your task is similar to the task the model of the checkpoint was trained on, you can already use Qwen2ForCausalLM for predictions without further training.
[INFO|configuration_utils.py:1092] 2025-04-27 14:35:12,274 >> loading configuration file /scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL/generation_config.json
[INFO|configuration_utils.py:1139] 2025-04-27 14:35:12,275 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "do_sample": true,
  "eos_token_id": [
    151645,
    151643
  ],
  "pad_token_id": 151643,
  "repetition_penalty": 1.05,
  "temperature": 0.7,
  "top_k": 20,
  "top_p": 0.8
}

/scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/train.py:274: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `OfflineREINFORCETrainer.__init__`. Use `processing_class` instead.
  trainer = OfflineREINFORCETrainer(
Using custom data configuration default-74d43af3a2a8aa75
INFO:datasets.builder:Using custom data configuration default-74d43af3a2a8aa75
Loading Dataset Infos from /home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/datasets/packaged_modules/json
INFO:datasets.info:Loading Dataset Infos from /home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/datasets/packaged_modules/json
Overwrite dataset info from restored data version if exists.
INFO:datasets.builder:Overwrite dataset info from restored data version if exists.
Loading Dataset info from /home/jg9904/.cache/huggingface/datasets/json/default-74d43af3a2a8aa75/0.0.0/f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092
INFO:datasets.info:Loading Dataset info from /home/jg9904/.cache/huggingface/datasets/json/default-74d43af3a2a8aa75/0.0.0/f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092
Found cached dataset json (/home/jg9904/.cache/huggingface/datasets/json/default-74d43af3a2a8aa75/0.0.0/f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092)
INFO:datasets.builder:Found cached dataset json (/home/jg9904/.cache/huggingface/datasets/json/default-74d43af3a2a8aa75/0.0.0/f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092)
Loading Dataset info from /home/jg9904/.cache/huggingface/datasets/json/default-74d43af3a2a8aa75/0.0.0/f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092
INFO:datasets.info:Loading Dataset info from /home/jg9904/.cache/huggingface/datasets/json/default-74d43af3a2a8aa75/0.0.0/f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092
/scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/train.py:274: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `OfflineREINFORCETrainer.__init__`. Use `processing_class` instead.
  trainer = OfflineREINFORCETrainer(
/scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/train.py:274: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `OfflineREINFORCETrainer.__init__`. Use `processing_class` instead.
  trainer = OfflineREINFORCETrainer(
/scratch/gpfs/jg9904/cogbehaveRL/RL/offline_rl_v2/train.py:274: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `OfflineREINFORCETrainer.__init__`. Use `processing_class` instead.
  trainer = OfflineREINFORCETrainer(
[INFO|trainer.py:748] 2025-04-27 14:35:12,471 >> Using auto half precision backend
INFO:__main__:*** Train ***
[INFO|deepspeed.py:386] 2025-04-27 14:35:12,704 >> Detected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)
Installed CUDA version 12.8 does not match the version torch was compiled with 12.1 but since the APIs are compatible, accepting this combination
Using /home/jg9904/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Emitting ninja build file /home/jg9904/.cache/torch_extensions/py310_cu121/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 0.646265983581543 seconds
Installed CUDA version 12.8 does not match the version torch was compiled with 12.1 but since the APIs are compatible, accepting this combination
Using /home/jg9904/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Installed CUDA version 12.8 does not match the version torch was compiled with 12.1 but since the APIs are compatible, accepting this combination
Using /home/jg9904/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Installed CUDA version 12.8 does not match the version torch was compiled with 12.1 but since the APIs are compatible, accepting this combination
Using /home/jg9904/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Emitting ninja build file /home/jg9904/.cache/torch_extensions/py310_cu121/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 0.8059849739074707 seconds
Installed CUDA version 12.8 does not match the version torch was compiled with 12.1 but since the APIs are compatible, accepting this combination
Using /home/jg9904/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Emitting ninja build file /home/jg9904/.cache/torch_extensions/py310_cu121/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 0.504305362701416 seconds
Installed CUDA version 12.8 does not match the version torch was compiled with 12.1 but since the APIs are compatible, accepting this combination
Using /home/jg9904/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Emitting ninja build file /home/jg9904/.cache/torch_extensions/py310_cu121/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 0.5033798217773438 seconds
Loading extension module cpu_adam...
Loading extension module cpu_adam...
Time to load cpu_adam op: 0.8922135829925537 seconds
Time to load cpu_adam op: 0.825369119644165 seconds
Adam Optimizer #0 is created with AVX512 arithmetic capability.
Config: alpha=0.000001, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1
[2025-04-27 14:35:13,719] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed info: version=0.16.5, git-hash=unknown, git-branch=unknown
[2025-04-27 14:35:13,719] [INFO] [config.py:734:__init__] Config mesh_device None world_size = 6
[2025-04-27 14:35:13,730] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2025-04-27 14:35:13,731] [INFO] [logging.py:107:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2025-04-27 14:35:13,731] [INFO] [logging.py:107:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2025-04-27 14:35:13,753] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
[2025-04-27 14:35:13,753] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2025-04-27 14:35:13,753] [INFO] [logging.py:107:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False
[2025-04-27 14:35:13,753] [INFO] [logging.py:107:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer
[2025-04-27 14:35:13,877] [INFO] [utils.py:781:see_memory_usage] Stage 3 initialize beginning
[2025-04-27 14:35:13,878] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB         Max_MA 2.9 GB         CA 0.0 GB         Max_CA 3 GB 
[2025-04-27 14:35:13,878] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 82.67 GB, percent = 8.2%
[2025-04-27 14:35:13,880] [INFO] [stage3.py:170:__init__] Reduce bucket size 100000000
[2025-04-27 14:35:13,880] [INFO] [stage3.py:171:__init__] Prefetch bucket size 100000000
[2025-04-27 14:35:13,988] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]
[2025-04-27 14:35:13,988] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB 
[2025-04-27 14:35:13,989] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 82.67 GB, percent = 8.2%
Parameter Offload: Total persistent parameters: 840704 in 241 params
[2025-04-27 14:35:14,134] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [end]
[2025-04-27 14:35:14,134] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB 
[2025-04-27 14:35:14,135] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 82.68 GB, percent = 8.2%
[2025-04-27 14:35:14,237] [INFO] [utils.py:781:see_memory_usage] Before creating fp16 partitions
[2025-04-27 14:35:14,237] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB 
[2025-04-27 14:35:14,237] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 82.68 GB, percent = 8.2%
[2025-04-27 14:35:34,767] [INFO] [utils.py:781:see_memory_usage] After creating fp16 partitions: 24
[2025-04-27 14:35:34,768] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB 
[2025-04-27 14:35:34,768] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 142.43 GB, percent = 14.1%
[2025-04-27 14:35:35,072] [INFO] [utils.py:781:see_memory_usage] Before creating fp32 partitions
[2025-04-27 14:35:35,073] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB 
[2025-04-27 14:35:35,073] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 147.59 GB, percent = 14.7%
[2025-04-27 14:35:37,493] [INFO] [utils.py:781:see_memory_usage] After creating fp32 partitions
[2025-04-27 14:35:37,494] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB 
[2025-04-27 14:35:37,494] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 187.16 GB, percent = 18.6%
[2025-04-27 14:35:37,671] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2025-04-27 14:35:37,672] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB 
[2025-04-27 14:35:37,672] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 193.51 GB, percent = 19.2%
[2025-04-27 14:35:43,524] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2025-04-27 14:35:43,525] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB 
[2025-04-27 14:35:43,525] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 268.0 GB, percent = 26.6%
[2025-04-27 14:35:43,607] [INFO] [stage3.py:534:_setup_for_real_optimizer] optimizer state initialized
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/transformers/data/data_collator.py:741: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:278.)
  batch["labels"] = torch.tensor(batch["labels"], dtype=torch.int64)
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/transformers/data/data_collator.py:741: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:278.)
  batch["labels"] = torch.tensor(batch["labels"], dtype=torch.int64)
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/transformers/data/data_collator.py:741: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:278.)
  batch["labels"] = torch.tensor(batch["labels"], dtype=torch.int64)
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/transformers/data/data_collator.py:741: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:278.)
  batch["labels"] = torch.tensor(batch["labels"], dtype=torch.int64)
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/transformers/data/data_collator.py:741: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:278.)
  batch["labels"] = torch.tensor(batch["labels"], dtype=torch.int64)
[2025-04-27 14:35:47,490] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2025-04-27 14:35:47,491] [INFO] [utils.py:782:see_memory_usage] MA 0.19 GB         Max_MA 3.09 GB         CA 3.09 GB         Max_CA 3 GB 
[2025-04-27 14:35:47,491] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 315.26 GB, percent = 31.3%
[2025-04-27 14:35:47,491] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer_Stage3
[2025-04-27 14:35:47,491] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = None
[2025-04-27 14:35:47,491] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed LR Scheduler = None
[2025-04-27 14:35:47,491] [INFO] [logging.py:107:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0, 0.0], mom=[(0.9, 0.999), (0.9, 0.999)]
[2025-04-27 14:35:47,492] [INFO] [config.py:1000:print] DeepSpeedEngine configuration:
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'intra_op_parallelism': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   amp_enabled .................. False
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   amp_params ................... False
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   bfloat16_enabled ............. True
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   bfloat16_immediate_grad_update  True
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   checkpoint_parallel_write_pipeline  False
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   checkpoint_tag_validation_enabled  True
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   checkpoint_tag_validation_fail  False
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x14e674dc6890>
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   communication_data_type ...... None
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   curriculum_enabled_legacy .... False
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   curriculum_params_legacy ..... False
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'pin_memory': False, 'curriculum_learning': {'enabled': False}, 'dynamic_batching': {'enabled': False, 'lr_scaling_method': 'linear', 'min_batch_size': 1, 'max_batch_size': None, 'sequence_picking_order': 'dataloader', 'verbose': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   data_efficiency_enabled ...... False
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   dataloader_drop_last ......... False
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   disable_allgather ............ False
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   dump_state ................... False
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   dynamic_loss_scale_args ...... None
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   eigenvalue_enabled ........... False
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   eigenvalue_gas_boundary_resolution  1
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   eigenvalue_layer_num ......... 0
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   eigenvalue_max_iter .......... 100
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   eigenvalue_stability ......... 1e-06
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   eigenvalue_tol ............... 0.01
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   eigenvalue_verbose ........... False
[2025-04-27 14:35:47,493] [INFO] [config.py:1004:print]   elasticity_enabled ........... False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   fp16_auto_cast ............... None
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   fp16_enabled ................. False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   fp16_master_weights_and_gradients  False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   global_rank .................. 0
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   grad_accum_dtype ............. None
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   gradient_accumulation_steps .. 2
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   gradient_clipping ............ 1.0
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   gradient_predivide_factor .... 1.0
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   graph_harvesting ............. False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   initial_dynamic_scale ........ 1
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   load_universal_checkpoint .... False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   loss_scale ................... 1.0
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   memory_breakdown ............. False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   mics_hierarchial_params_gather  False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   mics_shard_size .............. -1
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   optimizer_legacy_fusion ...... False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   optimizer_name ............... None
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   optimizer_params ............. None
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   pld_enabled .................. False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   pld_params ................... False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   prescale_gradients ........... False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   scheduler_name ............... None
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   scheduler_params ............. None
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   seq_parallel_communication_data_type  torch.float32
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   sparse_attention ............. None
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   sparse_gradients_enabled ..... False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   steps_per_print .............. inf
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   tensor_parallel_config ....... dtype=torch.float16 autotp_size=0 tensor_parallel=TPConfig(tp_size=1, tp_grain_size=1, mpu=None, tp_group=None) injection_policy_tuple=None keep_module_on_host=False replace_with_kernel_inject=False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   timers_config ................ enabled=True synchronized=True
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   train_batch_size ............. 12
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   train_micro_batch_size_per_gpu  1
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   use_data_before_expert_parallel_  False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   use_node_local_storage ....... False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   wall_clock_breakdown ......... False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   weight_quantization_config ... None
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   world_size ................... 6
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   zero_allow_untested_optimizer  True
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=100000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='cpu', nvme_path=None, buffer_count=5, buffer_size=100000000, max_in_cpu=1000000000, pin_memory=True) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=100000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=100000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=100000000 max_reuse_distance=100000000 gather_16bit_weights_on_model_save=True module_granularity_threshold=0 use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False zeropp_loco_param=None mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True log_trace_cache_warnings=False
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   zero_enabled ................. True
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   zero_force_ds_cpu_optimizer .. True
[2025-04-27 14:35:47,494] [INFO] [config.py:1004:print]   zero_optimization_stage ...... 3
[2025-04-27 14:35:47,494] [INFO] [config.py:990:print_user_config]   json = {
    "fp16": {
        "enabled": false
    }, 
    "bf16": {
        "enabled": true
    }, 
    "train_micro_batch_size_per_gpu": 1, 
    "gradient_accumulation_steps": 2, 
    "zero_optimization": {
        "stage": 3, 
        "offload_optimizer": {
            "device": "cpu", 
            "pin_memory": true
        }, 
        "offload_param": {
            "device": "cpu", 
            "pin_memory": true
        }, 
        "overlap_comm": true, 
        "contiguous_gradients": true, 
        "sub_group_size": 1.000000e+08, 
        "reduce_bucket_size": 1.000000e+08, 
        "stage3_prefetch_bucket_size": 1.000000e+08, 
        "stage3_param_persistence_threshold": 1.000000e+05, 
        "stage3_max_live_parameters": 1.000000e+08, 
        "stage3_max_reuse_distance": 1.000000e+08, 
        "stage3_gather_16bit_weights_on_model_save": true
    }, 
    "gradient_clipping": 1.0, 
    "wall_clock_breakdown": false, 
    "steps_per_print": inf, 
    "zero_allow_untested_optimizer": true
}
[INFO|trainer.py:2409] 2025-04-27 14:35:47,495 >> ***** Running training *****
[INFO|trainer.py:2410] 2025-04-27 14:35:47,495 >>   Num examples = 8,460
[INFO|trainer.py:2411] 2025-04-27 14:35:47,495 >>   Num Epochs = 2
[INFO|trainer.py:2412] 2025-04-27 14:35:47,495 >>   Instantaneous batch size per device = 1
[INFO|trainer.py:2415] 2025-04-27 14:35:47,495 >>   Total train batch size (w. parallel, distributed & accumulation) = 12
[INFO|trainer.py:2416] 2025-04-27 14:35:47,495 >>   Gradient Accumulation steps = 2
[INFO|trainer.py:2417] 2025-04-27 14:35:47,495 >>   Total optimization steps = 1,410
[INFO|trainer.py:2418] 2025-04-27 14:35:47,496 >>   Number of trainable parameters = 14,770,033,664

  0%|          | 0/1410 [00:00<?, ?it/s]/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/transformers/data/data_collator.py:741: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:278.)
  batch["labels"] = torch.tensor(batch["labels"], dtype=torch.int64)
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
/home/jg9904/.conda/envs/rlenv/lib/python3.10/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]

  0%|          | 1/1410 [00:16<6:33:10, 16.74s/it]
                                                  
{'loss': 0.037, 'grad_norm': 2.6090409755706787, 'learning_rate': 2.0930232558139533e-08, 'kl': -0.0008, 'entropy': -0.0182, 'ce_loss': 0.0105, 'epoch': 0.0}

  0%|          | 1/1410 [00:16<6:33:10, 16.74s/it]
  0%|          | 2/1410 [00:28<5:25:35, 13.87s/it]
                                                  
{'loss': 0.0327, 'grad_norm': 2.1782753467559814, 'learning_rate': 4.1860465116279067e-08, 'kl': -0.0057, 'entropy': -0.0596, 'ce_loss': 0.0218, 'epoch': 0.0}

  0%|          | 2/1410 [00:28<5:25:35, 13.87s/it]
  0%|          | 3/1410 [00:40<5:05:32, 13.03s/it]
                                                  
{'loss': 0.0305, 'grad_norm': 2.813340425491333, 'learning_rate': 6.27906976744186e-08, 'kl': 0.0001, 'entropy': -0.0476, 'ce_loss': 0.0018, 'epoch': 0.0}

  0%|          | 3/1410 [00:40<5:05:32, 13.03s/it]
  0%|          | 4/1410 [00:52<4:55:14, 12.60s/it]
                                                  
{'loss': 0.0455, 'grad_norm': 2.8519327640533447, 'learning_rate': 8.372093023255813e-08, 'kl': -0.0006, 'entropy': -0.0371, 'ce_loss': 0.0199, 'epoch': 0.01}

  0%|          | 4/1410 [00:52<4:55:14, 12.60s/it]
  0%|          | 5/1410 [01:04<4:47:30, 12.28s/it]
                                                  
{'loss': 0.0286, 'grad_norm': 2.076698064804077, 'learning_rate': 1.0465116279069767e-07, 'kl': -0.0012, 'entropy': -0.0593, 'ce_loss': 0.027, 'epoch': 0.01}

  0%|          | 5/1410 [01:04<4:47:30, 12.28s/it]
  0%|          | 6/1410 [01:16<4:44:40, 12.17s/it]
                                                  
{'loss': 0.0195, 'grad_norm': 2.022712230682373, 'learning_rate': 1.255813953488372e-07, 'kl': -0.0104, 'entropy': 0.1348, 'ce_loss': 0.152, 'epoch': 0.01}

  0%|          | 6/1410 [01:16<4:44:40, 12.17s/it]
  0%|          | 7/1410 [01:27<4:41:14, 12.03s/it]
                                                  
{'loss': 0.0412, 'grad_norm': 2.8324785232543945, 'learning_rate': 1.4651162790697676e-07, 'kl': -0.0045, 'entropy': -0.063, 'ce_loss': 0.0468, 'epoch': 0.01}

  0%|          | 7/1410 [01:27<4:41:14, 12.03s/it]
  1%|          | 8/1410 [01:39<4:40:29, 12.00s/it]
                                                  
{'loss': 0.0128, 'grad_norm': 1.1771903038024902, 'learning_rate': 1.6744186046511627e-07, 'kl': -0.0067, 'entropy': 0.0442, 'ce_loss': 0.0913, 'epoch': 0.01}

  1%|          | 8/1410 [01:39<4:40:29, 12.00s/it]
  1%|          | 9/1410 [01:51<4:39:01, 11.95s/it]
                                                  
{'loss': 0.0167, 'grad_norm': 1.5746617317199707, 'learning_rate': 1.8837209302325583e-07, 'kl': -0.0057, 'entropy': -0.04, 'ce_loss': 0.0932, 'epoch': 0.01}

  1%|          | 9/1410 [01:51<4:39:01, 11.95s/it]
  1%|          | 10/1410 [02:03<4:37:27, 11.89s/it]
                                                   
{'loss': 0.0193, 'grad_norm': 1.2631957530975342, 'learning_rate': 2.0930232558139533e-07, 'kl': -0.0043, 'entropy': -0.0432, 'ce_loss': 0.0471, 'epoch': 0.01}

  1%|          | 10/1410 [02:03<4:37:27, 11.89s/it]
  1%|          | 11/1410 [02:14<4:29:16, 11.55s/it]
                                                   
{'loss': 0.0347, 'grad_norm': 2.740506649017334, 'learning_rate': 2.302325581395349e-07, 'kl': -0.0045, 'entropy': -0.0417, 'ce_loss': 0.0158, 'epoch': 0.02}

  1%|          | 11/1410 [02:14<4:29:16, 11.55s/it]
  1%|          | 12/1410 [02:26<4:32:43, 11.70s/it]
                                                   
{'loss': 0.0333, 'grad_norm': 1.913511037826538, 'learning_rate': 2.511627906976744e-07, 'kl': -0.0032, 'entropy': -0.0535, 'ce_loss': 0.0097, 'epoch': 0.02}

  1%|          | 12/1410 [02:26<4:32:43, 11.70s/it]
  1%|          | 13/1410 [02:37<4:29:03, 11.56s/it]
                                                   
{'loss': 0.0223, 'grad_norm': 1.7267472743988037, 'learning_rate': 2.720930232558139e-07, 'kl': -0.0047, 'entropy': -0.0405, 'ce_loss': 0.0271, 'epoch': 0.02}

  1%|          | 13/1410 [02:37<4:29:03, 11.56s/it]
  1%|          | 14/1410 [02:49<4:30:50, 11.64s/it]
                                                   
{'loss': 0.0148, 'grad_norm': 1.6506633758544922, 'learning_rate': 2.930232558139535e-07, 'kl': -0.0, 'entropy': -0.0139, 'ce_loss': 0.0008, 'epoch': 0.02}

  1%|          | 14/1410 [02:49<4:30:50, 11.64s/it]
  1%|          | 15/1410 [03:01<4:33:08, 11.75s/it]
                                                   
{'loss': 0.0281, 'grad_norm': 1.779132604598999, 'learning_rate': 3.13953488372093e-07, 'kl': -0.0021, 'entropy': -0.024, 'ce_loss': 0.0091, 'epoch': 0.02}

  1%|          | 15/1410 [03:01<4:33:08, 11.75s/it]
  1%|          | 16/1410 [03:13<4:35:22, 11.85s/it]
                                                   
{'loss': 0.0293, 'grad_norm': 1.8786593675613403, 'learning_rate': 3.3488372093023253e-07, 'kl': -0.0096, 'entropy': 0.0249, 'ce_loss': 0.0667, 'epoch': 0.02}

  1%|          | 16/1410 [03:13<4:35:22, 11.85s/it]
  1%|          | 17/1410 [03:25<4:34:44, 11.83s/it]
                                                   
{'loss': 0.0287, 'grad_norm': 2.4448020458221436, 'learning_rate': 3.5581395348837204e-07, 'kl': 0.0003, 'entropy': -0.0181, 'ce_loss': 0.0007, 'epoch': 0.02}

  1%|          | 17/1410 [03:25<4:34:44, 11.83s/it]
  1%|▏         | 18/1410 [03:37<4:35:22, 11.87s/it]
                                                   
{'loss': 0.0238, 'grad_norm': 2.0132672786712646, 'learning_rate': 3.7674418604651165e-07, 'kl': -0.0064, 'entropy': 0.0144, 'ce_loss': 0.1112, 'epoch': 0.03}

  1%|▏         | 18/1410 [03:37<4:35:22, 11.87s/it]
  1%|▏         | 19/1410 [03:49<4:35:37, 11.89s/it]
                                                   
{'loss': 0.0298, 'grad_norm': 2.300950050354004, 'learning_rate': 3.9767441860465116e-07, 'kl': -0.008, 'entropy': -0.042, 'ce_loss': 0.0816, 'epoch': 0.03}

  1%|▏         | 19/1410 [03:49<4:35:37, 11.89s/it]
  1%|▏         | 20/1410 [04:01<4:37:17, 11.97s/it]
                                                   
{'loss': 0.0575, 'grad_norm': 2.5422427654266357, 'learning_rate': 4.1860465116279067e-07, 'kl': -0.0032, 'entropy': -0.0586, 'ce_loss': 0.0219, 'epoch': 0.03}

  1%|▏         | 20/1410 [04:01<4:37:17, 11.97s/it]
  1%|▏         | 21/1410 [04:13<4:36:21, 11.94s/it]
                                                   
{'loss': 0.0225, 'grad_norm': 1.9197542667388916, 'learning_rate': 4.3953488372093017e-07, 'kl': -0.0089, 'entropy': 0.032, 'ce_loss': 0.1219, 'epoch': 0.03}

  1%|▏         | 21/1410 [04:13<4:36:21, 11.94s/it]
  2%|▏         | 22/1410 [04:25<4:36:50, 11.97s/it]
                                                   
{'loss': 0.0146, 'grad_norm': 1.4824256896972656, 'learning_rate': 4.604651162790698e-07, 'kl': -0.0121, 'entropy': 0.1504, 'ce_loss': 0.1291, 'epoch': 0.03}

  2%|▏         | 22/1410 [04:25<4:36:50, 11.97s/it]
  2%|▏         | 23/1410 [04:36<4:34:49, 11.89s/it]
                                                   
{'loss': 0.0227, 'grad_norm': 2.549474000930786, 'learning_rate': 4.813953488372092e-07, 'kl': -0.0017, 'entropy': -0.0359, 'ce_loss': 0.0185, 'epoch': 0.03}

  2%|▏         | 23/1410 [04:36<4:34:49, 11.89s/it]
  2%|▏         | 24/1410 [04:47<4:28:06, 11.61s/it]
                                                   
{'loss': 0.033, 'grad_norm': 2.35790753364563, 'learning_rate': 5.023255813953488e-07, 'kl': -0.0084, 'entropy': -0.0311, 'ce_loss': 0.0727, 'epoch': 0.03}

  2%|▏         | 24/1410 [04:47<4:28:06, 11.61s/it]
  2%|▏         | 25/1410 [04:59<4:29:54, 11.69s/it]
                                                   
{'loss': 0.0295, 'grad_norm': 1.8562809228897095, 'learning_rate': 5.232558139534884e-07, 'kl': 0.0001, 'entropy': -0.0403, 'ce_loss': 0.0077, 'epoch': 0.04}

  2%|▏         | 25/1410 [04:59<4:29:54, 11.69s/it]
  2%|▏         | 26/1410 [05:11<4:32:27, 11.81s/it]
                                                   
{'loss': 0.0226, 'grad_norm': 1.7917625904083252, 'learning_rate': 5.441860465116278e-07, 'kl': -0.0042, 'entropy': 0.1562, 'ce_loss': 0.138, 'epoch': 0.04}

  2%|▏         | 26/1410 [05:11<4:32:27, 11.81s/it]
  2%|▏         | 27/1410 [05:23<4:33:54, 11.88s/it]
                                                   
{'loss': 0.0285, 'grad_norm': 1.981705904006958, 'learning_rate': 5.651162790697674e-07, 'kl': -0.0026, 'entropy': -0.0454, 'ce_loss': 0.0483, 'epoch': 0.04}

  2%|▏         | 27/1410 [05:23<4:33:54, 11.88s/it]
  2%|▏         | 28/1410 [05:35<4:34:50, 11.93s/it]
                                                   
{'loss': 0.0162, 'grad_norm': 1.2059988975524902, 'learning_rate': 5.86046511627907e-07, 'kl': -0.0042, 'entropy': -0.0613, 'ce_loss': 0.0122, 'epoch': 0.04}

  2%|▏         | 28/1410 [05:35<4:34:50, 11.93s/it]
  2%|▏         | 29/1410 [05:47<4:34:56, 11.95s/it]
                                                   
{'loss': 0.0456, 'grad_norm': 2.7936789989471436, 'learning_rate': 6.069767441860464e-07, 'kl': -0.0, 'entropy': -0.0273, 'ce_loss': 0.0002, 'epoch': 0.04}

  2%|▏         | 29/1410 [05:47<4:34:56, 11.95s/it]
  2%|▏         | 30/1410 [05:59<4:34:50, 11.95s/it]
                                                   
{'loss': 0.0179, 'grad_norm': 1.4028782844543457, 'learning_rate': 6.27906976744186e-07, 'kl': -0.0009, 'entropy': -0.0442, 'ce_loss': 0.0137, 'epoch': 0.04}

  2%|▏         | 30/1410 [05:59<4:34:50, 11.95s/it]
  2%|▏         | 31/1410 [06:11<4:34:37, 11.95s/it]
                                                   
{'loss': 0.0215, 'grad_norm': 2.052412271499634, 'learning_rate': 6.488372093023255e-07, 'kl': -0.0057, 'entropy': 0.1748, 'ce_loss': 0.2086, 'epoch': 0.04}

  2%|▏         | 31/1410 [06:11<4:34:37, 11.95s/it]
  2%|▏         | 32/1410 [06:23<4:34:26, 11.95s/it]
                                                   
{'loss': 0.0276, 'grad_norm': 1.994560956954956, 'learning_rate': 6.697674418604651e-07, 'kl': -0.0005, 'entropy': -0.0444, 'ce_loss': 0.011, 'epoch': 0.05}

  2%|▏         | 32/1410 [06:23<4:34:26, 11.95s/it]
  2%|▏         | 33/1410 [06:35<4:33:44, 11.93s/it]
                                                   
{'loss': 0.0386, 'grad_norm': 2.845829963684082, 'learning_rate': 6.906976744186047e-07, 'kl': -0.0103, 'entropy': -0.0635, 'ce_loss': 0.0355, 'epoch': 0.05}

  2%|▏         | 33/1410 [06:35<4:33:44, 11.93s/it]
  2%|▏         | 34/1410 [06:47<4:34:36, 11.97s/it]
                                                   
{'loss': 0.0314, 'grad_norm': 1.9791022539138794, 'learning_rate': 7.116279069767441e-07, 'kl': -0.0071, 'entropy': 0.0947, 'ce_loss': 0.0725, 'epoch': 0.05}

  2%|▏         | 34/1410 [06:47<4:34:36, 11.97s/it]
  2%|▏         | 35/1410 [06:59<4:33:45, 11.95s/it]
                                                   
{'loss': 0.0315, 'grad_norm': 1.8979451656341553, 'learning_rate': 7.325581395348837e-07, 'kl': -0.0069, 'entropy': -0.053, 'ce_loss': 0.0193, 'epoch': 0.05}

  2%|▏         | 35/1410 [06:59<4:33:45, 11.95s/it]
  3%|▎         | 36/1410 [07:11<4:32:15, 11.89s/it]
                                                   
{'loss': 0.0386, 'grad_norm': 2.7001023292541504, 'learning_rate': 7.534883720930233e-07, 'kl': -0.0051, 'entropy': -0.0361, 'ce_loss': 0.0274, 'epoch': 0.05}

  3%|▎         | 36/1410 [07:11<4:32:15, 11.89s/it]
  3%|▎         | 37/1410 [07:22<4:29:01, 11.76s/it]
                                                   
{'loss': 0.0203, 'grad_norm': 1.9392054080963135, 'learning_rate': 7.744186046511627e-07, 'kl': -0.0021, 'entropy': -0.0728, 'ce_loss': 0.0237, 'epoch': 0.05}

  3%|▎         | 37/1410 [07:22<4:29:01, 11.76s/it]
  3%|▎         | 38/1410 [07:34<4:29:42, 11.79s/it]
                                                   
{'loss': 0.0241, 'grad_norm': 1.7863115072250366, 'learning_rate': 7.953488372093023e-07, 'kl': -0.0046, 'entropy': 0.0364, 'ce_loss': 0.0678, 'epoch': 0.05}

  3%|▎         | 38/1410 [07:34<4:29:42, 11.79s/it]
  3%|▎         | 39/1410 [07:46<4:30:13, 11.83s/it]
                                                   
{'loss': 0.024, 'grad_norm': 1.7691344022750854, 'learning_rate': 8.162790697674418e-07, 'kl': 0.0021, 'entropy': -0.0625, 'ce_loss': 0.0381, 'epoch': 0.06}

  3%|▎         | 39/1410 [07:46<4:30:13, 11.83s/it]
  3%|▎         | 40/1410 [07:58<4:28:03, 11.74s/it]
                                                   
{'loss': 0.0202, 'grad_norm': 2.11460018157959, 'learning_rate': 8.372093023255813e-07, 'kl': -0.0088, 'entropy': 0.0532, 'ce_loss': 0.1353, 'epoch': 0.06}

  3%|▎         | 40/1410 [07:58<4:28:03, 11.74s/it]
  3%|▎         | 41/1410 [08:10<4:28:59, 11.79s/it]
                                                   
{'loss': 0.0284, 'grad_norm': 2.125370740890503, 'learning_rate': 8.581395348837209e-07, 'kl': 0.0012, 'entropy': -0.0352, 'ce_loss': 0.0254, 'epoch': 0.06}

  3%|▎         | 41/1410 [08:10<4:28:59, 11.79s/it]
  3%|▎         | 42/1410 [08:21<4:29:02, 11.80s/it]
                                                   
{'loss': 0.0172, 'grad_norm': 1.6003546714782715, 'learning_rate': 8.790697674418603e-07, 'kl': -0.0045, 'entropy': -0.0244, 'ce_loss': 0.015, 'epoch': 0.06}

  3%|▎         | 42/1410 [08:21<4:29:02, 11.80s/it]
  3%|▎         | 43/1410 [08:33<4:26:39, 11.70s/it]
                                                   
{'loss': 0.029, 'grad_norm': 2.454036235809326, 'learning_rate': 9e-07, 'kl': -0.0042, 'entropy': -0.053, 'ce_loss': 0.0593, 'epoch': 0.06}

  3%|▎         | 43/1410 [08:33<4:26:39, 11.70s/it]
  3%|▎         | 44/1410 [08:45<4:26:54, 11.72s/it]
                                                   
{'loss': 0.0164, 'grad_norm': 1.6958414316177368, 'learning_rate': 8.999988116481592e-07, 'kl': -0.0001, 'entropy': -0.0067, 'ce_loss': 0.0022, 'epoch': 0.06}

  3%|▎         | 44/1410 [08:45<4:26:54, 11.72s/it]
  3%|▎         | 45/1410 [08:57<4:28:53, 11.82s/it]
                                                   
{'loss': 0.0436, 'grad_norm': 2.3342905044555664, 'learning_rate': 8.999952465989135e-07, 'kl': -0.0008, 'entropy': -0.0214, 'ce_loss': 0.0045, 'epoch': 0.06}

  3%|▎         | 45/1410 [08:57<4:28:53, 11.82s/it]
  3%|▎         | 46/1410 [09:09<4:30:46, 11.91s/it]
                                                   
{'loss': 0.0286, 'grad_norm': 1.9590696096420288, 'learning_rate': 8.999893048710917e-07, 'kl': 0.0006, 'entropy': -0.022, 'ce_loss': 0.0009, 'epoch': 0.07}

  3%|▎         | 46/1410 [09:09<4:30:46, 11.91s/it]
  3%|▎         | 47/1410 [09:20<4:26:34, 11.73s/it]
                                                   
{'loss': 0.0337, 'grad_norm': 2.6012158393859863, 'learning_rate': 8.999809864960755e-07, 'kl': -0.0025, 'entropy': -0.042, 'ce_loss': 0.0196, 'epoch': 0.07}

  3%|▎         | 47/1410 [09:20<4:26:34, 11.73s/it]
  3%|▎         | 48/1410 [09:32<4:25:47, 11.71s/it]
                                                   
{'loss': 0.0217, 'grad_norm': 1.7618882656097412, 'learning_rate': 8.999702915177988e-07, 'kl': -0.0021, 'entropy': -0.084, 'ce_loss': 0.0163, 'epoch': 0.07}

  3%|▎         | 48/1410 [09:32<4:25:47, 11.71s/it]
  3%|▎         | 49/1410 [09:44<4:27:47, 11.81s/it]
                                                   
{'loss': 0.0197, 'grad_norm': 1.7799535989761353, 'learning_rate': 8.999572199927481e-07, 'kl': -0.0107, 'entropy': -0.001, 'ce_loss': 0.0587, 'epoch': 0.07}

  3%|▎         | 49/1410 [09:44<4:27:47, 11.81s/it]
  4%|▎         | 50/1410 [09:56<4:27:44, 11.81s/it]
                                                   
{'loss': 0.0438, 'grad_norm': 5.00831937789917, 'learning_rate': 8.999417719899611e-07, 'kl': -0.0042, 'entropy': -0.0266, 'ce_loss': 0.0921, 'epoch': 0.07}

  4%|▎         | 50/1410 [09:56<4:27:44, 11.81s/it]
  4%|▎         | 51/1410 [10:07<4:27:34, 11.81s/it]
                                                   
{'loss': 0.0286, 'grad_norm': 1.7004528045654297, 'learning_rate': 8.999239475910279e-07, 'kl': 0.0058, 'entropy': -0.0535, 'ce_loss': 0.0152, 'epoch': 0.07}

  4%|▎         | 51/1410 [10:07<4:27:34, 11.81s/it]
  4%|▎         | 52/1410 [10:19<4:28:24, 11.86s/it]
                                                   
{'loss': 0.0233, 'grad_norm': 1.9457582235336304, 'learning_rate': 8.999037468900887e-07, 'kl': -0.0, 'entropy': -0.0154, 'ce_loss': 0.0122, 'epoch': 0.07}

  4%|▎         | 52/1410 [10:19<4:28:24, 11.86s/it]
  4%|▍         | 53/1410 [10:31<4:28:51, 11.89s/it]
                                                   
{'loss': 0.0413, 'grad_norm': 1.8673481941223145, 'learning_rate': 8.998811699938352e-07, 'kl': -0.0014, 'entropy': -0.0684, 'ce_loss': 0.0263, 'epoch': 0.08}

  4%|▍         | 53/1410 [10:31<4:28:51, 11.89s/it]
  4%|▍         | 54/1410 [10:43<4:28:58, 11.90s/it]
                                                   
{'loss': 0.0351, 'grad_norm': 2.0347328186035156, 'learning_rate': 8.998562170215083e-07, 'kl': -0.0069, 'entropy': -0.0013, 'ce_loss': 0.1155, 'epoch': 0.08}

  4%|▍         | 54/1410 [10:43<4:28:58, 11.90s/it]
  4%|▍         | 55/1410 [10:55<4:28:45, 11.90s/it]
                                                   
{'loss': 0.0279, 'grad_norm': 2.073336362838745, 'learning_rate': 8.998288881048991e-07, 'kl': 0.0016, 'entropy': -0.0154, 'ce_loss': 0.0248, 'epoch': 0.08}

  4%|▍         | 55/1410 [10:55<4:28:45, 11.90s/it]
  4%|▍         | 56/1410 [11:07<4:25:11, 11.75s/it]
                                                   
{'loss': 0.0236, 'grad_norm': 2.5604512691497803, 'learning_rate': 8.997991833883469e-07, 'kl': -0.0057, 'entropy': -0.0889, 'ce_loss': 0.0346, 'epoch': 0.08}

  4%|▍         | 56/1410 [11:07<4:25:11, 11.75s/it]
  4%|▍         | 57/1410 [11:17<4:18:13, 11.45s/it]
                                                   
{'loss': 0.0516, 'grad_norm': 2.835190773010254, 'learning_rate': 8.99767103028739e-07, 'kl': -0.0003, 'entropy': -0.0237, 'ce_loss': 0.0009, 'epoch': 0.08}

  4%|▍         | 57/1410 [11:17<4:18:13, 11.45s/it]
  4%|▍         | 58/1410 [11:29<4:21:03, 11.59s/it]
                                                   
{'loss': 0.0422, 'grad_norm': 1.8689215183258057, 'learning_rate': 8.997326471955098e-07, 'kl': 0.0053, 'entropy': -0.0449, 'ce_loss': 0.0307, 'epoch': 0.08}

  4%|▍         | 58/1410 [11:29<4:21:03, 11.59s/it]
  4%|▍         | 59/1410 [11:41<4:23:03, 11.68s/it]
                                                   
{'loss': 0.0307, 'grad_norm': 2.21761155128479, 'learning_rate': 8.9969581607064e-07, 'kl': -0.0039, 'entropy': 0.0171, 'ce_loss': 0.1116, 'epoch': 0.08}

  4%|▍         | 59/1410 [11:41<4:23:03, 11.68s/it]
  4%|▍         | 60/1410 [11:53<4:23:29, 11.71s/it]
                                                   
{'loss': 0.0238, 'grad_norm': 2.3317489624023438, 'learning_rate': 8.996566098486556e-07, 'kl': 0.0063, 'entropy': -0.0679, 'ce_loss': 0.0184, 'epoch': 0.09}

  4%|▍         | 60/1410 [11:53<4:23:29, 11.71s/it]
  4%|▍         | 61/1410 [12:05<4:24:41, 11.77s/it]
                                                   
{'loss': 0.0204, 'grad_norm': 4.185949802398682, 'learning_rate': 8.996150287366269e-07, 'kl': -0.0046, 'entropy': -0.0427, 'ce_loss': 0.1004, 'epoch': 0.09}

  4%|▍         | 61/1410 [12:05<4:24:41, 11.77s/it]
  4%|▍         | 62/1410 [12:17<4:25:18, 11.81s/it]
                                                   
{'loss': 0.0106, 'grad_norm': 1.2724953889846802, 'learning_rate': 8.995710729541671e-07, 'kl': -0.0071, 'entropy': 0.0259, 'ce_loss': 0.0721, 'epoch': 0.09}

  4%|▍         | 62/1410 [12:17<4:25:18, 11.81s/it]
  4%|▍         | 63/1410 [12:28<4:18:08, 11.50s/it]
                                                   
{'loss': 0.0157, 'grad_norm': 1.7939341068267822, 'learning_rate': 8.995247427334312e-07, 'kl': 0.0005, 'entropy': -0.0344, 'ce_loss': 0.0139, 'epoch': 0.09}

  4%|▍         | 63/1410 [12:28<4:18:08, 11.50s/it]
  5%|▍         | 64/1410 [12:40<4:21:24, 11.65s/it]
                                                   
{'loss': 0.0333, 'grad_norm': 1.9352843761444092, 'learning_rate': 8.994760383191154e-07, 'kl': 0.0004, 'entropy': -0.0356, 'ce_loss': 0.0354, 'epoch': 0.09}

  5%|▍         | 64/1410 [12:40<4:21:24, 11.65s/it]
  5%|▍         | 65/1410 [12:51<4:22:44, 11.72s/it]
                                                   
{'loss': 0.0343, 'grad_norm': 2.0232486724853516, 'learning_rate': 8.994249599684553e-07, 'kl': 0.0008, 'entropy': -0.0481, 'ce_loss': 0.0356, 'epoch': 0.09}

  5%|▍         | 65/1410 [12:51<4:22:44, 11.72s/it]
  5%|▍         | 66/1410 [13:03<4:20:17, 11.62s/it]
                                                   
{'loss': 0.0255, 'grad_norm': 2.1877570152282715, 'learning_rate': 8.993715079512245e-07, 'kl': -0.0022, 'entropy': -0.051, 'ce_loss': 0.0589, 'epoch': 0.09}

  5%|▍         | 66/1410 [13:03<4:20:17, 11.62s/it]
  5%|▍         | 67/1410 [13:15<4:22:42, 11.74s/it]
                                                   
{'loss': 0.0418, 'grad_norm': 2.4774906635284424, 'learning_rate': 8.993156825497328e-07, 'kl': 0.0049, 'entropy': -0.031, 'ce_loss': 0.0021, 'epoch': 0.1}

  5%|▍         | 67/1410 [13:15<4:22:42, 11.74s/it]
  5%|▍         | 68/1410 [13:27<4:22:13, 11.72s/it]
                                                   
{'loss': 0.0176, 'grad_norm': 1.9139611721038818, 'learning_rate': 8.992574840588261e-07, 'kl': 0.0063, 'entropy': -0.0488, 'ce_loss': 0.0119, 'epoch': 0.1}

  5%|▍         | 68/1410 [13:27<4:22:13, 11.72s/it]
  5%|▍         | 69/1410 [13:39<4:24:16, 11.82s/it]
                                                   
{'loss': 0.0434, 'grad_norm': 2.2980456352233887, 'learning_rate': 8.991969127858831e-07, 'kl': -0.0014, 'entropy': -0.0522, 'ce_loss': 0.0226, 'epoch': 0.1}

  5%|▍         | 69/1410 [13:39<4:24:16, 11.82s/it]
  5%|▍         | 70/1410 [13:50<4:23:47, 11.81s/it]
                                                   
{'loss': 0.0175, 'grad_norm': 2.189495801925659, 'learning_rate': 8.991339690508153e-07, 'kl': 0.0026, 'entropy': -0.0466, 'ce_loss': 0.0077, 'epoch': 0.1}

  5%|▍         | 70/1410 [13:50<4:23:47, 11.81s/it]
  5%|▌         | 71/1410 [14:02<4:23:08, 11.79s/it]
                                                   
{'loss': 0.0261, 'grad_norm': 2.2474820613861084, 'learning_rate': 8.990686531860635e-07, 'kl': -0.0007, 'entropy': -0.0245, 'ce_loss': 0.0416, 'epoch': 0.1}

  5%|▌         | 71/1410 [14:02<4:23:08, 11.79s/it]
  5%|▌         | 72/1410 [14:14<4:23:37, 11.82s/it]
                                                   
{'loss': 0.0437, 'grad_norm': 2.0656344890594482, 'learning_rate': 8.990009655365978e-07, 'kl': -0.0008, 'entropy': -0.04, 'ce_loss': 0.0179, 'epoch': 0.1}

  5%|▌         | 72/1410 [14:14<4:23:37, 11.82s/it]
  5%|▌         | 73/1410 [14:26<4:24:24, 11.87s/it]
                                                   
{'loss': 0.0315, 'grad_norm': 1.9964438676834106, 'learning_rate': 8.989309064599148e-07, 'kl': 0.0035, 'entropy': -0.017, 'ce_loss': 0.0171, 'epoch': 0.1}

  5%|▌         | 73/1410 [14:26<4:24:24, 11.87s/it]
  5%|▌         | 74/1410 [14:38<4:25:47, 11.94s/it]
                                                   
{'loss': 0.0496, 'grad_norm': 2.1579558849334717, 'learning_rate': 8.988584763260362e-07, 'kl': -0.0075, 'entropy': 0.0981, 'ce_loss': 0.1423, 'epoch': 0.1}

  5%|▌         | 74/1410 [14:38<4:25:47, 11.94s/it]
  5%|▌         | 75/1410 [14:49<4:17:50, 11.59s/it]
                                                   
{'loss': 0.0401, 'grad_norm': 3.224050283432007, 'learning_rate': 8.987836755175063e-07, 'kl': 0.0005, 'entropy': -0.0771, 'ce_loss': 0.0347, 'epoch': 0.11}

  5%|▌         | 75/1410 [14:49<4:17:50, 11.59s/it]
  5%|▌         | 76/1410 [14:59<4:11:25, 11.31s/it]
                                                   
{'loss': 0.0262, 'grad_norm': 2.450028657913208, 'learning_rate': 8.987065044293903e-07, 'kl': 0.0009, 'entropy': -0.0017, 'ce_loss': 0.0171, 'epoch': 0.11}

  5%|▌         | 76/1410 [14:59<4:11:25, 11.31s/it]
  5%|▌         | 77/1410 [15:11<4:14:26, 11.45s/it]
                                                   
{'loss': 0.0153, 'grad_norm': 1.4712834358215332, 'learning_rate': 8.98626963469272e-07, 'kl': -0.002, 'entropy': -0.0649, 'ce_loss': 0.0232, 'epoch': 0.11}

  5%|▌         | 77/1410 [15:11<4:14:26, 11.45s/it]
  6%|▌         | 78/1410 [15:22<4:11:54, 11.35s/it]
                                                   
{'loss': 0.025, 'grad_norm': 2.0323119163513184, 'learning_rate': 8.985450530572522e-07, 'kl': 0.0045, 'entropy': -0.0796, 'ce_loss': 0.0278, 'epoch': 0.11}

  6%|▌         | 78/1410 [15:22<4:11:54, 11.35s/it]
  6%|▌         | 79/1410 [15:34<4:15:57, 11.54s/it]
                                                   
{'loss': 0.0272, 'grad_norm': 2.002166986465454, 'learning_rate': 8.984607736259462e-07, 'kl': -0.0065, 'entropy': 0.0393, 'ce_loss': 0.1164, 'epoch': 0.11}

  6%|▌         | 79/1410 [15:34<4:15:57, 11.54s/it]
  6%|▌         | 80/1410 [15:46<4:14:09, 11.47s/it]
                                                   
{'loss': 0.0279, 'grad_norm': 2.501282215118408, 'learning_rate': 8.983741256204807e-07, 'kl': -0.001, 'entropy': -0.084, 'ce_loss': 0.0279, 'epoch': 0.11}

  6%|▌         | 80/1410 [15:46<4:14:09, 11.47s/it]
  6%|▌         | 81/1410 [15:58<4:18:28, 11.67s/it]
                                                   
{'loss': 0.0371, 'grad_norm': 1.9937599897384644, 'learning_rate': 8.982851094984932e-07, 'kl': 0.0064, 'entropy': -0.0181, 'ce_loss': 0.0033, 'epoch': 0.11}

  6%|▌         | 81/1410 [15:58<4:18:28, 11.67s/it]
  6%|▌         | 82/1410 [16:09<4:18:08, 11.66s/it]
                                                   
{'loss': 0.0145, 'grad_norm': 1.8456424474716187, 'learning_rate': 8.981937257301276e-07, 'kl': 0.0009, 'entropy': -0.0161, 'ce_loss': 0.0007, 'epoch': 0.12}

  6%|▌         | 82/1410 [16:09<4:18:08, 11.66s/it]
  6%|▌         | 83/1410 [16:20<4:12:03, 11.40s/it]
                                                   
{'loss': 0.0252, 'grad_norm': 2.2807226181030273, 'learning_rate': 8.980999747980334e-07, 'kl': 0.0027, 'entropy': -0.0393, 'ce_loss': 0.0243, 'epoch': 0.12}

  6%|▌         | 83/1410 [16:20<4:12:03, 11.40s/it]
  6%|▌         | 84/1410 [16:32<4:13:45, 11.48s/it]
                                                   
{'loss': 0.015, 'grad_norm': 2.022230386734009, 'learning_rate': 8.980038571973618e-07, 'kl': -0.0074, 'entropy': 0.0334, 'ce_loss': 0.1142, 'epoch': 0.12}

  6%|▌         | 84/1410 [16:32<4:13:45, 11.48s/it]
  6%|▌         | 85/1410 [16:44<4:16:40, 11.62s/it]
                                                   
{'loss': 0.0542, 'grad_norm': 2.7313082218170166, 'learning_rate': 8.979053734357644e-07, 'kl': 0.0022, 'entropy': -0.0403, 'ce_loss': 0.0078, 'epoch': 0.12}

  6%|▌         | 85/1410 [16:44<4:16:40, 11.62s/it]
  6%|▌         | 86/1410 [16:56<4:18:08, 11.70s/it]
                                                   
{'loss': 0.026, 'grad_norm': 2.115839958190918, 'learning_rate': 8.978045240333891e-07, 'kl': -0.0024, 'entropy': -0.0308, 'ce_loss': 0.0157, 'epoch': 0.12}

  6%|▌         | 86/1410 [16:56<4:18:08, 11.70s/it]
  6%|▌         | 87/1410 [17:07<4:18:22, 11.72s/it]
                                                   
{'loss': 0.0198, 'grad_norm': 1.8684135675430298, 'learning_rate': 8.977013095228788e-07, 'kl': -0.0033, 'entropy': -0.0269, 'ce_loss': 0.0184, 'epoch': 0.12}

  6%|▌         | 87/1410 [17:07<4:18:22, 11.72s/it]
  6%|▌         | 88/1410 [17:19<4:18:40, 11.74s/it]
                                                   
{'loss': 0.0442, 'grad_norm': 2.8882293701171875, 'learning_rate': 8.975957304493672e-07, 'kl': 0.0029, 'entropy': -0.0352, 'ce_loss': 0.0094, 'epoch': 0.12}

  6%|▌         | 88/1410 [17:19<4:18:40, 11.74s/it]
  6%|▋         | 89/1410 [17:31<4:18:16, 11.73s/it]
                                                   
{'loss': 0.0344, 'grad_norm': 2.717046022415161, 'learning_rate': 8.974877873704772e-07, 'kl': -0.0013, 'entropy': -0.0304, 'ce_loss': 0.008, 'epoch': 0.13}

  6%|▋         | 89/1410 [17:31<4:18:16, 11.73s/it]
  6%|▋         | 90/1410 [17:43<4:18:22, 11.74s/it]
                                                   
{'loss': 0.0371, 'grad_norm': 2.9057703018188477, 'learning_rate': 8.973774808563169e-07, 'kl': -0.0011, 'entropy': -0.0588, 'ce_loss': 0.0346, 'epoch': 0.13}

  6%|▋         | 90/1410 [17:43<4:18:22, 11.74s/it]
  6%|▋         | 91/1410 [17:55<4:18:57, 11.78s/it]
                                                   
{'loss': 0.0326, 'grad_norm': 2.3817310333251953, 'learning_rate': 8.972648114894771e-07, 'kl': 0.0074, 'entropy': -0.0493, 'ce_loss': 0.0334, 'epoch': 0.13}

  6%|▋         | 91/1410 [17:55<4:18:57, 11.78s/it]
  7%|▋         | 92/1410 [18:07<4:20:55, 11.88s/it]
                                                   
{'loss': 0.0435, 'grad_norm': 2.9468696117401123, 'learning_rate': 8.971497798650283e-07, 'kl': -0.0052, 'entropy': 0.2012, 'ce_loss': 0.2352, 'epoch': 0.13}

  7%|▋         | 92/1410 [18:07<4:20:55, 11.88s/it]
  7%|▋         | 93/1410 [18:19<4:22:17, 11.95s/it]
                                                   
{'loss': 0.0515, 'grad_norm': 2.3117260932922363, 'learning_rate': 8.970323865905175e-07, 'kl': -0.0002, 'entropy': -0.0381, 'ce_loss': 0.013, 'epoch': 0.13}

  7%|▋         | 93/1410 [18:19<4:22:17, 11.95s/it]
  7%|▋         | 94/1410 [18:31<4:21:06, 11.90s/it]
                                                   
{'loss': 0.0253, 'grad_norm': 1.9742525815963745, 'learning_rate': 8.969126322859645e-07, 'kl': -0.0061, 'entropy': -0.0811, 'ce_loss': 0.029, 'epoch': 0.13}

  7%|▋         | 94/1410 [18:31<4:21:06, 11.90s/it]
  7%|▋         | 95/1410 [18:42<4:20:20, 11.88s/it]
                                                   
{'loss': 0.022, 'grad_norm': 1.7628836631774902, 'learning_rate': 8.967905175838594e-07, 'kl': -0.0072, 'entropy': 0.0116, 'ce_loss': 0.057, 'epoch': 0.13}

  7%|▋         | 95/1410 [18:42<4:20:20, 11.88s/it]
  7%|▋         | 96/1410 [18:54<4:19:47, 11.86s/it]
                                                   
{'loss': 0.0189, 'grad_norm': 1.7540868520736694, 'learning_rate': 8.966660431291592e-07, 'kl': -0.0053, 'entropy': -0.0469, 'ce_loss': 0.0356, 'epoch': 0.14}

  7%|▋         | 96/1410 [18:54<4:19:47, 11.86s/it]
  7%|▋         | 97/1410 [19:06<4:20:26, 11.90s/it]
                                                   
{'loss': 0.043, 'grad_norm': 2.399646759033203, 'learning_rate': 8.96539209579283e-07, 'kl': -0.0071, 'entropy': -0.05, 'ce_loss': 0.0453, 'epoch': 0.14}

  7%|▋         | 97/1410 [19:06<4:20:26, 11.90s/it]
  7%|▋         | 98/1410 [19:18<4:19:05, 11.85s/it]
                                                   
{'loss': 0.014, 'grad_norm': 3.687840700149536, 'learning_rate': 8.964100176041106e-07, 'kl': -0.0005, 'entropy': -0.0422, 'ce_loss': 0.0258, 'epoch': 0.14}

  7%|▋         | 98/1410 [19:18<4:19:05, 11.85s/it]
  7%|▋         | 99/1410 [19:29<4:16:15, 11.73s/it]
                                                   
{'loss': 0.0279, 'grad_norm': 2.6390295028686523, 'learning_rate': 8.962784678859775e-07, 'kl': 0.0027, 'entropy': -0.064, 'ce_loss': 0.0274, 'epoch': 0.14}

  7%|▋         | 99/1410 [19:29<4:16:15, 11.73s/it]
  7%|▋         | 100/1410 [19:41<4:16:09, 11.73s/it]
                                                    
{'loss': 0.0157, 'grad_norm': 1.7360345125198364, 'learning_rate': 8.961445611196722e-07, 'kl': -0.0061, 'entropy': 0.0334, 'ce_loss': 0.0799, 'epoch': 0.14}

  7%|▋         | 100/1410 [19:41<4:16:09, 11.73s/it]
  7%|▋         | 101/1410 [19:53<4:16:30, 11.76s/it]
                                                    
{'loss': 0.0195, 'grad_norm': 2.2705891132354736, 'learning_rate': 8.960082980124316e-07, 'kl': -0.0054, 'entropy': -0.0095, 'ce_loss': 0.0903, 'epoch': 0.14}

  7%|▋         | 101/1410 [19:53<4:16:30, 11.76s/it]
  7%|▋         | 102/1410 [20:04<4:14:01, 11.65s/it]
                                                    
{'loss': 0.0267, 'grad_norm': 1.8619930744171143, 'learning_rate': 8.95869679283938e-07, 'kl': -0.0013, 'entropy': -0.0427, 'ce_loss': 0.0222, 'epoch': 0.14}

  7%|▋         | 102/1410 [20:04<4:14:01, 11.65s/it]
  7%|▋         | 103/1410 [20:16<4:15:57, 11.75s/it]
                                                    
{'loss': 0.0253, 'grad_norm': 2.0307083129882812, 'learning_rate': 8.957287056663152e-07, 'kl': -0.0064, 'entropy': 0.0029, 'ce_loss': 0.0573, 'epoch': 0.15}

  7%|▋         | 103/1410 [20:16<4:15:57, 11.75s/it]
  7%|▋         | 104/1410 [20:28<4:12:23, 11.60s/it]
                                                    
{'loss': 0.0216, 'grad_norm': 1.9831293821334839, 'learning_rate': 8.955853779041242e-07, 'kl': -0.0069, 'entropy': -0.0674, 'ce_loss': 0.0389, 'epoch': 0.15}

  7%|▋         | 104/1410 [20:28<4:12:23, 11.60s/it]
  7%|▋         | 105/1410 [20:39<4:12:40, 11.62s/it]
                                                    
{'loss': 0.0347, 'grad_norm': 2.599536657333374, 'learning_rate': 8.954396967543596e-07, 'kl': 0.0049, 'entropy': -0.053, 'ce_loss': 0.0022, 'epoch': 0.15}

  7%|▋         | 105/1410 [20:39<4:12:40, 11.62s/it]
  8%|▊         | 106/1410 [20:51<4:15:28, 11.75s/it]
                                                    
{'loss': 0.0251, 'grad_norm': 1.947945475578308, 'learning_rate': 8.952916629864462e-07, 'kl': -0.0008, 'entropy': -0.0806, 'ce_loss': 0.0067, 'epoch': 0.15}

  8%|▊         | 106/1410 [20:51<4:15:28, 11.75s/it]
  8%|▊         | 107/1410 [21:03<4:14:02, 11.70s/it]
                                                    
{'loss': 0.0213, 'grad_norm': 1.7868080139160156, 'learning_rate': 8.951412773822331e-07, 'kl': 0.0052, 'entropy': -0.0187, 'ce_loss': 0.008, 'epoch': 0.15}

  8%|▊         | 107/1410 [21:03<4:14:02, 11.70s/it]
  8%|▊         | 108/1410 [21:15<4:14:53, 11.75s/it]
                                                    
{'loss': 0.0232, 'grad_norm': 1.8136327266693115, 'learning_rate': 8.949885407359918e-07, 'kl': -0.0059, 'entropy': 0.0253, 'ce_loss': 0.0638, 'epoch': 0.15}

  8%|▊         | 108/1410 [21:15<4:14:53, 11.75s/it]
  8%|▊         | 109/1410 [21:26<4:11:14, 11.59s/it]
                                                    
{'loss': 0.0319, 'grad_norm': 2.2980411052703857, 'learning_rate': 8.948334538544107e-07, 'kl': -0.0072, 'entropy': -0.001, 'ce_loss': 0.0661, 'epoch': 0.15}

  8%|▊         | 109/1410 [21:26<4:11:14, 11.59s/it]
  8%|▊         | 110/1410 [21:38<4:11:26, 11.61s/it]
                                                    
{'loss': 0.0278, 'grad_norm': 2.593278169631958, 'learning_rate': 8.946760175565909e-07, 'kl': 0.0002, 'entropy': -0.0227, 'ce_loss': 0.0226, 'epoch': 0.16}

  8%|▊         | 110/1410 [21:38<4:11:26, 11.61s/it]
  8%|▊         | 111/1410 [21:49<4:08:23, 11.47s/it]
                                                    
{'loss': 0.0193, 'grad_norm': 1.8690730333328247, 'learning_rate': 8.945162326740422e-07, 'kl': 0.0111, 'entropy': -0.0264, 'ce_loss': 0.0012, 'epoch': 0.16}

  8%|▊         | 111/1410 [21:49<4:08:23, 11.47s/it]
  8%|▊         | 112/1410 [22:00<4:06:38, 11.40s/it]
                                                    
{'loss': 0.0144, 'grad_norm': 1.5100630521774292, 'learning_rate': 8.943541000506788e-07, 'kl': -0.0017, 'entropy': 0.1147, 'ce_loss': 0.141, 'epoch': 0.16}

  8%|▊         | 112/1410 [22:00<4:06:38, 11.40s/it]
  8%|▊         | 113/1410 [22:12<4:09:02, 11.52s/it]
                                                    
{'loss': 0.0431, 'grad_norm': 1.8397074937820435, 'learning_rate': 8.941896205428144e-07, 'kl': -0.0042, 'entropy': -0.0334, 'ce_loss': 0.023, 'epoch': 0.16}

  8%|▊         | 113/1410 [22:12<4:09:02, 11.52s/it]
  8%|▊         | 114/1410 [22:24<4:11:52, 11.66s/it]
                                                    
{'loss': 0.0416, 'grad_norm': 1.6681710481643677, 'learning_rate': 8.940227950191581e-07, 'kl': -0.0036, 'entropy': 0.1152, 'ce_loss': 0.1873, 'epoch': 0.16}

  8%|▊         | 114/1410 [22:24<4:11:52, 11.66s/it]
  8%|▊         | 115/1410 [22:36<4:11:51, 11.67s/it]
                                                    
{'loss': 0.0298, 'grad_norm': 2.1397714614868164, 'learning_rate': 8.938536243608093e-07, 'kl': -0.0054, 'entropy': -0.04, 'ce_loss': 0.0454, 'epoch': 0.16}

  8%|▊         | 115/1410 [22:36<4:11:51, 11.67s/it]
  8%|▊         | 116/1410 [22:47<4:11:35, 11.67s/it]
                                                    
{'loss': 0.0166, 'grad_norm': 1.885011911392212, 'learning_rate': 8.936821094612539e-07, 'kl': 0.0073, 'entropy': -0.0864, 'ce_loss': 0.0126, 'epoch': 0.16}

  8%|▊         | 116/1410 [22:47<4:11:35, 11.67s/it]
  8%|▊         | 117/1410 [22:59<4:13:22, 11.76s/it]
                                                    
{'loss': 0.0592, 'grad_norm': 1.9950677156448364, 'learning_rate': 8.935082512263586e-07, 'kl': -0.0063, 'entropy': 0.1318, 'ce_loss': 0.1129, 'epoch': 0.17}

  8%|▊         | 117/1410 [22:59<4:13:22, 11.76s/it]
  8%|▊         | 118/1410 [23:11<4:14:26, 11.82s/it]
                                                    
{'loss': 0.0327, 'grad_norm': 2.5256237983703613, 'learning_rate': 8.933320505743668e-07, 'kl': -0.0032, 'entropy': -0.0474, 'ce_loss': 0.0279, 'epoch': 0.17}

  8%|▊         | 118/1410 [23:11<4:14:26, 11.82s/it]
  8%|▊         | 119/1410 [23:23<4:15:17, 11.87s/it]
                                                    
{'loss': 0.024, 'grad_norm': 1.5586438179016113, 'learning_rate': 8.931535084358934e-07, 'kl': -0.0044, 'entropy': -0.0649, 'ce_loss': 0.0389, 'epoch': 0.17}

  8%|▊         | 119/1410 [23:23<4:15:17, 11.87s/it]
  9%|▊         | 120/1410 [23:35<4:13:57, 11.81s/it]
                                                    
{'loss': 0.0231, 'grad_norm': 2.3735880851745605, 'learning_rate': 8.929726257539202e-07, 'kl': -0.0005, 'entropy': -0.0157, 'ce_loss': 0.0005, 'epoch': 0.17}

  9%|▊         | 120/1410 [23:35<4:13:57, 11.81s/it]
  9%|▊         | 121/1410 [23:47<4:13:45, 11.81s/it]
                                                    
{'loss': 0.027, 'grad_norm': 1.9805855751037598, 'learning_rate': 8.927894034837905e-07, 'kl': -0.0012, 'entropy': 0.0076, 'ce_loss': 0.0846, 'epoch': 0.17}

  9%|▊         | 121/1410 [23:47<4:13:45, 11.81s/it]
  9%|▊         | 122/1410 [23:58<4:08:25, 11.57s/it]
                                                    
{'loss': 0.0226, 'grad_norm': 1.731331706047058, 'learning_rate': 8.926038425932046e-07, 'kl': 0.0006, 'entropy': -0.0259, 'ce_loss': 0.0277, 'epoch': 0.17}

  9%|▊         | 122/1410 [23:58<4:08:25, 11.57s/it]
  9%|▊         | 123/1410 [24:09<4:07:30, 11.54s/it]
                                                    
{'loss': 0.0292, 'grad_norm': 2.5843825340270996, 'learning_rate': 8.924159440622139e-07, 'kl': 0.0051, 'entropy': -0.0437, 'ce_loss': 0.0327, 'epoch': 0.17}

  9%|▊         | 123/1410 [24:09<4:07:30, 11.54s/it]
  9%|▉         | 124/1410 [24:21<4:09:38, 11.65s/it]
                                                    
{'loss': 0.0394, 'grad_norm': 2.1005136966705322, 'learning_rate': 8.922257088832166e-07, 'kl': -0.001, 'entropy': -0.061, 'ce_loss': 0.0469, 'epoch': 0.18}

  9%|▉         | 124/1410 [24:21<4:09:38, 11.65s/it]
  9%|▉         | 125/1410 [24:33<4:09:48, 11.66s/it]
                                                    
{'loss': 0.0148, 'grad_norm': 1.4676662683486938, 'learning_rate': 8.92033138060952e-07, 'kl': 0.0057, 'entropy': -0.0122, 'ce_loss': 0.0011, 'epoch': 0.18}

  9%|▉         | 125/1410 [24:33<4:09:48, 11.66s/it]
  9%|▉         | 126/1410 [24:44<4:09:31, 11.66s/it]
                                                    
{'loss': 0.0213, 'grad_norm': 1.7294516563415527, 'learning_rate': 8.91838232612495e-07, 'kl': -0.0029, 'entropy': -0.0649, 'ce_loss': 0.0156, 'epoch': 0.18}

  9%|▉         | 126/1410 [24:44<4:09:31, 11.66s/it]
  9%|▉         | 127/1410 [24:56<4:08:41, 11.63s/it]
                                                    
{'loss': 0.0321, 'grad_norm': 2.875805377960205, 'learning_rate': 8.916409935672512e-07, 'kl': 0.0124, 'entropy': -0.0713, 'ce_loss': 0.048, 'epoch': 0.18}

  9%|▉         | 127/1410 [24:56<4:08:41, 11.63s/it]
  9%|▉         | 128/1410 [25:08<4:09:59, 11.70s/it]
                                                    
{'loss': 0.0288, 'grad_norm': 1.744226098060608, 'learning_rate': 8.914414219669514e-07, 'kl': -0.0089, 'entropy': -0.0703, 'ce_loss': 0.0563, 'epoch': 0.18}

  9%|▉         | 128/1410 [25:08<4:09:59, 11.70s/it]
  9%|▉         | 129/1410 [25:20<4:11:52, 11.80s/it]
                                                    
{'loss': 0.044, 'grad_norm': 2.118183135986328, 'learning_rate': 8.912395188656455e-07, 'kl': 0.0013, 'entropy': -0.0649, 'ce_loss': 0.0229, 'epoch': 0.18}

  9%|▉         | 129/1410 [25:20<4:11:52, 11.80s/it]
  9%|▉         | 130/1410 [25:31<4:07:06, 11.58s/it]
                                                    
{'loss': 0.0125, 'grad_norm': 1.2783715724945068, 'learning_rate': 8.910352853296976e-07, 'kl': 0.0025, 'entropy': -0.0742, 'ce_loss': 0.0068, 'epoch': 0.18}

  9%|▉         | 130/1410 [25:31<4:07:06, 11.58s/it]
  9%|▉         | 131/1410 [25:43<4:08:18, 11.65s/it]
                                                    
{'loss': 0.038, 'grad_norm': 2.526668071746826, 'learning_rate': 8.908287224377802e-07, 'kl': -0.0006, 'entropy': -0.0508, 'ce_loss': 0.0276, 'epoch': 0.19}

  9%|▉         | 131/1410 [25:43<4:08:18, 11.65s/it]
  9%|▉         | 132/1410 [25:55<4:09:33, 11.72s/it]
                                                    
{'loss': 0.0207, 'grad_norm': 1.7010676860809326, 'learning_rate': 8.906198312808684e-07, 'kl': 0.0038, 'entropy': -0.0664, 'ce_loss': 0.0133, 'epoch': 0.19}

  9%|▉         | 132/1410 [25:55<4:09:33, 11.72s/it]
  9%|▉         | 133/1410 [26:06<4:10:33, 11.77s/it]
                                                    
{'loss': 0.0333, 'grad_norm': 2.5807628631591797, 'learning_rate': 8.904086129622341e-07, 'kl': 0.002, 'entropy': -0.0138, 'ce_loss': 0.0001, 'epoch': 0.19}

  9%|▉         | 133/1410 [26:06<4:10:33, 11.77s/it]
 10%|▉         | 134/1410 [26:18<4:08:02, 11.66s/it]
                                                    
{'loss': 0.0342, 'grad_norm': 2.4233169555664062, 'learning_rate': 8.901950685974403e-07, 'kl': 0.0009, 'entropy': -0.0718, 'ce_loss': 0.0417, 'epoch': 0.19}

 10%|▉         | 134/1410 [26:18<4:08:02, 11.66s/it]
 10%|▉         | 135/1410 [26:30<4:08:19, 11.69s/it]
                                                    
{'loss': 0.0188, 'grad_norm': 2.005946159362793, 'learning_rate': 8.899791993143353e-07, 'kl': 0.0093, 'entropy': -0.0752, 'ce_loss': 0.0217, 'epoch': 0.19}

 10%|▉         | 135/1410 [26:30<4:08:19, 11.69s/it]
 10%|▉         | 136/1410 [26:41<4:08:51, 11.72s/it]
                                                    
{'loss': 0.0411, 'grad_norm': 3.0774271488189697, 'learning_rate': 8.897610062530463e-07, 'kl': 0.0125, 'entropy': -0.0781, 'ce_loss': 0.0366, 'epoch': 0.19}

 10%|▉         | 136/1410 [26:41<4:08:51, 11.72s/it]
 10%|▉         | 137/1410 [26:53<4:09:49, 11.77s/it]
                                                    
{'loss': 0.0125, 'grad_norm': 1.1994493007659912, 'learning_rate': 8.89540490565974e-07, 'kl': -0.0089, 'entropy': 0.0248, 'ce_loss': 0.0902, 'epoch': 0.19}

 10%|▉         | 137/1410 [26:53<4:09:49, 11.77s/it]
 10%|▉         | 138/1410 [27:05<4:09:54, 11.79s/it]
                                                    
{'loss': 0.0182, 'grad_norm': 1.5567814111709595, 'learning_rate': 8.89317653417786e-07, 'kl': -0.0055, 'entropy': 0.1162, 'ce_loss': 0.1162, 'epoch': 0.2}

 10%|▉         | 138/1410 [27:05<4:09:54, 11.79s/it]
 10%|▉         | 139/1410 [27:17<4:10:43, 11.84s/it]
                                                    
{'loss': 0.028, 'grad_norm': 1.7669671773910522, 'learning_rate': 8.89092495985411e-07, 'kl': -0.0047, 'entropy': 0.03, 'ce_loss': 0.0767, 'epoch': 0.2}

 10%|▉         | 139/1410 [27:17<4:10:43, 11.84s/it]
 10%|▉         | 140/1410 [27:28<4:06:13, 11.63s/it]
                                                    
{'loss': 0.0339, 'grad_norm': 2.5215675830841064, 'learning_rate': 8.888650194580321e-07, 'kl': -0.0035, 'entropy': -0.0737, 'ce_loss': 0.042, 'epoch': 0.2}

 10%|▉         | 140/1410 [27:28<4:06:13, 11.63s/it]
 10%|█         | 141/1410 [27:40<4:07:33, 11.70s/it]
                                                    
{'loss': 0.0221, 'grad_norm': 2.01941180229187, 'learning_rate': 8.886352250370813e-07, 'kl': -0.0041, 'entropy': 0.012, 'ce_loss': 0.0937, 'epoch': 0.2}

 10%|█         | 141/1410 [27:40<4:07:33, 11.70s/it]
 10%|█         | 142/1410 [27:52<4:06:45, 11.68s/it]
                                                    
{'loss': 0.0136, 'grad_norm': 1.1539289951324463, 'learning_rate': 8.884031139362324e-07, 'kl': 0.007, 'entropy': -0.0544, 'ce_loss': 0.0089, 'epoch': 0.2}

 10%|█         | 142/1410 [27:52<4:06:45, 11.68s/it]
 10%|█         | 143/1410 [28:03<4:02:35, 11.49s/it]
                                                    
{'loss': 0.0279, 'grad_norm': 2.263587236404419, 'learning_rate': 8.88168687381395e-07, 'kl': 0.0089, 'entropy': -0.0708, 'ce_loss': 0.0274, 'epoch': 0.2}

 10%|█         | 143/1410 [28:03<4:02:35, 11.49s/it]
 10%|█         | 144/1410 [28:15<4:05:04, 11.61s/it]
                                                    
{'loss': 0.0296, 'grad_norm': 2.3008315563201904, 'learning_rate': 8.879319466107077e-07, 'kl': -0.006, 'entropy': 0.0688, 'ce_loss': 0.0972, 'epoch': 0.2}

 10%|█         | 144/1410 [28:15<4:05:04, 11.61s/it]
 10%|█         | 145/1410 [28:27<4:07:25, 11.74s/it]
                                                    
{'loss': 0.0319, 'grad_norm': 2.1233561038970947, 'learning_rate': 8.876928928745322e-07, 'kl': -0.004, 'entropy': -0.0312, 'ce_loss': 0.0345, 'epoch': 0.21}

 10%|█         | 145/1410 [28:27<4:07:25, 11.74s/it]
 10%|█         | 146/1410 [28:38<4:05:02, 11.63s/it]
                                                    
{'loss': 0.0267, 'grad_norm': 2.338345527648926, 'learning_rate': 8.874515274354461e-07, 'kl': 0.0073, 'entropy': -0.0366, 'ce_loss': 0.0624, 'epoch': 0.21}

 10%|█         | 146/1410 [28:38<4:05:02, 11.63s/it]
 10%|█         | 147/1410 [28:50<4:05:19, 11.65s/it]
                                                    
{'loss': 0.0257, 'grad_norm': 1.8541796207427979, 'learning_rate': 8.872078515682362e-07, 'kl': -0.0024, 'entropy': 0.0184, 'ce_loss': 0.1175, 'epoch': 0.21}

 10%|█         | 147/1410 [28:50<4:05:19, 11.65s/it]
 10%|█         | 148/1410 [29:01<4:05:17, 11.66s/it]
                                                    
{'loss': 0.0123, 'grad_norm': 1.0376230478286743, 'learning_rate': 8.86961866559892e-07, 'kl': 0.0011, 'entropy': -0.025, 'ce_loss': 0.0148, 'epoch': 0.21}

 10%|█         | 148/1410 [29:01<4:05:17, 11.66s/it]
 11%|█         | 149/1410 [29:13<4:06:05, 11.71s/it]
                                                    
{'loss': 0.0296, 'grad_norm': 2.4096970558166504, 'learning_rate': 8.867135737095992e-07, 'kl': -0.002, 'entropy': -0.085, 'ce_loss': 0.0299, 'epoch': 0.21}

 11%|█         | 149/1410 [29:13<4:06:05, 11.71s/it]
 11%|█         | 150/1410 [29:25<4:07:06, 11.77s/it]
                                                    
{'loss': 0.0315, 'grad_norm': 1.9142616987228394, 'learning_rate': 8.864629743287324e-07, 'kl': 0.0013, 'entropy': -0.0457, 'ce_loss': 0.0466, 'epoch': 0.21}

 11%|█         | 150/1410 [29:25<4:07:06, 11.77s/it]
 11%|█         | 151/1410 [29:37<4:08:42, 11.85s/it]
                                                    
{'loss': 0.0426, 'grad_norm': 2.2935690879821777, 'learning_rate': 8.862100697408479e-07, 'kl': -0.0021, 'entropy': 0.0248, 'ce_loss': 0.0692, 'epoch': 0.21}

 11%|█         | 151/1410 [29:37<4:08:42, 11.85s/it]
 11%|█         | 152/1410 [29:49<4:09:32, 11.90s/it]
                                                    
{'loss': 0.0237, 'grad_norm': 1.2608349323272705, 'learning_rate': 8.859548612816777e-07, 'kl': -0.0038, 'entropy': 0.0757, 'ce_loss': 0.1021, 'epoch': 0.22}

 11%|█         | 152/1410 [29:49<4:09:32, 11.90s/it]
 11%|█         | 153/1410 [30:01<4:08:25, 11.86s/it]
                                                    
{'loss': 0.0205, 'grad_norm': 1.8238412141799927, 'learning_rate': 8.856973502991213e-07, 'kl': 0.0027, 'entropy': -0.0432, 'ce_loss': 0.0021, 'epoch': 0.22}

 11%|█         | 153/1410 [30:01<4:08:25, 11.86s/it]
 11%|█         | 154/1410 [30:13<4:07:33, 11.83s/it]
                                                    
{'loss': 0.0237, 'grad_norm': 1.713611364364624, 'learning_rate': 8.854375381532394e-07, 'kl': -0.0012, 'entropy': -0.0444, 'ce_loss': 0.031, 'epoch': 0.22}

 11%|█         | 154/1410 [30:13<4:07:33, 11.83s/it]
 11%|█         | 155/1410 [30:24<4:01:21, 11.54s/it]
                                                    
{'loss': 0.022, 'grad_norm': 2.046689748764038, 'learning_rate': 8.851754262162467e-07, 'kl': 0.0073, 'entropy': -0.0334, 'ce_loss': 0.0124, 'epoch': 0.22}

 11%|█         | 155/1410 [30:24<4:01:21, 11.54s/it]
 11%|█         | 156/1410 [30:36<4:03:41, 11.66s/it]
                                                    
{'loss': 0.0157, 'grad_norm': 1.3620365858078003, 'learning_rate': 8.849110158725038e-07, 'kl': 0.0026, 'entropy': -0.0349, 'ce_loss': 0.0099, 'epoch': 0.22}

 11%|█         | 156/1410 [30:36<4:03:41, 11.66s/it]
 11%|█         | 157/1410 [30:47<4:01:10, 11.55s/it]
                                                    
{'loss': 0.0227, 'grad_norm': 2.146960735321045, 'learning_rate': 8.846443085185109e-07, 'kl': 0.01, 'entropy': -0.0149, 'ce_loss': 0.0058, 'epoch': 0.22}

 11%|█         | 157/1410 [30:47<4:01:10, 11.55s/it]
 11%|█         | 158/1410 [30:59<4:04:11, 11.70s/it]
                                                    
{'loss': 0.0561, 'grad_norm': 2.6942272186279297, 'learning_rate': 8.843753055628997e-07, 'kl': -0.0002, 'entropy': -0.0649, 'ce_loss': 0.0322, 'epoch': 0.22}

 11%|█         | 158/1410 [30:59<4:04:11, 11.70s/it]
 11%|█▏        | 159/1410 [31:11<4:05:06, 11.76s/it]
                                                    
{'loss': 0.0155, 'grad_norm': 1.7527759075164795, 'learning_rate': 8.841040084264268e-07, 'kl': 0.0089, 'entropy': -0.0332, 'ce_loss': 0.0238, 'epoch': 0.23}

 11%|█▏        | 159/1410 [31:11<4:05:06, 11.76s/it]
 11%|█▏        | 160/1410 [31:22<4:03:38, 11.69s/it]
                                                    
{'loss': 0.0555, 'grad_norm': 3.0367400646209717, 'learning_rate': 8.838304185419651e-07, 'kl': 0.0018, 'entropy': -0.0557, 'ce_loss': 0.0208, 'epoch': 0.23}

 11%|█▏        | 160/1410 [31:22<4:03:38, 11.69s/it]
 11%|█▏        | 161/1410 [31:34<4:03:13, 11.68s/it]
                                                    
{'loss': 0.0285, 'grad_norm': 2.228408098220825, 'learning_rate': 8.835545373544969e-07, 'kl': -0.0047, 'entropy': -0.0649, 'ce_loss': 0.03, 'epoch': 0.23}

 11%|█▏        | 161/1410 [31:34<4:03:13, 11.68s/it]
 11%|█▏        | 162/1410 [31:46<4:02:30, 11.66s/it]
                                                    
{'loss': 0.0331, 'grad_norm': 2.455033540725708, 'learning_rate': 8.832763663211066e-07, 'kl': -0.0075, 'entropy': -0.0669, 'ce_loss': 0.0275, 'epoch': 0.23}

 11%|█▏        | 162/1410 [31:46<4:02:30, 11.66s/it]
 12%|█▏        | 163/1410 [31:58<4:04:19, 11.76s/it]
                                                    
{'loss': 0.0387, 'grad_norm': 2.384688377380371, 'learning_rate': 8.82995906910972e-07, 'kl': -0.0017, 'entropy': -0.0598, 'ce_loss': 0.0226, 'epoch': 0.23}

 12%|█▏        | 163/1410 [31:58<4:04:19, 11.76s/it]
 12%|█▏        | 164/1410 [32:10<4:06:36, 11.88s/it]
                                                    
{'loss': 0.0387, 'grad_norm': 1.873095989227295, 'learning_rate': 8.827131606053573e-07, 'kl': -0.0021, 'entropy': -0.061, 'ce_loss': 0.0151, 'epoch': 0.23}

 12%|█▏        | 164/1410 [32:10<4:06:36, 11.88s/it]
 12%|█▏        | 165/1410 [32:21<4:05:06, 11.81s/it]
                                                    
{'loss': 0.024, 'grad_norm': 1.775846004486084, 'learning_rate': 8.824281288976053e-07, 'kl': 0.0002, 'entropy': -0.0579, 'ce_loss': 0.0077, 'epoch': 0.23}

 12%|█▏        | 165/1410 [32:21<4:05:06, 11.81s/it]
 12%|█▏        | 166/1410 [32:33<4:04:27, 11.79s/it]
                                                    
{'loss': 0.0216, 'grad_norm': 2.036162853240967, 'learning_rate': 8.821408132931291e-07, 'kl': -0.0002, 'entropy': -0.0159, 'ce_loss': 0.0004, 'epoch': 0.24}

 12%|█▏        | 166/1410 [32:33<4:04:27, 11.79s/it]
 12%|█▏        | 167/1410 [32:45<4:04:14, 11.79s/it]
                                                    
{'loss': 0.0285, 'grad_norm': 2.0997116565704346, 'learning_rate': 8.818512153094043e-07, 'kl': -0.0058, 'entropy': 0.0972, 'ce_loss': 0.1799, 'epoch': 0.24}

 12%|█▏        | 167/1410 [32:45<4:04:14, 11.79s/it]
 12%|█▏        | 168/1410 [32:56<4:00:16, 11.61s/it]
                                                    
{'loss': 0.0322, 'grad_norm': 2.5833756923675537, 'learning_rate': 8.815593364759613e-07, 'kl': 0.003, 'entropy': -0.0747, 'ce_loss': 0.0238, 'epoch': 0.24}

 12%|█▏        | 168/1410 [32:56<4:00:16, 11.61s/it]
 12%|█▏        | 169/1410 [33:08<4:02:10, 11.71s/it]
                                                    
{'loss': 0.0353, 'grad_norm': 2.4343533515930176, 'learning_rate': 8.812651783343766e-07, 'kl': -0.0027, 'entropy': -0.0635, 'ce_loss': 0.0264, 'epoch': 0.24}

 12%|█▏        | 169/1410 [33:08<4:02:10, 11.71s/it]
 12%|█▏        | 170/1410 [33:20<4:03:49, 11.80s/it]
                                                    
{'loss': 0.0171, 'grad_norm': 1.1924530267715454, 'learning_rate': 8.80968742438265e-07, 'kl': -0.0056, 'entropy': -0.0493, 'ce_loss': 0.0516, 'epoch': 0.24}

 12%|█▏        | 170/1410 [33:20<4:03:49, 11.80s/it]
 12%|█▏        | 171/1410 [33:32<4:03:13, 11.78s/it]
                                                    
{'loss': 0.0272, 'grad_norm': 1.998048186302185, 'learning_rate': 8.80670030353272e-07, 'kl': -0.0007, 'entropy': -0.0352, 'ce_loss': 0.0027, 'epoch': 0.24}

 12%|█▏        | 171/1410 [33:32<4:03:13, 11.78s/it]
 12%|█▏        | 172/1410 [33:44<4:04:33, 11.85s/it]
                                                    
{'loss': 0.0327, 'grad_norm': 2.4303677082061768, 'learning_rate': 8.803690436570642e-07, 'kl': -0.0028, 'entropy': -0.0918, 'ce_loss': 0.0171, 'epoch': 0.24}

 12%|█▏        | 172/1410 [33:44<4:04:33, 11.85s/it]
 12%|█▏        | 173/1410 [33:56<4:03:58, 11.83s/it]
                                                    
{'loss': 0.0171, 'grad_norm': 5.315902233123779, 'learning_rate': 8.800657839393221e-07, 'kl': -0.0046, 'entropy': -0.084, 'ce_loss': 0.0515, 'epoch': 0.25}

 12%|█▏        | 173/1410 [33:56<4:03:58, 11.83s/it]
 12%|█▏        | 174/1410 [34:07<4:03:45, 11.83s/it]
                                                    
{'loss': 0.0176, 'grad_norm': 2.1545450687408447, 'learning_rate': 8.797602528017315e-07, 'kl': -0.0063, 'entropy': -0.0276, 'ce_loss': 0.0959, 'epoch': 0.25}

 12%|█▏        | 174/1410 [34:07<4:03:45, 11.83s/it]
 12%|█▏        | 175/1410 [34:19<4:03:30, 11.83s/it]
                                                    
{'loss': 0.0289, 'grad_norm': 2.6656248569488525, 'learning_rate': 8.79452451857974e-07, 'kl': 0.0101, 'entropy': -0.0664, 'ce_loss': 0.0427, 'epoch': 0.25}

 12%|█▏        | 175/1410 [34:19<4:03:30, 11.83s/it]
 12%|█▏        | 176/1410 [34:31<4:02:19, 11.78s/it]
                                                    
{'loss': 0.0327, 'grad_norm': 2.294839859008789, 'learning_rate': 8.791423827337204e-07, 'kl': -0.0159, 'entropy': -0.0157, 'ce_loss': 0.0049, 'epoch': 0.25}

 12%|█▏        | 176/1410 [34:31<4:02:19, 11.78s/it]
 13%|█▎        | 177/1410 [34:43<4:03:16, 11.84s/it]
                                                    
{'loss': 0.0202, 'grad_norm': 1.5288941860198975, 'learning_rate': 8.788300470666202e-07, 'kl': -0.0019, 'entropy': -0.0469, 'ce_loss': 0.0058, 'epoch': 0.25}

 13%|█▎        | 177/1410 [34:43<4:03:16, 11.84s/it]
 13%|█▎        | 178/1410 [34:55<4:01:45, 11.77s/it]
                                                    
{'loss': 0.0102, 'grad_norm': 1.304471492767334, 'learning_rate': 8.785154465062943e-07, 'kl': 0.0093, 'entropy': -0.0206, 'ce_loss': 0.0014, 'epoch': 0.25}

 13%|█▎        | 178/1410 [34:55<4:01:45, 11.77s/it]
 13%|█▎        | 179/1410 [35:06<3:57:35, 11.58s/it]
                                                    
{'loss': 0.0358, 'grad_norm': 2.799355983734131, 'learning_rate': 8.781985827143257e-07, 'kl': -0.0062, 'entropy': 0.0227, 'ce_loss': 0.0953, 'epoch': 0.25}

 13%|█▎        | 179/1410 [35:06<3:57:35, 11.58s/it]
 13%|█▎        | 180/1410 [35:18<3:59:26, 11.68s/it]
                                                    
{'loss': 0.0168, 'grad_norm': 1.5767536163330078, 'learning_rate': 8.778794573642506e-07, 'kl': -0.0058, 'entropy': -0.0747, 'ce_loss': 0.02, 'epoch': 0.26}

 13%|█▎        | 180/1410 [35:18<3:59:26, 11.68s/it]
 13%|█▎        | 181/1410 [35:28<3:53:28, 11.40s/it]
                                                    
{'loss': 0.0311, 'grad_norm': 2.737301826477051, 'learning_rate': 8.775580721415498e-07, 'kl': -0.0004, 'entropy': -0.0359, 'ce_loss': 0.0088, 'epoch': 0.26}

 13%|█▎        | 181/1410 [35:28<3:53:28, 11.40s/it]
 13%|█▎        | 182/1410 [35:40<3:55:42, 11.52s/it]
                                                    
{'loss': 0.0322, 'grad_norm': 2.611693859100342, 'learning_rate': 8.772344287436401e-07, 'kl': 0.0001, 'entropy': -0.0292, 'ce_loss': 0.0252, 'epoch': 0.26}

 13%|█▎        | 182/1410 [35:40<3:55:42, 11.52s/it]
 13%|█▎        | 183/1410 [35:52<3:56:25, 11.56s/it]
                                                    
{'loss': 0.023, 'grad_norm': 2.0022456645965576, 'learning_rate': 8.769085288798644e-07, 'kl': -0.0001, 'entropy': -0.0503, 'ce_loss': 0.0406, 'epoch': 0.26}

 13%|█▎        | 183/1410 [35:52<3:56:25, 11.56s/it]
 13%|█▎        | 184/1410 [36:04<3:58:17, 11.66s/it]
                                                    
{'loss': 0.0167, 'grad_norm': 1.698491096496582, 'learning_rate': 8.76580374271484e-07, 'kl': -0.0024, 'entropy': -0.0325, 'ce_loss': 0.0584, 'epoch': 0.26}

 13%|█▎        | 184/1410 [36:04<3:58:17, 11.66s/it]
 13%|█▎        | 185/1410 [36:15<3:58:54, 11.70s/it]
                                                    
{'loss': 0.0163, 'grad_norm': 2.102858066558838, 'learning_rate': 8.762499666516679e-07, 'kl': -0.0017, 'entropy': -0.0298, 'ce_loss': 0.0071, 'epoch': 0.26}

 13%|█▎        | 185/1410 [36:15<3:58:54, 11.70s/it]
 13%|█▎        | 186/1410 [36:27<3:59:14, 11.73s/it]
                                                    
{'loss': 0.0165, 'grad_norm': 1.3824653625488281, 'learning_rate': 8.759173077654854e-07, 'kl': -0.0019, 'entropy': -0.0698, 'ce_loss': 0.0246, 'epoch': 0.26}

 13%|█▎        | 186/1410 [36:27<3:59:14, 11.73s/it]
 13%|█▎        | 187/1410 [36:39<3:57:35, 11.66s/it]
                                                    
{'loss': 0.0299, 'grad_norm': 3.2314462661743164, 'learning_rate': 8.755823993698954e-07, 'kl': -0.0033, 'entropy': 0.0271, 'ce_loss': 0.0829, 'epoch': 0.27}

 13%|█▎        | 187/1410 [36:39<3:57:35, 11.66s/it]
 13%|█▎        | 188/1410 [36:51<3:58:46, 11.72s/it]
                                                    
{'loss': 0.036, 'grad_norm': 2.2050631046295166, 'learning_rate': 8.752452432337381e-07, 'kl': -0.0058, 'entropy': -0.043, 'ce_loss': 0.0384, 'epoch': 0.27}

 13%|█▎        | 188/1410 [36:51<3:58:46, 11.72s/it]
 13%|█▎        | 189/1410 [37:01<3:49:53, 11.30s/it]
                                                    
{'loss': 0.0162, 'grad_norm': 1.746134638786316, 'learning_rate': 8.749058411377249e-07, 'kl': 0.0006, 'entropy': -0.027, 'ce_loss': 0.0012, 'epoch': 0.27}

 13%|█▎        | 189/1410 [37:01<3:49:53, 11.30s/it]
 13%|█▎        | 190/1410 [37:13<3:53:47, 11.50s/it]
                                                    
{'loss': 0.0266, 'grad_norm': 1.4897689819335938, 'learning_rate': 8.745641948744298e-07, 'kl': 0.0157, 'entropy': -0.0815, 'ce_loss': 0.0106, 'epoch': 0.27}

 13%|█▎        | 190/1410 [37:13<3:53:47, 11.50s/it]
 14%|█▎        | 191/1410 [37:25<3:55:56, 11.61s/it]
                                                    
{'loss': 0.0471, 'grad_norm': 2.053213596343994, 'learning_rate': 8.742203062482792e-07, 'kl': 0.0019, 'entropy': -0.0121, 'ce_loss': 0.0002, 'epoch': 0.27}

 14%|█▎        | 191/1410 [37:25<3:55:56, 11.61s/it]
 14%|█▎        | 192/1410 [37:37<3:57:10, 11.68s/it]
                                                    
{'loss': 0.0368, 'grad_norm': 2.4130702018737793, 'learning_rate': 8.738741770755429e-07, 'kl': 0.0015, 'entropy': -0.0518, 'ce_loss': 0.0325, 'epoch': 0.27}

 14%|█▎        | 192/1410 [37:37<3:57:10, 11.68s/it]
 14%|█▎        | 193/1410 [37:48<3:56:59, 11.68s/it]
                                                    
{'loss': 0.0191, 'grad_norm': 1.720656394958496, 'learning_rate': 8.735258091843241e-07, 'kl': -0.0061, 'entropy': -0.0635, 'ce_loss': 0.0289, 'epoch': 0.27}

 14%|█▎        | 193/1410 [37:48<3:56:59, 11.68s/it]
 14%|█▍        | 194/1410 [38:00<3:57:00, 11.69s/it]
                                                    
{'loss': 0.037, 'grad_norm': 2.444038152694702, 'learning_rate': 8.7317520441455e-07, 'kl': 0.002, 'entropy': -0.0591, 'ce_loss': 0.0057, 'epoch': 0.28}

 14%|█▍        | 194/1410 [38:00<3:57:00, 11.69s/it]
 14%|█▍        | 195/1410 [38:11<3:52:16, 11.47s/it]
                                                    
{'loss': 0.025, 'grad_norm': 2.1342055797576904, 'learning_rate': 8.728223646179622e-07, 'kl': -0.0033, 'entropy': 0.0057, 'ce_loss': 0.0823, 'epoch': 0.28}

 14%|█▍        | 195/1410 [38:11<3:52:16, 11.47s/it]
 14%|█▍        | 196/1410 [38:23<3:53:02, 11.52s/it]
                                                    
{'loss': 0.0148, 'grad_norm': 1.6970397233963013, 'learning_rate': 8.724672916581063e-07, 'kl': -0.0076, 'entropy': -0.051, 'ce_loss': 0.0457, 'epoch': 0.28}

 14%|█▍        | 196/1410 [38:23<3:53:02, 11.52s/it]
 14%|█▍        | 197/1410 [38:34<3:49:41, 11.36s/it]
                                                    
{'loss': 0.0321, 'grad_norm': 2.8506081104278564, 'learning_rate': 8.72109987410323e-07, 'kl': -0.0019, 'entropy': -0.0422, 'ce_loss': 0.0293, 'epoch': 0.28}

 14%|█▍        | 197/1410 [38:34<3:49:41, 11.36s/it]
 14%|█▍        | 198/1410 [38:45<3:52:24, 11.51s/it]
                                                    
{'loss': 0.017, 'grad_norm': 2.0170929431915283, 'learning_rate': 8.717504537617374e-07, 'kl': -0.0003, 'entropy': -0.0742, 'ce_loss': 0.0225, 'epoch': 0.28}

 14%|█▍        | 198/1410 [38:45<3:52:24, 11.51s/it]
 14%|█▍        | 199/1410 [38:57<3:54:25, 11.61s/it]
                                                    
{'loss': 0.023, 'grad_norm': 1.8252702951431274, 'learning_rate': 8.713886926112494e-07, 'kl': -0.0087, 'entropy': -0.0366, 'ce_loss': 0.0285, 'epoch': 0.28}

 14%|█▍        | 199/1410 [38:57<3:54:25, 11.61s/it]
 14%|█▍        | 200/1410 [39:09<3:55:39, 11.69s/it]
                                                    
{'loss': 0.0345, 'grad_norm': 2.8174798488616943, 'learning_rate': 8.710247058695234e-07, 'kl': -0.0025, 'entropy': -0.0518, 'ce_loss': 0.0486, 'epoch': 0.28}

 14%|█▍        | 200/1410 [39:09<3:55:39, 11.69s/it]
 14%|█▍        | 201/1410 [39:21<3:56:44, 11.75s/it]
                                                    
{'loss': 0.0414, 'grad_norm': 2.330188512802124, 'learning_rate': 8.706584954589788e-07, 'kl': -0.0022, 'entropy': -0.0796, 'ce_loss': 0.0521, 'epoch': 0.29}

 14%|█▍        | 201/1410 [39:21<3:56:44, 11.75s/it]
 14%|█▍        | 202/1410 [39:33<3:54:45, 11.66s/it]
                                                    
{'loss': 0.0247, 'grad_norm': 2.3266329765319824, 'learning_rate': 8.702900633137791e-07, 'kl': 0.0032, 'entropy': -0.0679, 'ce_loss': 0.0479, 'epoch': 0.29}

 14%|█▍        | 202/1410 [39:33<3:54:45, 11.66s/it]
 14%|█▍        | 203/1410 [39:44<3:52:22, 11.55s/it]
                                                    
{'loss': 0.0289, 'grad_norm': 2.698166847229004, 'learning_rate': 8.699194113798221e-07, 'kl': -0.001, 'entropy': -0.0398, 'ce_loss': 0.0248, 'epoch': 0.29}

 14%|█▍        | 203/1410 [39:44<3:52:22, 11.55s/it]
 14%|█▍        | 204/1410 [39:56<3:53:52, 11.64s/it]
                                                    
{'loss': 0.0193, 'grad_norm': 1.7725911140441895, 'learning_rate': 8.6954654161473e-07, 'kl': -0.0071, 'entropy': -0.0148, 'ce_loss': 0.074, 'epoch': 0.29}

 14%|█▍        | 204/1410 [39:56<3:53:52, 11.64s/it]
 15%|█▍        | 205/1410 [40:07<3:54:57, 11.70s/it]
                                                    
{'loss': 0.0315, 'grad_norm': 2.186006546020508, 'learning_rate': 8.691714559878378e-07, 'kl': 0.0061, 'entropy': -0.0503, 'ce_loss': 0.0104, 'epoch': 0.29}

 15%|█▍        | 205/1410 [40:07<3:54:57, 11.70s/it]
 15%|█▍        | 206/1410 [40:19<3:56:03, 11.76s/it]
                                                    
{'loss': 0.0329, 'grad_norm': 1.769464135169983, 'learning_rate': 8.687941564801842e-07, 'kl': 0.0023, 'entropy': -0.0742, 'ce_loss': 0.0111, 'epoch': 0.29}

 15%|█▍        | 206/1410 [40:19<3:56:03, 11.76s/it]
 15%|█▍        | 207/1410 [40:32<3:58:34, 11.90s/it]
                                                    
{'loss': 0.0531, 'grad_norm': 1.765413761138916, 'learning_rate': 8.684146450845006e-07, 'kl': -0.0046, 'entropy': 0.1128, 'ce_loss': 0.177, 'epoch': 0.29}

 15%|█▍        | 207/1410 [40:32<3:58:34, 11.90s/it]
 15%|█▍        | 208/1410 [40:44<3:58:43, 11.92s/it]
                                                    
{'loss': 0.0472, 'grad_norm': 2.3690850734710693, 'learning_rate': 8.680329238052009e-07, 'kl': -0.0038, 'entropy': 0.1826, 'ce_loss': 0.1455, 'epoch': 0.3}

 15%|█▍        | 208/1410 [40:44<3:58:43, 11.92s/it]
 15%|█▍        | 209/1410 [40:56<3:58:45, 11.93s/it]
                                                    
{'loss': 0.0266, 'grad_norm': 1.9142658710479736, 'learning_rate': 8.676489946583701e-07, 'kl': -0.0118, 'entropy': -0.0045, 'ce_loss': 0.08, 'epoch': 0.3}

 15%|█▍        | 209/1410 [40:56<3:58:45, 11.93s/it]
 15%|█▍        | 210/1410 [41:07<3:58:08, 11.91s/it]
                                                    
{'loss': 0.0352, 'grad_norm': 1.7599425315856934, 'learning_rate': 8.672628596717545e-07, 'kl': 0.005, 'entropy': -0.0308, 'ce_loss': 0.0097, 'epoch': 0.3}

 15%|█▍        | 210/1410 [41:07<3:58:08, 11.91s/it]
 15%|█▍        | 211/1410 [41:18<3:49:44, 11.50s/it]
                                                    
{'loss': 0.0243, 'grad_norm': 1.9260705709457397, 'learning_rate': 8.668745208847507e-07, 'kl': 0.0042, 'entropy': -0.0464, 'ce_loss': 0.015, 'epoch': 0.3}

 15%|█▍        | 211/1410 [41:18<3:49:44, 11.50s/it]
 15%|█▌        | 212/1410 [41:30<3:50:11, 11.53s/it]
                                                    
{'loss': 0.0261, 'grad_norm': 2.510650157928467, 'learning_rate': 8.664839803483949e-07, 'kl': 0.0017, 'entropy': -0.0156, 'ce_loss': 0.0005, 'epoch': 0.3}

 15%|█▌        | 212/1410 [41:30<3:50:11, 11.53s/it]
 15%|█▌        | 213/1410 [41:41<3:52:17, 11.64s/it]
                                                    
{'loss': 0.0381, 'grad_norm': 2.1162538528442383, 'learning_rate': 8.660912401253514e-07, 'kl': 0.0, 'entropy': -0.042, 'ce_loss': 0.0373, 'epoch': 0.3}

 15%|█▌        | 213/1410 [41:41<3:52:17, 11.64s/it]
 15%|█▌        | 214/1410 [41:53<3:52:33, 11.67s/it]
                                                    
{'loss': 0.0149, 'grad_norm': 1.505334734916687, 'learning_rate': 8.656963022899031e-07, 'kl': -0.0104, 'entropy': 0.1001, 'ce_loss': 0.1236, 'epoch': 0.3}

 15%|█▌        | 214/1410 [41:53<3:52:33, 11.67s/it]
 15%|█▌        | 215/1410 [42:05<3:51:06, 11.60s/it]
                                                    
{'loss': 0.0325, 'grad_norm': 2.5300464630126953, 'learning_rate': 8.652991689279393e-07, 'kl': 0.0081, 'entropy': -0.053, 'ce_loss': 0.0104, 'epoch': 0.3}

 15%|█▌        | 215/1410 [42:05<3:51:06, 11.60s/it]
 15%|█▌        | 216/1410 [42:15<3:42:38, 11.19s/it]
                                                    
{'loss': 0.0425, 'grad_norm': 3.3191094398498535, 'learning_rate': 8.64899842136945e-07, 'kl': 0.0062, 'entropy': -0.0742, 'ce_loss': 0.0154, 'epoch': 0.31}

 15%|█▌        | 216/1410 [42:15<3:42:38, 11.19s/it]
 15%|█▌        | 217/1410 [42:27<3:45:30, 11.34s/it]
                                                    
{'loss': 0.0239, 'grad_norm': 2.082554578781128, 'learning_rate': 8.644983240259901e-07, 'kl': -0.0001, 'entropy': -0.0238, 'ce_loss': 0.0009, 'epoch': 0.31}

 15%|█▌        | 217/1410 [42:27<3:45:30, 11.34s/it]
 15%|█▌        | 218/1410 [42:38<3:47:28, 11.45s/it]
                                                    
{'loss': 0.0292, 'grad_norm': 2.2465128898620605, 'learning_rate': 8.640946167157183e-07, 'kl': -0.007, 'entropy': 0.0165, 'ce_loss': 0.0878, 'epoch': 0.31}

 15%|█▌        | 218/1410 [42:38<3:47:28, 11.45s/it]
 16%|█▌        | 219/1410 [42:50<3:48:27, 11.51s/it]
                                                    
{'loss': 0.0154, 'grad_norm': 1.519482135772705, 'learning_rate': 8.636887223383354e-07, 'kl': -0.0014, 'entropy': -0.0254, 'ce_loss': 0.0102, 'epoch': 0.31}

 16%|█▌        | 219/1410 [42:50<3:48:27, 11.51s/it]
 16%|█▌        | 220/1410 [43:02<3:49:19, 11.56s/it]
                                                    
{'loss': 0.0112, 'grad_norm': 1.3298521041870117, 'learning_rate': 8.632806430375984e-07, 'kl': -0.0098, 'entropy': 0.0508, 'ce_loss': 0.1121, 'epoch': 0.31}

 16%|█▌        | 220/1410 [43:02<3:49:19, 11.56s/it]
 16%|█▌        | 221/1410 [43:13<3:51:10, 11.67s/it]
                                                    
{'loss': 0.0516, 'grad_norm': 2.430288553237915, 'learning_rate': 8.628703809688041e-07, 'kl': -0.0079, 'entropy': -0.0737, 'ce_loss': 0.0265, 'epoch': 0.31}

 16%|█▌        | 221/1410 [43:13<3:51:10, 11.67s/it]
 16%|█▌        | 222/1410 [43:25<3:51:18, 11.68s/it]
                                                    
{'loss': 0.0207, 'grad_norm': 2.0034663677215576, 'learning_rate': 8.624579382987778e-07, 'kl': 0.0065, 'entropy': -0.0344, 'ce_loss': 0.0253, 'epoch': 0.31}

 16%|█▌        | 222/1410 [43:25<3:51:18, 11.68s/it]
 16%|█▌        | 223/1410 [43:37<3:51:18, 11.69s/it]
                                                    
{'loss': 0.0333, 'grad_norm': 1.906245231628418, 'learning_rate': 8.620433172058619e-07, 'kl': -0.0048, 'entropy': -0.0552, 'ce_loss': 0.0592, 'epoch': 0.32}

 16%|█▌        | 223/1410 [43:37<3:51:18, 11.69s/it]
 16%|█▌        | 224/1410 [43:48<3:50:22, 11.65s/it]
                                                    
{'loss': 0.0196, 'grad_norm': 2.072567939758301, 'learning_rate': 8.616265198799037e-07, 'kl': -0.0036, 'entropy': -0.051, 'ce_loss': 0.0648, 'epoch': 0.32}

 16%|█▌        | 224/1410 [43:48<3:50:22, 11.65s/it]
 16%|█▌        | 225/1410 [43:59<3:45:13, 11.40s/it]
                                                    
{'loss': 0.0375, 'grad_norm': 3.1533443927764893, 'learning_rate': 8.612075485222454e-07, 'kl': 0.0011, 'entropy': -0.0771, 'ce_loss': 0.0276, 'epoch': 0.32}

 16%|█▌        | 225/1410 [43:59<3:45:13, 11.40s/it]
 16%|█▌        | 226/1410 [44:11<3:46:18, 11.47s/it]
                                                    
{'loss': 0.0222, 'grad_norm': 2.123735189437866, 'learning_rate': 8.607864053457106e-07, 'kl': 0.006, 'entropy': -0.0388, 'ce_loss': 0.0102, 'epoch': 0.32}

 16%|█▌        | 226/1410 [44:11<3:46:18, 11.47s/it]
 16%|█▌        | 227/1410 [44:22<3:45:20, 11.43s/it]
                                                    
{'loss': 0.0302, 'grad_norm': 2.3001766204833984, 'learning_rate': 8.603630925745938e-07, 'kl': -0.0031, 'entropy': -0.051, 'ce_loss': 0.0515, 'epoch': 0.32}

 16%|█▌        | 227/1410 [44:22<3:45:20, 11.43s/it]
 16%|█▌        | 228/1410 [44:34<3:45:05, 11.43s/it]
                                                    
{'loss': 0.0221, 'grad_norm': 2.0005671977996826, 'learning_rate': 8.599376124446484e-07, 'kl': 0.0029, 'entropy': -0.064, 'ce_loss': 0.0285, 'epoch': 0.32}

 16%|█▌        | 228/1410 [44:34<3:45:05, 11.43s/it]
 16%|█▌        | 229/1410 [44:45<3:47:04, 11.54s/it]
                                                    
{'loss': 0.0258, 'grad_norm': 2.0312249660491943, 'learning_rate': 8.595099672030751e-07, 'kl': -0.0059, 'entropy': -0.0187, 'ce_loss': 0.048, 'epoch': 0.32}

 16%|█▌        | 229/1410 [44:45<3:47:04, 11.54s/it]
 16%|█▋        | 230/1410 [44:57<3:48:09, 11.60s/it]
                                                    
{'loss': 0.0255, 'grad_norm': 1.9119294881820679, 'learning_rate': 8.590801591085091e-07, 'kl': 0.0036, 'entropy': -0.0413, 'ce_loss': 0.0139, 'epoch': 0.33}

 16%|█▋        | 230/1410 [44:57<3:48:09, 11.60s/it]
 16%|█▋        | 231/1410 [45:09<3:48:16, 11.62s/it]
                                                    
{'loss': 0.0324, 'grad_norm': 2.209028959274292, 'learning_rate': 8.586481904310096e-07, 'kl': -0.0035, 'entropy': -0.0131, 'ce_loss': 0.0726, 'epoch': 0.33}

 16%|█▋        | 231/1410 [45:09<3:48:16, 11.62s/it]
 16%|█▋        | 232/1410 [45:21<3:49:42, 11.70s/it]
                                                    
{'loss': 0.0228, 'grad_norm': 1.4929653406143188, 'learning_rate': 8.582140634520464e-07, 'kl': -0.0087, 'entropy': 0.0129, 'ce_loss': 0.0973, 'epoch': 0.33}

 16%|█▋        | 232/1410 [45:21<3:49:42, 11.70s/it]
 17%|█▋        | 233/1410 [45:33<3:50:02, 11.73s/it]
                                                    
{'loss': 0.014, 'grad_norm': 1.2456824779510498, 'learning_rate': 8.57777780464489e-07, 'kl': -0.0047, 'entropy': -0.0723, 'ce_loss': 0.0323, 'epoch': 0.33}

 17%|█▋        | 233/1410 [45:33<3:50:02, 11.73s/it]
 17%|█▋        | 234/1410 [45:44<3:49:52, 11.73s/it]
                                                    
{'loss': 0.0195, 'grad_norm': 1.698911428451538, 'learning_rate': 8.573393437725936e-07, 'kl': -0.0066, 'entropy': -0.0127, 'ce_loss': 0.0574, 'epoch': 0.33}

 17%|█▋        | 234/1410 [45:44<3:49:52, 11.73s/it]
 17%|█▋        | 235/1410 [45:56<3:49:39, 11.73s/it]
                                                    
{'loss': 0.0351, 'grad_norm': 2.1479437351226807, 'learning_rate': 8.56898755691992e-07, 'kl': -0.0065, 'entropy': -0.0334, 'ce_loss': 0.0392, 'epoch': 0.33}

 17%|█▋        | 235/1410 [45:56<3:49:39, 11.73s/it]
 17%|█▋        | 236/1410 [46:08<3:49:21, 11.72s/it]
                                                    
{'loss': 0.037, 'grad_norm': 2.091522216796875, 'learning_rate': 8.564560185496777e-07, 'kl': 0.001, 'entropy': -0.016, 'ce_loss': 0.0008, 'epoch': 0.33}

 17%|█▋        | 236/1410 [46:08<3:49:21, 11.72s/it]
 17%|█▋        | 237/1410 [46:19<3:49:12, 11.72s/it]
                                                    
{'loss': 0.0443, 'grad_norm': 3.066519260406494, 'learning_rate': 8.560111346839954e-07, 'kl': 0.0088, 'entropy': -0.0167, 'ce_loss': 0.0066, 'epoch': 0.34}

 17%|█▋        | 237/1410 [46:19<3:49:12, 11.72s/it]
 17%|█▋        | 238/1410 [46:31<3:50:25, 11.80s/it]
                                                    
{'loss': 0.0355, 'grad_norm': 2.0706350803375244, 'learning_rate': 8.555641064446275e-07, 'kl': -0.0002, 'entropy': -0.0262, 'ce_loss': 0.0017, 'epoch': 0.34}

 17%|█▋        | 238/1410 [46:31<3:50:25, 11.80s/it]
 17%|█▋        | 239/1410 [46:43<3:46:54, 11.63s/it]
                                                    
{'loss': 0.0342, 'grad_norm': 2.507093906402588, 'learning_rate': 8.551149361925822e-07, 'kl': -0.0044, 'entropy': -0.0391, 'ce_loss': 0.0783, 'epoch': 0.34}

 17%|█▋        | 239/1410 [46:43<3:46:54, 11.63s/it]
 17%|█▋        | 240/1410 [46:54<3:47:27, 11.66s/it]
                                                    
{'loss': 0.0302, 'grad_norm': 2.4125900268554688, 'learning_rate': 8.546636263001808e-07, 'kl': -0.0037, 'entropy': 0.0874, 'ce_loss': 0.1351, 'epoch': 0.34}

 17%|█▋        | 240/1410 [46:54<3:47:27, 11.66s/it]
 17%|█▋        | 241/1410 [47:06<3:48:21, 11.72s/it]
                                                    
{'loss': 0.0119, 'grad_norm': 1.663472294807434, 'learning_rate': 8.542101791510453e-07, 'kl': -0.0028, 'entropy': 0.0381, 'ce_loss': 0.1532, 'epoch': 0.34}

 17%|█▋        | 241/1410 [47:06<3:48:21, 11.72s/it]
 17%|█▋        | 242/1410 [47:18<3:47:50, 11.70s/it]
                                                    
{'loss': 0.0608, 'grad_norm': 2.8864264488220215, 'learning_rate': 8.537545971400856e-07, 'kl': -0.0002, 'entropy': -0.0659, 'ce_loss': 0.0331, 'epoch': 0.34}

 17%|█▋        | 242/1410 [47:18<3:47:50, 11.70s/it]
 17%|█▋        | 243/1410 [47:30<3:47:45, 11.71s/it]
                                                    
{'loss': 0.021, 'grad_norm': 1.6858301162719727, 'learning_rate': 8.532968826734873e-07, 'kl': -0.0093, 'entropy': 0.0317, 'ce_loss': 0.1598, 'epoch': 0.34}

 17%|█▋        | 243/1410 [47:30<3:47:45, 11.71s/it]
 17%|█▋        | 244/1410 [47:41<3:47:46, 11.72s/it]
                                                    
{'loss': 0.0258, 'grad_norm': 2.138296604156494, 'learning_rate': 8.528370381686984e-07, 'kl': 0.0062, 'entropy': -0.0415, 'ce_loss': 0.0255, 'epoch': 0.35}

 17%|█▋        | 244/1410 [47:41<3:47:46, 11.72s/it]
 17%|█▋        | 245/1410 [47:53<3:48:14, 11.76s/it]
                                                    
{'loss': 0.0197, 'grad_norm': 1.8467832803726196, 'learning_rate': 8.52375066054417e-07, 'kl': 0.0095, 'entropy': -0.0371, 'ce_loss': 0.0076, 'epoch': 0.35}

 17%|█▋        | 245/1410 [47:53<3:48:14, 11.76s/it]
 17%|█▋        | 246/1410 [48:05<3:48:39, 11.79s/it]
                                                    
{'loss': 0.0412, 'grad_norm': 2.308516025543213, 'learning_rate': 8.519109687705782e-07, 'kl': -0.0019, 'entropy': 0.0396, 'ce_loss': 0.0954, 'epoch': 0.35}

 17%|█▋        | 246/1410 [48:05<3:48:39, 11.79s/it]
 18%|█▊        | 247/1410 [48:17<3:47:44, 11.75s/it]
                                                    
{'loss': 0.0279, 'grad_norm': 2.5611934661865234, 'learning_rate': 8.514447487683414e-07, 'kl': -0.0004, 'entropy': 0.0425, 'ce_loss': 0.089, 'epoch': 0.35}

 18%|█▊        | 247/1410 [48:17<3:47:44, 11.75s/it]
 18%|█▊        | 248/1410 [48:29<3:48:15, 11.79s/it]
                                                    
{'loss': 0.0155, 'grad_norm': 1.4250229597091675, 'learning_rate': 8.509764085100775e-07, 'kl': -0.0024, 'entropy': -0.0334, 'ce_loss': 0.0252, 'epoch': 0.35}

 18%|█▊        | 248/1410 [48:29<3:48:15, 11.79s/it]
 18%|█▊        | 249/1410 [48:40<3:48:04, 11.79s/it]
                                                    
{'loss': 0.0245, 'grad_norm': 1.21779465675354, 'learning_rate': 8.505059504693551e-07, 'kl': -0.0049, 'entropy': 0.1172, 'ce_loss': 0.1593, 'epoch': 0.35}

 18%|█▊        | 249/1410 [48:40<3:48:04, 11.79s/it]
 18%|█▊        | 250/1410 [48:52<3:47:42, 11.78s/it]
                                                    
{'loss': 0.0148, 'grad_norm': 2.0944676399230957, 'learning_rate': 8.500333771309285e-07, 'kl': -0.0054, 'entropy': 0.0212, 'ce_loss': 0.0946, 'epoch': 0.35}

 18%|█▊        | 250/1410 [48:52<3:47:42, 11.78s/it]
 18%|█▊        | 251/1410 [49:04<3:46:01, 11.70s/it]
                                                    
{'loss': 0.0419, 'grad_norm': 2.791252851486206, 'learning_rate': 8.495586909907237e-07, 'kl': 0.0003, 'entropy': -0.0452, 'ce_loss': 0.0355, 'epoch': 0.36}

 18%|█▊        | 251/1410 [49:04<3:46:01, 11.70s/it]
 18%|█▊        | 252/1410 [49:16<3:47:38, 11.79s/it]
                                                    
{'loss': 0.042, 'grad_norm': 2.1892738342285156, 'learning_rate': 8.490818945558261e-07, 'kl': -0.0052, 'entropy': -0.0515, 'ce_loss': 0.0263, 'epoch': 0.36}

 18%|█▊        | 252/1410 [49:16<3:47:38, 11.79s/it]
 18%|█▊        | 253/1410 [49:28<3:47:48, 11.81s/it]
                                                    
{'loss': 0.0273, 'grad_norm': 16.8605899810791, 'learning_rate': 8.486029903444663e-07, 'kl': -0.0005, 'entropy': -0.0287, 'ce_loss': 0.0006, 'epoch': 0.36}

 18%|█▊        | 253/1410 [49:28<3:47:48, 11.81s/it]
 18%|█▊        | 254/1410 [49:39<3:43:06, 11.58s/it]
                                                    
{'loss': 0.0224, 'grad_norm': 2.0961427688598633, 'learning_rate': 8.481219808860075e-07, 'kl': -0.0087, 'entropy': -0.0015, 'ce_loss': 0.0031, 'epoch': 0.36}

 18%|█▊        | 254/1410 [49:39<3:43:06, 11.58s/it]
 18%|█▊        | 255/1410 [49:49<3:38:13, 11.34s/it]
                                                    
{'loss': 0.0224, 'grad_norm': 1.6304606199264526, 'learning_rate': 8.476388687209317e-07, 'kl': 0.0005, 'entropy': -0.0786, 'ce_loss': 0.0406, 'epoch': 0.36}

 18%|█▊        | 255/1410 [49:49<3:38:13, 11.34s/it]
 18%|█▊        | 256/1410 [50:01<3:40:09, 11.45s/it]
                                                    
{'loss': 0.02, 'grad_norm': 1.755338191986084, 'learning_rate': 8.471536564008266e-07, 'kl': 0.0014, 'entropy': -0.033, 'ce_loss': 0.0061, 'epoch': 0.36}

 18%|█▊        | 256/1410 [50:01<3:40:09, 11.45s/it]
 18%|█▊        | 257/1410 [50:12<3:39:09, 11.40s/it]
                                                    
{'loss': 0.0322, 'grad_norm': 3.318338632583618, 'learning_rate': 8.466663464883721e-07, 'kl': 0.0007, 'entropy': 0.0581, 'ce_loss': 0.1105, 'epoch': 0.36}

 18%|█▊        | 257/1410 [50:12<3:39:09, 11.40s/it]
 18%|█▊        | 258/1410 [50:24<3:41:18, 11.53s/it]
                                                    
{'loss': 0.0326, 'grad_norm': 2.000060558319092, 'learning_rate': 8.461769415573264e-07, 'kl': -0.0019, 'entropy': -0.0708, 'ce_loss': 0.0628, 'epoch': 0.37}

 18%|█▊        | 258/1410 [50:24<3:41:18, 11.53s/it]
 18%|█▊        | 259/1410 [50:36<3:43:13, 11.64s/it]
                                                    
{'loss': 0.0244, 'grad_norm': 1.4396331310272217, 'learning_rate': 8.456854441925132e-07, 'kl': -0.0016, 'entropy': -0.0864, 'ce_loss': 0.0207, 'epoch': 0.37}

 18%|█▊        | 259/1410 [50:36<3:43:13, 11.64s/it]
 18%|█▊        | 260/1410 [50:48<3:43:31, 11.66s/it]
                                                    
{'loss': 0.0519, 'grad_norm': 2.328523874282837, 'learning_rate': 8.451918569898067e-07, 'kl': -0.008, 'entropy': -0.0496, 'ce_loss': 0.0096, 'epoch': 0.37}

 18%|█▊        | 260/1410 [50:48<3:43:31, 11.66s/it]
 19%|█▊        | 261/1410 [51:00<3:44:02, 11.70s/it]
                                                    
{'loss': 0.015, 'grad_norm': 1.4155197143554688, 'learning_rate': 8.446961825561194e-07, 'kl': -0.0003, 'entropy': 0.1025, 'ce_loss': 0.2041, 'epoch': 0.37}

 19%|█▊        | 261/1410 [51:00<3:44:02, 11.70s/it]
 19%|█▊        | 262/1410 [51:10<3:39:00, 11.45s/it]
                                                    
{'loss': 0.0194, 'grad_norm': 1.8592802286148071, 'learning_rate': 8.441984235093873e-07, 'kl': -0.0002, 'entropy': -0.0155, 'ce_loss': 0.0002, 'epoch': 0.37}

 19%|█▊        | 262/1410 [51:10<3:39:00, 11.45s/it]
 19%|█▊        | 263/1410 [51:22<3:41:12, 11.57s/it]
                                                    
{'loss': 0.0252, 'grad_norm': 1.4338747262954712, 'learning_rate': 8.436985824785566e-07, 'kl': 0.0067, 'entropy': -0.0713, 'ce_loss': 0.0166, 'epoch': 0.37}

 19%|█▊        | 263/1410 [51:22<3:41:12, 11.57s/it]
 19%|█▊        | 264/1410 [51:34<3:41:56, 11.62s/it]
                                                    
{'loss': 0.0209, 'grad_norm': 1.641374945640564, 'learning_rate': 8.431966621035697e-07, 'kl': -0.0025, 'entropy': -0.0349, 'ce_loss': 0.0095, 'epoch': 0.37}

 19%|█▊        | 264/1410 [51:34<3:41:56, 11.62s/it]
 19%|█▉        | 265/1410 [51:46<3:42:32, 11.66s/it]
                                                    
{'loss': 0.0293, 'grad_norm': 2.135709524154663, 'learning_rate': 8.426926650353508e-07, 'kl': -0.0067, 'entropy': -0.0586, 'ce_loss': 0.03, 'epoch': 0.38}

 19%|█▉        | 265/1410 [51:46<3:42:32, 11.66s/it]
 19%|█▉        | 266/1410 [51:58<3:44:04, 11.75s/it]
                                                    
{'loss': 0.0386, 'grad_norm': 2.20245361328125, 'learning_rate': 8.421865939357928e-07, 'kl': 0.0189, 'entropy': -0.0908, 'ce_loss': 0.0042, 'epoch': 0.38}

 19%|█▉        | 266/1410 [51:58<3:44:04, 11.75s/it]
 19%|█▉        | 267/1410 [52:09<3:39:56, 11.55s/it]
                                                    
{'loss': 0.0259, 'grad_norm': 2.4808919429779053, 'learning_rate': 8.416784514777424e-07, 'kl': 0.0066, 'entropy': -0.0031, 'ce_loss': 0.0047, 'epoch': 0.38}

 19%|█▉        | 267/1410 [52:09<3:39:56, 11.55s/it]
 19%|█▉        | 268/1410 [52:21<3:42:12, 11.68s/it]
                                                    
{'loss': 0.0506, 'grad_norm': 2.2384612560272217, 'learning_rate': 8.411682403449861e-07, 'kl': 0.0076, 'entropy': -0.0221, 'ce_loss': 0.0005, 'epoch': 0.38}

 19%|█▉        | 268/1410 [52:21<3:42:12, 11.68s/it]
 19%|█▉        | 269/1410 [52:33<3:42:57, 11.72s/it]
                                                    
{'loss': 0.0256, 'grad_norm': 2.12123966217041, 'learning_rate': 8.406559632322368e-07, 'kl': -0.0011, 'entropy': -0.0254, 'ce_loss': 0.0225, 'epoch': 0.38}

 19%|█▉        | 269/1410 [52:33<3:42:57, 11.72s/it]
 19%|█▉        | 270/1410 [52:44<3:43:27, 11.76s/it]
                                                    
{'loss': 0.0201, 'grad_norm': 1.623236894607544, 'learning_rate': 8.401416228451187e-07, 'kl': -0.0052, 'entropy': 0.0664, 'ce_loss': 0.1365, 'epoch': 0.38}

 19%|█▉        | 270/1410 [52:45<3:43:27, 11.76s/it]
 19%|█▉        | 271/1410 [52:56<3:43:09, 11.76s/it]
                                                    
{'loss': 0.0195, 'grad_norm': 2.4756627082824707, 'learning_rate': 8.396252219001531e-07, 'kl': -0.0066, 'entropy': 0.0105, 'ce_loss': 0.0742, 'epoch': 0.38}

 19%|█▉        | 271/1410 [52:56<3:43:09, 11.76s/it]
 19%|█▉        | 272/1410 [53:08<3:41:10, 11.66s/it]
                                                    
{'loss': 0.0218, 'grad_norm': 1.9386786222457886, 'learning_rate': 8.391067631247449e-07, 'kl': 0.0002, 'entropy': -0.0698, 'ce_loss': 0.0292, 'epoch': 0.39}

 19%|█▉        | 272/1410 [53:08<3:41:10, 11.66s/it]
 19%|█▉        | 273/1410 [53:20<3:42:09, 11.72s/it]
                                                    
{'loss': 0.0297, 'grad_norm': 2.669538974761963, 'learning_rate': 8.38586249257167e-07, 'kl': -0.0081, 'entropy': 0.0099, 'ce_loss': 0.0803, 'epoch': 0.39}

 19%|█▉        | 273/1410 [53:20<3:42:09, 11.72s/it]
 19%|█▉        | 274/1410 [53:31<3:40:39, 11.65s/it]
                                                    
{'loss': 0.0266, 'grad_norm': 2.095167875289917, 'learning_rate': 8.380636830465464e-07, 'kl': 0.0009, 'entropy': -0.0688, 'ce_loss': 0.0223, 'epoch': 0.39}

 19%|█▉        | 274/1410 [53:31<3:40:39, 11.65s/it]
 20%|█▉        | 275/1410 [53:42<3:35:07, 11.37s/it]
                                                    
{'loss': 0.0406, 'grad_norm': 2.067129135131836, 'learning_rate': 8.375390672528499e-07, 'kl': 0.0, 'entropy': -0.0205, 'ce_loss': 0.0035, 'epoch': 0.39}

 20%|█▉        | 275/1410 [53:42<3:35:07, 11.37s/it]
 20%|█▉        | 276/1410 [53:53<3:36:49, 11.47s/it]
                                                    
{'loss': 0.0203, 'grad_norm': 1.9907550811767578, 'learning_rate': 8.370124046468695e-07, 'kl': 0.0096, 'entropy': -0.0339, 'ce_loss': 0.0054, 'epoch': 0.39}

 20%|█▉        | 276/1410 [53:53<3:36:49, 11.47s/it]
 20%|█▉        | 277/1410 [54:05<3:39:16, 11.61s/it]
                                                    
{'loss': 0.0572, 'grad_norm': 2.5763611793518066, 'learning_rate': 8.364836980102071e-07, 'kl': 0.0028, 'entropy': -0.0281, 'ce_loss': 0.0127, 'epoch': 0.39}

 20%|█▉        | 277/1410 [54:05<3:39:16, 11.61s/it]
 20%|█▉        | 278/1410 [54:17<3:40:22, 11.68s/it]
                                                    
{'loss': 0.0308, 'grad_norm': 2.226619005203247, 'learning_rate': 8.359529501352605e-07, 'kl': -0.0022, 'entropy': -0.0601, 'ce_loss': 0.0451, 'epoch': 0.39}

 20%|█▉        | 278/1410 [54:17<3:40:22, 11.68s/it]
 20%|█▉        | 279/1410 [54:29<3:41:19, 11.74s/it]
                                                    
{'loss': 0.0301, 'grad_norm': 2.0258731842041016, 'learning_rate': 8.354201638252085e-07, 'kl': -0.0037, 'entropy': -0.0566, 'ce_loss': 0.0245, 'epoch': 0.4}

 20%|█▉        | 279/1410 [54:29<3:41:19, 11.74s/it]
 20%|█▉        | 280/1410 [54:41<3:41:13, 11.75s/it]
                                                    
{'loss': 0.0201, 'grad_norm': 1.8426525592803955, 'learning_rate': 8.348853418939959e-07, 'kl': -0.0082, 'entropy': 0.21, 'ce_loss': 0.1579, 'epoch': 0.4}

 20%|█▉        | 280/1410 [54:41<3:41:13, 11.75s/it]
 20%|█▉        | 281/1410 [54:53<3:41:02, 11.75s/it]
                                                    
{'loss': 0.0235, 'grad_norm': 1.9443719387054443, 'learning_rate': 8.343484871663188e-07, 'kl': 0.0033, 'entropy': -0.053, 'ce_loss': 0.0119, 'epoch': 0.4}

 20%|█▉        | 281/1410 [54:53<3:41:02, 11.75s/it]
 20%|██        | 282/1410 [55:04<3:35:58, 11.49s/it]
                                                    
{'loss': 0.0265, 'grad_norm': 1.771904706954956, 'learning_rate': 8.338096024776098e-07, 'kl': -0.0079, 'entropy': -0.0036, 'ce_loss': 0.1071, 'epoch': 0.4}

 20%|██        | 282/1410 [55:04<3:35:58, 11.49s/it]
 20%|██        | 283/1410 [55:15<3:37:58, 11.60s/it]
                                                    
{'loss': 0.0284, 'grad_norm': 2.0528719425201416, 'learning_rate': 8.332686906740226e-07, 'kl': -0.0087, 'entropy': 0.1104, 'ce_loss': 0.1545, 'epoch': 0.4}

 20%|██        | 283/1410 [55:15<3:37:58, 11.60s/it]
 20%|██        | 284/1410 [55:27<3:35:07, 11.46s/it]
                                                    
{'loss': 0.0268, 'grad_norm': 2.0147705078125, 'learning_rate': 8.327257546124174e-07, 'kl': -0.0012, 'entropy': -0.0309, 'ce_loss': 0.0068, 'epoch': 0.4}

 20%|██        | 284/1410 [55:27<3:35:07, 11.46s/it]
 20%|██        | 285/1410 [55:37<3:32:01, 11.31s/it]
                                                    
{'loss': 0.0336, 'grad_norm': 2.1278228759765625, 'learning_rate': 8.321807971603456e-07, 'kl': 0.0042, 'entropy': -0.033, 'ce_loss': 0.0116, 'epoch': 0.4}

 20%|██        | 285/1410 [55:37<3:32:01, 11.31s/it]
 20%|██        | 286/1410 [55:49<3:33:48, 11.41s/it]
                                                    
{'loss': 0.0087, 'grad_norm': 1.258486032485962, 'learning_rate': 8.316338211960347e-07, 'kl': 0.0082, 'entropy': -0.0232, 'ce_loss': 0.0094, 'epoch': 0.41}

 20%|██        | 286/1410 [55:49<3:33:48, 11.41s/it]
 20%|██        | 287/1410 [56:01<3:35:31, 11.52s/it]
                                                    
{'loss': 0.0165, 'grad_norm': 1.6607071161270142, 'learning_rate': 8.310848296083734e-07, 'kl': -0.0078, 'entropy': -0.0354, 'ce_loss': 0.0395, 'epoch': 0.41}

 20%|██        | 287/1410 [56:01<3:35:31, 11.52s/it]
 20%|██        | 288/1410 [56:13<3:37:55, 11.65s/it]
                                                    
{'loss': 0.0417, 'grad_norm': 2.2005419731140137, 'learning_rate': 8.305338252968953e-07, 'kl': 0.0066, 'entropy': -0.0349, 'ce_loss': 0.0136, 'epoch': 0.41}

 20%|██        | 288/1410 [56:13<3:37:55, 11.65s/it]
 20%|██        | 289/1410 [56:24<3:35:59, 11.56s/it]
                                                    
{'loss': 0.0535, 'grad_norm': 2.7959885597229004, 'learning_rate': 8.29980811171765e-07, 'kl': 0.0017, 'entropy': -0.0249, 'ce_loss': 0.0218, 'epoch': 0.41}

 20%|██        | 289/1410 [56:24<3:35:59, 11.56s/it]
 21%|██        | 290/1410 [56:36<3:36:40, 11.61s/it]
                                                    
{'loss': 0.0384, 'grad_norm': 2.720519781112671, 'learning_rate': 8.294257901537619e-07, 'kl': -0.001, 'entropy': -0.0698, 'ce_loss': 0.0412, 'epoch': 0.41}

 21%|██        | 290/1410 [56:36<3:36:40, 11.61s/it]
 21%|██        | 291/1410 [56:47<3:32:44, 11.41s/it]
                                                    
{'loss': 0.0175, 'grad_norm': 1.9195491075515747, 'learning_rate': 8.288687651742648e-07, 'kl': 0.0025, 'entropy': -0.0913, 'ce_loss': 0.0106, 'epoch': 0.41}

 21%|██        | 291/1410 [56:47<3:32:44, 11.41s/it]
 21%|██        | 292/1410 [56:59<3:34:08, 11.49s/it]
                                                    
{'loss': 0.0264, 'grad_norm': 2.0654404163360596, 'learning_rate': 8.283097391752368e-07, 'kl': -0.002, 'entropy': -0.0432, 'ce_loss': 0.0279, 'epoch': 0.41}

 21%|██        | 292/1410 [56:59<3:34:08, 11.49s/it]
 21%|██        | 293/1410 [57:10<3:33:34, 11.47s/it]
                                                    
{'loss': 0.0222, 'grad_norm': 2.2901220321655273, 'learning_rate': 8.277487151092089e-07, 'kl': 0.0021, 'entropy': -0.0209, 'ce_loss': 0.0038, 'epoch': 0.42}

 21%|██        | 293/1410 [57:10<3:33:34, 11.47s/it]
 21%|██        | 294/1410 [57:22<3:34:23, 11.53s/it]
                                                    
{'loss': 0.0196, 'grad_norm': 1.893428921699524, 'learning_rate': 8.27185695939266e-07, 'kl': -0.0013, 'entropy': -0.0703, 'ce_loss': 0.0261, 'epoch': 0.42}

 21%|██        | 294/1410 [57:22<3:34:23, 11.53s/it]
 21%|██        | 295/1410 [57:33<3:30:52, 11.35s/it]
                                                    
{'loss': 0.0336, 'grad_norm': 2.47265625, 'learning_rate': 8.266206846390295e-07, 'kl': 0.006, 'entropy': -0.027, 'ce_loss': 0.0065, 'epoch': 0.42}

 21%|██        | 295/1410 [57:33<3:30:52, 11.35s/it]
 21%|██        | 296/1410 [57:44<3:32:36, 11.45s/it]
                                                    
{'loss': 0.0216, 'grad_norm': 11.34348201751709, 'learning_rate': 8.260536841926426e-07, 'kl': -0.0039, 'entropy': 0.0388, 'ce_loss': 0.0692, 'epoch': 0.42}

 21%|██        | 296/1410 [57:44<3:32:36, 11.45s/it]
 21%|██        | 297/1410 [57:56<3:33:42, 11.52s/it]
                                                    
{'loss': 0.0268, 'grad_norm': 2.0386104583740234, 'learning_rate': 8.254846975947544e-07, 'kl': -0.0081, 'entropy': -0.041, 'ce_loss': 0.0824, 'epoch': 0.42}

 21%|██        | 297/1410 [57:56<3:33:42, 11.52s/it]
 21%|██        | 298/1410 [58:08<3:34:31, 11.57s/it]
                                                    
{'loss': 0.0207, 'grad_norm': 2.0674941539764404, 'learning_rate': 8.249137278505035e-07, 'kl': -0.0013, 'entropy': -0.0591, 'ce_loss': 0.043, 'epoch': 0.42}

 21%|██        | 298/1410 [58:08<3:34:31, 11.57s/it]
 21%|██        | 299/1410 [58:19<3:35:29, 11.64s/it]
                                                    
{'loss': 0.0258, 'grad_norm': 1.9483294486999512, 'learning_rate': 8.243407779755035e-07, 'kl': 0.0036, 'entropy': -0.0649, 'ce_loss': 0.0216, 'epoch': 0.42}

 21%|██        | 299/1410 [58:19<3:35:29, 11.64s/it]
 21%|██▏       | 300/1410 [58:31<3:36:05, 11.68s/it]
                                                    
{'loss': 0.0335, 'grad_norm': 1.836483359336853, 'learning_rate': 8.237658509958254e-07, 'kl': 0.0028, 'entropy': -0.026, 'ce_loss': 0.0018, 'epoch': 0.43}

 21%|██▏       | 300/1410 [58:31<3:36:05, 11.68s/it]
 21%|██▏       | 301/1410 [58:43<3:36:57, 11.74s/it]
                                                    
{'loss': 0.0383, 'grad_norm': 2.28363299369812, 'learning_rate': 8.231889499479826e-07, 'kl': -0.0087, 'entropy': 0.0011, 'ce_loss': 0.0561, 'epoch': 0.43}

 21%|██▏       | 301/1410 [58:43<3:36:57, 11.74s/it]
 21%|██▏       | 302/1410 [58:54<3:31:49, 11.47s/it]
                                                    
{'loss': 0.0158, 'grad_norm': 1.9841837882995605, 'learning_rate': 8.22610077878915e-07, 'kl': 0.0011, 'entropy': -0.0154, 'ce_loss': 0.0005, 'epoch': 0.43}

 21%|██▏       | 302/1410 [58:54<3:31:49, 11.47s/it]
 21%|██▏       | 303/1410 [59:06<3:33:25, 11.57s/it]
                                                    
{'loss': 0.0429, 'grad_norm': 2.3307340145111084, 'learning_rate': 8.220292378459724e-07, 'kl': 0.0036, 'entropy': -0.0581, 'ce_loss': 0.0224, 'epoch': 0.43}

 21%|██▏       | 303/1410 [59:06<3:33:25, 11.57s/it]
 22%|██▏       | 304/1410 [59:17<3:31:43, 11.49s/it]
                                                    
{'loss': 0.0199, 'grad_norm': 1.7453863620758057, 'learning_rate': 8.214464329168981e-07, 'kl': -0.0032, 'entropy': -0.0588, 'ce_loss': 0.0128, 'epoch': 0.43}

 22%|██▏       | 304/1410 [59:17<3:31:43, 11.49s/it]
 22%|██▏       | 305/1410 [59:29<3:34:29, 11.65s/it]
                                                    
{'loss': 0.0339, 'grad_norm': 1.8010146617889404, 'learning_rate': 8.208616661698136e-07, 'kl': -0.0056, 'entropy': -0.0688, 'ce_loss': 0.0415, 'epoch': 0.43}

 22%|██▏       | 305/1410 [59:29<3:34:29, 11.65s/it]
 22%|██▏       | 306/1410 [59:41<3:36:15, 11.75s/it]
                                                    
{'loss': 0.042, 'grad_norm': 2.561690092086792, 'learning_rate': 8.20274940693202e-07, 'kl': -0.0037, 'entropy': 0.1514, 'ce_loss': 0.1158, 'epoch': 0.43}

 22%|██▏       | 306/1410 [59:41<3:36:15, 11.75s/it]
 22%|██▏       | 307/1410 [59:53<3:34:46, 11.68s/it]
                                                    
{'loss': 0.0299, 'grad_norm': 2.4380712509155273, 'learning_rate': 8.19686259585891e-07, 'kl': -0.0002, 'entropy': -0.0216, 'ce_loss': 0.0016, 'epoch': 0.44}

 22%|██▏       | 307/1410 [59:53<3:34:46, 11.68s/it]
 22%|██▏       | 308/1410 [1:00:04<3:31:39, 11.52s/it]
                                                      
{'loss': 0.0229, 'grad_norm': 1.9524574279785156, 'learning_rate': 8.190956259570375e-07, 'kl': 0.0012, 'entropy': -0.0432, 'ce_loss': 0.0061, 'epoch': 0.44}

 22%|██▏       | 308/1410 [1:00:04<3:31:39, 11.52s/it]
 22%|██▏       | 309/1410 [1:00:15<3:31:41, 11.54s/it]
                                                      
{'loss': 0.0254, 'grad_norm': 2.018662452697754, 'learning_rate': 8.185030429261105e-07, 'kl': -0.0066, 'entropy': 0.0259, 'ce_loss': 0.1105, 'epoch': 0.44}

 22%|██▏       | 309/1410 [1:00:15<3:31:41, 11.54s/it]
 22%|██▏       | 310/1410 [1:00:27<3:33:40, 11.66s/it]
                                                      
{'loss': 0.0181, 'grad_norm': 1.7443665266036987, 'learning_rate': 8.179085136228753e-07, 'kl': 0.0048, 'entropy': -0.0581, 'ce_loss': 0.0043, 'epoch': 0.44}

 22%|██▏       | 310/1410 [1:00:27<3:33:40, 11.66s/it]
 22%|██▏       | 311/1410 [1:00:38<3:28:56, 11.41s/it]
                                                      
{'loss': 0.0366, 'grad_norm': 2.2926065921783447, 'learning_rate': 8.173120411873763e-07, 'kl': 0.0017, 'entropy': -0.0591, 'ce_loss': 0.0287, 'epoch': 0.44}

 22%|██▏       | 311/1410 [1:00:38<3:28:56, 11.41s/it]
 22%|██▏       | 312/1410 [1:00:49<3:27:44, 11.35s/it]
                                                      
{'loss': 0.0295, 'grad_norm': 3.002915859222412, 'learning_rate': 8.167136287699205e-07, 'kl': 0.0013, 'entropy': -0.0603, 'ce_loss': 0.0312, 'epoch': 0.44}

 22%|██▏       | 312/1410 [1:00:49<3:27:44, 11.35s/it]
 22%|██▏       | 313/1410 [1:01:01<3:29:04, 11.44s/it]
                                                      
{'loss': 0.0243, 'grad_norm': 2.405677080154419, 'learning_rate': 8.161132795310614e-07, 'kl': 0.0011, 'entropy': -0.0288, 'ce_loss': 0.0037, 'epoch': 0.44}

 22%|██▏       | 313/1410 [1:01:01<3:29:04, 11.44s/it]
 22%|██▏       | 314/1410 [1:01:13<3:30:18, 11.51s/it]
                                                      
{'loss': 0.0336, 'grad_norm': 2.5009472370147705, 'learning_rate': 8.155109966415815e-07, 'kl': -0.0098, 'entropy': -0.0444, 'ce_loss': 0.0309, 'epoch': 0.45}

 22%|██▏       | 314/1410 [1:01:13<3:30:18, 11.51s/it]
 22%|██▏       | 315/1410 [1:01:25<3:33:04, 11.68s/it]
                                                      
{'loss': 0.0502, 'grad_norm': 2.013730525970459, 'learning_rate': 8.149067832824767e-07, 'kl': -0.0052, 'entropy': 0.0522, 'ce_loss': 0.106, 'epoch': 0.45}

 22%|██▏       | 315/1410 [1:01:25<3:33:04, 11.68s/it]
 22%|██▏       | 316/1410 [1:01:37<3:35:13, 11.80s/it]
                                                      
{'loss': 0.0365, 'grad_norm': 2.2524797916412354, 'learning_rate': 8.14300642644938e-07, 'kl': -0.0064, 'entropy': 0.0374, 'ce_loss': 0.0721, 'epoch': 0.45}

 22%|██▏       | 316/1410 [1:01:37<3:35:13, 11.80s/it]
 22%|██▏       | 317/1410 [1:01:48<3:34:00, 11.75s/it]
                                                      
{'loss': 0.0305, 'grad_norm': 2.302945375442505, 'learning_rate': 8.136925779303359e-07, 'kl': 0.001, 'entropy': -0.012, 'ce_loss': 0.0053, 'epoch': 0.45}

 22%|██▏       | 317/1410 [1:01:48<3:34:00, 11.75s/it]
 23%|██▎       | 318/1410 [1:02:00<3:33:37, 11.74s/it]
                                                      
{'loss': 0.0383, 'grad_norm': 2.050584316253662, 'learning_rate': 8.130825923502029e-07, 'kl': -0.007, 'entropy': 0.031, 'ce_loss': 0.0982, 'epoch': 0.45}

 23%|██▎       | 318/1410 [1:02:00<3:33:37, 11.74s/it]
 23%|██▎       | 319/1410 [1:02:12<3:32:56, 11.71s/it]
                                                      
{'loss': 0.0193, 'grad_norm': 1.9737091064453125, 'learning_rate': 8.124706891262168e-07, 'kl': -0.0072, 'entropy': 0.0003, 'ce_loss': 0.0638, 'epoch': 0.45}

 23%|██▎       | 319/1410 [1:02:12<3:32:56, 11.71s/it]
 23%|██▎       | 320/1410 [1:02:24<3:34:23, 11.80s/it]
                                                      
{'loss': 0.035, 'grad_norm': 2.2234504222869873, 'learning_rate': 8.118568714901836e-07, 'kl': 0.0026, 'entropy': -0.0171, 'ce_loss': 0.0001, 'epoch': 0.45}

 23%|██▎       | 320/1410 [1:02:24<3:34:23, 11.80s/it]
 23%|██▎       | 321/1410 [1:02:35<3:30:41, 11.61s/it]
                                                      
{'loss': 0.0784, 'grad_norm': 3.0722618103027344, 'learning_rate': 8.1124114268402e-07, 'kl': 0.0047, 'entropy': -0.0275, 'ce_loss': 0.0046, 'epoch': 0.46}

 23%|██▎       | 321/1410 [1:02:35<3:30:41, 11.61s/it]
 23%|██▎       | 322/1410 [1:02:47<3:31:49, 11.68s/it]
                                                      
{'loss': 0.0277, 'grad_norm': 1.7611712217330933, 'learning_rate': 8.106235059597373e-07, 'kl': -0.0015, 'entropy': -0.062, 'ce_loss': 0.0265, 'epoch': 0.46}

 23%|██▎       | 322/1410 [1:02:47<3:31:49, 11.68s/it]
 23%|██▎       | 323/1410 [1:02:58<3:32:01, 11.70s/it]
                                                      
{'loss': 0.0128, 'grad_norm': 1.571034550666809, 'learning_rate': 8.100039645794228e-07, 'kl': -0.0061, 'entropy': -0.0094, 'ce_loss': 0.0769, 'epoch': 0.46}

 23%|██▎       | 323/1410 [1:02:59<3:32:01, 11.70s/it]
 23%|██▎       | 324/1410 [1:03:09<3:27:13, 11.45s/it]
                                                      
{'loss': 0.0304, 'grad_norm': 2.654660940170288, 'learning_rate': 8.09382521815224e-07, 'kl': -0.007, 'entropy': -0.0515, 'ce_loss': 0.0316, 'epoch': 0.46}

 23%|██▎       | 324/1410 [1:03:09<3:27:13, 11.45s/it]
 23%|██▎       | 325/1410 [1:03:21<3:28:55, 11.55s/it]
                                                      
{'loss': 0.0301, 'grad_norm': 2.587087392807007, 'learning_rate': 8.087591809493304e-07, 'kl': -0.0029, 'entropy': -0.0688, 'ce_loss': 0.0134, 'epoch': 0.46}

 23%|██▎       | 325/1410 [1:03:21<3:28:55, 11.55s/it]
 23%|██▎       | 326/1410 [1:03:33<3:29:39, 11.60s/it]
                                                      
{'loss': 0.0129, 'grad_norm': 1.1969705820083618, 'learning_rate': 8.081339452739568e-07, 'kl': 0.0013, 'entropy': -0.0442, 'ce_loss': 0.0303, 'epoch': 0.46}

 23%|██▎       | 326/1410 [1:03:33<3:29:39, 11.60s/it]
 23%|██▎       | 327/1410 [1:03:45<3:31:48, 11.73s/it]
                                                      
{'loss': 0.0252, 'grad_norm': 1.4173674583435059, 'learning_rate': 8.075068180913248e-07, 'kl': -0.0064, 'entropy': -0.0427, 'ce_loss': 0.0696, 'epoch': 0.46}

 23%|██▎       | 327/1410 [1:03:45<3:31:48, 11.73s/it]
 23%|██▎       | 328/1410 [1:03:57<3:31:10, 11.71s/it]
                                                      
{'loss': 0.0342, 'grad_norm': 2.489992141723633, 'learning_rate': 8.06877802713647e-07, 'kl': -0.0036, 'entropy': -0.0757, 'ce_loss': 0.0512, 'epoch': 0.47}

 23%|██▎       | 328/1410 [1:03:57<3:31:10, 11.71s/it]
 23%|██▎       | 329/1410 [1:04:08<3:31:50, 11.76s/it]
                                                      
{'loss': 0.0506, 'grad_norm': 2.3034110069274902, 'learning_rate': 8.06246902463108e-07, 'kl': -0.0112, 'entropy': 0.0051, 'ce_loss': 0.0809, 'epoch': 0.47}

 23%|██▎       | 329/1410 [1:04:08<3:31:50, 11.76s/it]
 23%|██▎       | 330/1410 [1:04:20<3:32:58, 11.83s/it]
                                                      
{'loss': 0.0373, 'grad_norm': 2.1139204502105713, 'learning_rate': 8.056141206718475e-07, 'kl': -0.0072, 'entropy': -0.0669, 'ce_loss': 0.0306, 'epoch': 0.47}

 23%|██▎       | 330/1410 [1:04:20<3:32:58, 11.83s/it]
 23%|██▎       | 331/1410 [1:04:32<3:32:50, 11.84s/it]
                                                      
{'loss': 0.024, 'grad_norm': 2.011848211288452, 'learning_rate': 8.049794606819433e-07, 'kl': -0.0072, 'entropy': 0.0615, 'ce_loss': 0.1072, 'epoch': 0.47}

 23%|██▎       | 331/1410 [1:04:32<3:32:50, 11.84s/it]
 24%|██▎       | 332/1410 [1:04:44<3:32:34, 11.83s/it]
                                                      
{'loss': 0.0185, 'grad_norm': 1.4530906677246094, 'learning_rate': 8.043429258453924e-07, 'kl': -0.0003, 'entropy': -0.0091, 'ce_loss': 0.0001, 'epoch': 0.47}

 24%|██▎       | 332/1410 [1:04:44<3:32:34, 11.83s/it]
 24%|██▎       | 333/1410 [1:04:56<3:31:36, 11.79s/it]
                                                      
{'loss': 0.0149, 'grad_norm': 1.5198711156845093, 'learning_rate': 8.037045195240938e-07, 'kl': -0.0082, 'entropy': 0.0669, 'ce_loss': 0.139, 'epoch': 0.47}

 24%|██▎       | 333/1410 [1:04:56<3:31:36, 11.79s/it]
 24%|██▎       | 334/1410 [1:05:08<3:32:12, 11.83s/it]
                                                      
{'loss': 0.0284, 'grad_norm': 1.5366148948669434, 'learning_rate': 8.030642450898316e-07, 'kl': -0.002, 'entropy': 0.0044, 'ce_loss': 0.0807, 'epoch': 0.47}

 24%|██▎       | 334/1410 [1:05:08<3:32:12, 11.83s/it]
 24%|██▍       | 335/1410 [1:05:19<3:31:00, 11.78s/it]
                                                      
{'loss': 0.02, 'grad_norm': 1.8508460521697998, 'learning_rate': 8.02422105924256e-07, 'kl': -0.0063, 'entropy': -0.0728, 'ce_loss': 0.0224, 'epoch': 0.48}

 24%|██▍       | 335/1410 [1:05:19<3:31:00, 11.78s/it]
 24%|██▍       | 336/1410 [1:05:30<3:25:19, 11.47s/it]
                                                      
{'loss': 0.0217, 'grad_norm': 2.221750020980835, 'learning_rate': 8.017781054188656e-07, 'kl': -0.0059, 'entropy': 0.0928, 'ce_loss': 0.1581, 'epoch': 0.48}

 24%|██▍       | 336/1410 [1:05:30<3:25:19, 11.47s/it]
 24%|██▍       | 337/1410 [1:05:42<3:25:48, 11.51s/it]
                                                      
{'loss': 0.0145, 'grad_norm': 1.637024164199829, 'learning_rate': 8.011322469749905e-07, 'kl': 0.0065, 'entropy': -0.0679, 'ce_loss': 0.0199, 'epoch': 0.48}

 24%|██▍       | 337/1410 [1:05:42<3:25:48, 11.51s/it]
 24%|██▍       | 338/1410 [1:05:53<3:22:02, 11.31s/it]
                                                      
{'loss': 0.0362, 'grad_norm': 2.661144971847534, 'learning_rate': 8.00484534003773e-07, 'kl': -0.0102, 'entropy': -0.0262, 'ce_loss': 0.0719, 'epoch': 0.48}

 24%|██▍       | 338/1410 [1:05:53<3:22:02, 11.31s/it]
 24%|██▍       | 339/1410 [1:06:04<3:22:00, 11.32s/it]
                                                      
{'loss': 0.0168, 'grad_norm': 1.7069265842437744, 'learning_rate': 7.998349699261505e-07, 'kl': -0.003, 'entropy': -0.0708, 'ce_loss': 0.018, 'epoch': 0.48}

 24%|██▍       | 339/1410 [1:06:04<3:22:00, 11.32s/it]
 24%|██▍       | 340/1410 [1:06:15<3:21:34, 11.30s/it]
                                                      
{'loss': 0.025, 'grad_norm': 1.8674899339675903, 'learning_rate': 7.991835581728372e-07, 'kl': -0.009, 'entropy': 0.0214, 'ce_loss': 0.1125, 'epoch': 0.48}

 24%|██▍       | 340/1410 [1:06:15<3:21:34, 11.30s/it]
 24%|██▍       | 341/1410 [1:06:27<3:24:08, 11.46s/it]
                                                      
{'loss': 0.0301, 'grad_norm': 1.9554826021194458, 'learning_rate': 7.985303021843055e-07, 'kl': -0.0064, 'entropy': -0.0038, 'ce_loss': 0.068, 'epoch': 0.48}

 24%|██▍       | 341/1410 [1:06:27<3:24:08, 11.46s/it]
 24%|██▍       | 342/1410 [1:06:39<3:26:09, 11.58s/it]
                                                      
{'loss': 0.0225, 'grad_norm': 1.6718438863754272, 'learning_rate': 7.978752054107688e-07, 'kl': -0.0074, 'entropy': -0.083, 'ce_loss': 0.0478, 'epoch': 0.49}

 24%|██▍       | 342/1410 [1:06:39<3:26:09, 11.58s/it]
 24%|██▍       | 343/1410 [1:06:51<3:28:01, 11.70s/it]
                                                      
{'loss': 0.0542, 'grad_norm': 2.8799290657043457, 'learning_rate': 7.972182713121623e-07, 'kl': -0.0029, 'entropy': -0.0542, 'ce_loss': 0.0148, 'epoch': 0.49}

 24%|██▍       | 343/1410 [1:06:51<3:28:01, 11.70s/it]
 24%|██▍       | 344/1410 [1:07:02<3:27:37, 11.69s/it]
                                                      
{'loss': 0.0326, 'grad_norm': 2.074296712875366, 'learning_rate': 7.965595033581255e-07, 'kl': 0.0041, 'entropy': -0.0356, 'ce_loss': 0.0121, 'epoch': 0.49}

 24%|██▍       | 344/1410 [1:07:02<3:27:37, 11.69s/it]
 24%|██▍       | 345/1410 [1:07:14<3:28:03, 11.72s/it]
                                                      
{'loss': 0.0198, 'grad_norm': 2.7251641750335693, 'learning_rate': 7.958989050279831e-07, 'kl': -0.0014, 'entropy': -0.05, 'ce_loss': 0.008, 'epoch': 0.49}

 24%|██▍       | 345/1410 [1:07:14<3:28:03, 11.72s/it]
 25%|██▍       | 346/1410 [1:07:26<3:28:14, 11.74s/it]
                                                      
{'loss': 0.0208, 'grad_norm': 1.4393495321273804, 'learning_rate': 7.952364798107276e-07, 'kl': 0.0067, 'entropy': -0.0026, 'ce_loss': 0.0068, 'epoch': 0.49}

 25%|██▍       | 346/1410 [1:07:26<3:28:14, 11.74s/it]
 25%|██▍       | 347/1410 [1:07:37<3:23:34, 11.49s/it]
                                                      
{'loss': 0.0432, 'grad_norm': 2.7028093338012695, 'learning_rate': 7.945722312049997e-07, 'kl': -0.0082, 'entropy': -0.0713, 'ce_loss': 0.0384, 'epoch': 0.49}

 25%|██▍       | 347/1410 [1:07:37<3:23:34, 11.49s/it]
 25%|██▍       | 348/1410 [1:07:49<3:25:03, 11.59s/it]
                                                      
{'loss': 0.0325, 'grad_norm': 1.9789303541183472, 'learning_rate': 7.939061627190709e-07, 'kl': 0.0025, 'entropy': -0.0415, 'ce_loss': 0.0219, 'epoch': 0.49}

 25%|██▍       | 348/1410 [1:07:49<3:25:03, 11.59s/it]
 25%|██▍       | 349/1410 [1:08:01<3:25:33, 11.62s/it]
                                                      
{'loss': 0.0421, 'grad_norm': 2.23335337638855, 'learning_rate': 7.932382778708243e-07, 'kl': 0.0035, 'entropy': -0.0205, 'ce_loss': 0.0017, 'epoch': 0.5}

 25%|██▍       | 349/1410 [1:08:01<3:25:33, 11.62s/it]
 25%|██▍       | 350/1410 [1:08:12<3:23:36, 11.53s/it]
                                                      
{'loss': 0.0316, 'grad_norm': 2.6040573120117188, 'learning_rate': 7.925685801877364e-07, 'kl': 0.0001, 'entropy': -0.0776, 'ce_loss': 0.0188, 'epoch': 0.5}

 25%|██▍       | 350/1410 [1:08:12<3:23:36, 11.53s/it]
 25%|██▍       | 351/1410 [1:08:24<3:25:44, 11.66s/it]
                                                      
{'loss': 0.0522, 'grad_norm': 2.5023326873779297, 'learning_rate': 7.91897073206858e-07, 'kl': -0.0045, 'entropy': 0.0186, 'ce_loss': 0.0845, 'epoch': 0.5}

 25%|██▍       | 351/1410 [1:08:24<3:25:44, 11.66s/it]
 25%|██▍       | 352/1410 [1:08:36<3:27:12, 11.75s/it]
                                                      
{'loss': 0.0454, 'grad_norm': 1.3511806726455688, 'learning_rate': 7.912237604747961e-07, 'kl': 0.0029, 'entropy': -0.0205, 'ce_loss': 0.0046, 'epoch': 0.5}

 25%|██▍       | 352/1410 [1:08:36<3:27:12, 11.75s/it]
 25%|██▌       | 353/1410 [1:08:47<3:22:13, 11.48s/it]
                                                      
{'loss': 0.0289, 'grad_norm': 2.188991069793701, 'learning_rate': 7.905486455476951e-07, 'kl': -0.0058, 'entropy': -0.0084, 'ce_loss': 0.0922, 'epoch': 0.5}

 25%|██▌       | 353/1410 [1:08:47<3:22:13, 11.48s/it]
 25%|██▌       | 354/1410 [1:08:57<3:16:25, 11.16s/it]
                                                      
{'loss': 0.035, 'grad_norm': 3.0288069248199463, 'learning_rate': 7.898717319912169e-07, 'kl': -0.0032, 'entropy': -0.0391, 'ce_loss': 0.0476, 'epoch': 0.5}

 25%|██▌       | 354/1410 [1:08:57<3:16:25, 11.16s/it]
 25%|██▌       | 355/1410 [1:09:09<3:18:52, 11.31s/it]
                                                      
{'loss': 0.0346, 'grad_norm': 2.821092128753662, 'learning_rate': 7.891930233805241e-07, 'kl': -0.0039, 'entropy': -0.0703, 'ce_loss': 0.008, 'epoch': 0.5}

 25%|██▌       | 355/1410 [1:09:09<3:18:52, 11.31s/it]
 25%|██▌       | 356/1410 [1:09:20<3:21:24, 11.47s/it]
                                                      
{'loss': 0.0366, 'grad_norm': 2.255718469619751, 'learning_rate': 7.885125233002595e-07, 'kl': -0.0057, 'entropy': -0.0354, 'ce_loss': 0.0242, 'epoch': 0.5}

 25%|██▌       | 356/1410 [1:09:20<3:21:24, 11.47s/it]
 25%|██▌       | 357/1410 [1:09:32<3:22:58, 11.57s/it]
                                                      
{'loss': 0.0273, 'grad_norm': 1.488949179649353, 'learning_rate': 7.878302353445274e-07, 'kl': 0.0016, 'entropy': -0.0776, 'ce_loss': 0.0186, 'epoch': 0.51}

 25%|██▌       | 357/1410 [1:09:32<3:22:58, 11.57s/it]
 25%|██▌       | 358/1410 [1:09:44<3:24:44, 11.68s/it]
                                                      
{'loss': 0.0215, 'grad_norm': 1.7989610433578491, 'learning_rate': 7.871461631168752e-07, 'kl': -0.001, 'entropy': -0.0337, 'ce_loss': 0.0074, 'epoch': 0.51}

 25%|██▌       | 358/1410 [1:09:44<3:24:44, 11.68s/it]
 25%|██▌       | 359/1410 [1:09:56<3:25:58, 11.76s/it]
                                                      
{'loss': 0.0233, 'grad_norm': 1.7012449502944946, 'learning_rate': 7.86460310230274e-07, 'kl': 0.0013, 'entropy': -0.0603, 'ce_loss': 0.0337, 'epoch': 0.51}

 25%|██▌       | 359/1410 [1:09:56<3:25:58, 11.76s/it]
 26%|██▌       | 360/1410 [1:10:08<3:25:11, 11.73s/it]
                                                      
{'loss': 0.0093, 'grad_norm': 1.3203771114349365, 'learning_rate': 7.857726803070995e-07, 'kl': 0.0101, 'entropy': -0.0879, 'ce_loss': 0.0232, 'epoch': 0.51}

 26%|██▌       | 360/1410 [1:10:08<3:25:11, 11.73s/it]
 26%|██▌       | 361/1410 [1:10:19<3:24:17, 11.69s/it]
                                                      
{'loss': 0.0161, 'grad_norm': 1.441791296005249, 'learning_rate': 7.85083276979113e-07, 'kl': 0.011, 'entropy': -0.0767, 'ce_loss': 0.0111, 'epoch': 0.51}

 26%|██▌       | 361/1410 [1:10:19<3:24:17, 11.69s/it][2025-04-27 15:46:19,665] [WARNING] [stage3.py:2139:step] 1 pytorch allocator cache flushes since last step. this happens when there is high memory pressure and is detrimental to performance. if this is happening frequently consider adjusting settings to reduce memory consumption. If you are unable to make the cache flushes go away consider adding get_accelerator().empty_cache() calls in your training loop to ensure that all ranks flush their caches at the same time

 26%|██▌       | 362/1410 [1:10:32<3:27:03, 11.85s/it]
                                                      
{'loss': 0.0153, 'grad_norm': 1.570387601852417, 'learning_rate': 7.843921038874422e-07, 'kl': -0.0033, 'entropy': 0.1377, 'ce_loss': 0.1737, 'epoch': 0.51}

 26%|██▌       | 362/1410 [1:10:32<3:27:03, 11.85s/it]
 26%|██▌       | 363/1410 [1:10:43<3:26:13, 11.82s/it]
                                                      
{'loss': 0.0346, 'grad_norm': 2.536686897277832, 'learning_rate': 7.836991646825616e-07, 'kl': 0.0006, 'entropy': -0.0952, 'ce_loss': 0.0206, 'epoch': 0.51}

 26%|██▌       | 363/1410 [1:10:43<3:26:13, 11.82s/it]
 26%|██▌       | 364/1410 [1:10:56<3:27:47, 11.92s/it]
                                                      
{'loss': 0.0372, 'grad_norm': 2.503627300262451, 'learning_rate': 7.83004463024274e-07, 'kl': -0.0062, 'entropy': -0.0075, 'ce_loss': 0.0895, 'epoch': 0.52}

 26%|██▌       | 364/1410 [1:10:56<3:27:47, 11.92s/it]
 26%|██▌       | 365/1410 [1:11:07<3:23:36, 11.69s/it]
                                                      
{'loss': 0.0138, 'grad_norm': 1.7957159280776978, 'learning_rate': 7.823080025816904e-07, 'kl': -0.0003, 'entropy': -0.0757, 'ce_loss': 0.0232, 'epoch': 0.52}

 26%|██▌       | 365/1410 [1:11:07<3:23:36, 11.69s/it]
 26%|██▌       | 366/1410 [1:11:18<3:22:04, 11.61s/it]
                                                      
{'loss': 0.0387, 'grad_norm': 2.416278600692749, 'learning_rate': 7.81609787033211e-07, 'kl': -0.0013, 'entropy': -0.0128, 'ce_loss': 0.0087, 'epoch': 0.52}

 26%|██▌       | 366/1410 [1:11:18<3:22:04, 11.61s/it]
 26%|██▌       | 367/1410 [1:11:30<3:23:17, 11.69s/it]
                                                      
{'loss': 0.0405, 'grad_norm': 2.376861572265625, 'learning_rate': 7.809098200665057e-07, 'kl': -0.0001, 'entropy': -0.0913, 'ce_loss': 0.0085, 'epoch': 0.52}

 26%|██▌       | 367/1410 [1:11:30<3:23:17, 11.69s/it]
 26%|██▌       | 368/1410 [1:11:42<3:23:23, 11.71s/it]
                                                      
{'loss': 0.0237, 'grad_norm': 1.281113624572754, 'learning_rate': 7.802081053784947e-07, 'kl': -0.0074, 'entropy': 0.1206, 'ce_loss': 0.1626, 'epoch': 0.52}

 26%|██▌       | 368/1410 [1:11:42<3:23:23, 11.71s/it]
 26%|██▌       | 369/1410 [1:11:54<3:24:35, 11.79s/it]
                                                      
{'loss': 0.0427, 'grad_norm': 1.9856488704681396, 'learning_rate': 7.795046466753288e-07, 'kl': -0.0054, 'entropy': -0.007, 'ce_loss': 0.1109, 'epoch': 0.52}

 26%|██▌       | 369/1410 [1:11:54<3:24:35, 11.79s/it]
 26%|██▌       | 370/1410 [1:12:06<3:24:16, 11.79s/it]
                                                      
{'loss': 0.0277, 'grad_norm': 2.234760284423828, 'learning_rate': 7.787994476723701e-07, 'kl': 0.0028, 'entropy': -0.0164, 'ce_loss': 0.0009, 'epoch': 0.52}

 26%|██▌       | 370/1410 [1:12:06<3:24:16, 11.79s/it]
 26%|██▋       | 371/1410 [1:12:17<3:24:24, 11.80s/it]
                                                      
{'loss': 0.0599, 'grad_norm': 2.638035535812378, 'learning_rate': 7.780925120941719e-07, 'kl': -0.0001, 'entropy': -0.0129, 'ce_loss': 0.0002, 'epoch': 0.53}

 26%|██▋       | 371/1410 [1:12:17<3:24:24, 11.80s/it]
 26%|██▋       | 372/1410 [1:12:29<3:24:40, 11.83s/it]
                                                      
{'loss': 0.0146, 'grad_norm': 0.9808434247970581, 'learning_rate': 7.773838436744597e-07, 'kl': -0.0009, 'entropy': 0.0259, 'ce_loss': 0.074, 'epoch': 0.53}

 26%|██▋       | 372/1410 [1:12:29<3:24:40, 11.83s/it]
 26%|██▋       | 373/1410 [1:12:40<3:19:04, 11.52s/it]
                                                      
{'loss': 0.0197, 'grad_norm': 1.8822197914123535, 'learning_rate': 7.766734461561109e-07, 'kl': -0.0011, 'entropy': -0.0583, 'ce_loss': 0.0233, 'epoch': 0.53}

 26%|██▋       | 373/1410 [1:12:40<3:19:04, 11.52s/it]
 27%|██▋       | 374/1410 [1:12:52<3:20:27, 11.61s/it]
                                                      
{'loss': 0.0324, 'grad_norm': 2.2234108448028564, 'learning_rate': 7.75961323291135e-07, 'kl': -0.0048, 'entropy': 0.0408, 'ce_loss': 0.1118, 'epoch': 0.53}

 27%|██▋       | 374/1410 [1:12:52<3:20:27, 11.61s/it]
 27%|██▋       | 375/1410 [1:13:04<3:20:55, 11.65s/it]
                                                      
{'loss': 0.0295, 'grad_norm': 1.7866374254226685, 'learning_rate': 7.752474788406545e-07, 'kl': -0.0018, 'entropy': -0.0223, 'ce_loss': 0.0038, 'epoch': 0.53}

 27%|██▋       | 375/1410 [1:13:04<3:20:55, 11.65s/it]
 27%|██▋       | 376/1410 [1:13:15<3:20:57, 11.66s/it]
                                                      
{'loss': 0.0221, 'grad_norm': 2.2574517726898193, 'learning_rate': 7.745319165748847e-07, 'kl': -0.0089, 'entropy': 0.0002, 'ce_loss': 0.0954, 'epoch': 0.53}

 27%|██▋       | 376/1410 [1:13:15<3:20:57, 11.66s/it]
 27%|██▋       | 377/1410 [1:13:27<3:20:15, 11.63s/it]
                                                      
{'loss': 0.0324, 'grad_norm': 2.46463942527771, 'learning_rate': 7.738146402731127e-07, 'kl': -0.0076, 'entropy': -0.0781, 'ce_loss': 0.0246, 'epoch': 0.53}

 27%|██▋       | 377/1410 [1:13:27<3:20:15, 11.63s/it]
 27%|██▋       | 378/1410 [1:13:39<3:20:06, 11.63s/it]
                                                      
{'loss': 0.0259, 'grad_norm': 2.3458967208862305, 'learning_rate': 7.730956537236793e-07, 'kl': 0.001, 'entropy': -0.0625, 'ce_loss': 0.0205, 'epoch': 0.54}

 27%|██▋       | 378/1410 [1:13:39<3:20:06, 11.63s/it]
 27%|██▋       | 379/1410 [1:13:50<3:19:47, 11.63s/it]
                                                      
{'loss': 0.0636, 'grad_norm': 2.9946703910827637, 'learning_rate': 7.72374960723958e-07, 'kl': -0.0057, 'entropy': -0.0454, 'ce_loss': 0.0291, 'epoch': 0.54}

 27%|██▋       | 379/1410 [1:13:50<3:19:47, 11.63s/it]
 27%|██▋       | 380/1410 [1:14:01<3:16:38, 11.46s/it]
                                                      
{'loss': 0.0169, 'grad_norm': 1.7585344314575195, 'learning_rate': 7.716525650803342e-07, 'kl': -0.0059, 'entropy': -0.0508, 'ce_loss': 0.0616, 'epoch': 0.54}

 27%|██▋       | 380/1410 [1:14:01<3:16:38, 11.46s/it]
 27%|██▋       | 381/1410 [1:14:13<3:18:43, 11.59s/it]
                                                      
{'loss': 0.0333, 'grad_norm': 1.9171826839447021, 'learning_rate': 7.709284706081874e-07, 'kl': -0.0026, 'entropy': -0.0167, 'ce_loss': 0.0133, 'epoch': 0.54}

 27%|██▋       | 381/1410 [1:14:13<3:18:43, 11.59s/it]
 27%|██▋       | 382/1410 [1:14:25<3:19:51, 11.66s/it]
                                                      
{'loss': 0.0296, 'grad_norm': 1.7862210273742676, 'learning_rate': 7.702026811318678e-07, 'kl': 0.0089, 'entropy': -0.0635, 'ce_loss': 0.0423, 'epoch': 0.54}

 27%|██▋       | 382/1410 [1:14:25<3:19:51, 11.66s/it]
 27%|██▋       | 383/1410 [1:14:37<3:20:51, 11.73s/it]
                                                      
{'loss': 0.0176, 'grad_norm': 1.421544075012207, 'learning_rate': 7.694752004846795e-07, 'kl': -0.0039, 'entropy': 0.0157, 'ce_loss': 0.053, 'epoch': 0.54}

 27%|██▋       | 383/1410 [1:14:37<3:20:51, 11.73s/it]
 27%|██▋       | 384/1410 [1:14:49<3:21:24, 11.78s/it]
                                                      
{'loss': 0.0404, 'grad_norm': 2.4590606689453125, 'learning_rate': 7.687460325088574e-07, 'kl': -0.0056, 'entropy': -0.026, 'ce_loss': 0.0139, 'epoch': 0.54}

 27%|██▋       | 384/1410 [1:14:49<3:21:24, 11.78s/it]
 27%|██▋       | 385/1410 [1:15:01<3:22:06, 11.83s/it]
                                                      
{'loss': 0.0433, 'grad_norm': 2.3827428817749023, 'learning_rate': 7.68015181055549e-07, 'kl': -0.0078, 'entropy': -0.0762, 'ce_loss': 0.0352, 'epoch': 0.55}

 27%|██▋       | 385/1410 [1:15:01<3:22:06, 11.83s/it]
 27%|██▋       | 386/1410 [1:15:12<3:20:58, 11.78s/it]
                                                      
{'loss': 0.0247, 'grad_norm': 2.118009328842163, 'learning_rate': 7.672826499847926e-07, 'kl': -0.0033, 'entropy': 0.1914, 'ce_loss': 0.157, 'epoch': 0.55}

 27%|██▋       | 386/1410 [1:15:12<3:20:58, 11.78s/it]
 27%|██▋       | 387/1410 [1:15:24<3:19:42, 11.71s/it]
                                                      
{'loss': 0.0144, 'grad_norm': 1.5483757257461548, 'learning_rate': 7.665484431654979e-07, 'kl': -0.0043, 'entropy': -0.0791, 'ce_loss': 0.0475, 'epoch': 0.55}

 27%|██▋       | 387/1410 [1:15:24<3:19:42, 11.71s/it]
 28%|██▊       | 388/1410 [1:15:36<3:19:46, 11.73s/it]
                                                      
{'loss': 0.0313, 'grad_norm': 2.3096983432769775, 'learning_rate': 7.65812564475425e-07, 'kl': 0.0015, 'entropy': -0.0957, 'ce_loss': 0.0196, 'epoch': 0.55}

 28%|██▊       | 388/1410 [1:15:36<3:19:46, 11.73s/it]
 28%|██▊       | 389/1410 [1:15:47<3:20:10, 11.76s/it]
                                                      
{'loss': 0.0231, 'grad_norm': 1.9348280429840088, 'learning_rate': 7.650750178011639e-07, 'kl': -0.002, 'entropy': -0.0435, 'ce_loss': 0.0279, 'epoch': 0.55}

 28%|██▊       | 389/1410 [1:15:47<3:20:10, 11.76s/it]
 28%|██▊       | 390/1410 [1:15:58<3:16:08, 11.54s/it]
                                                      
{'loss': 0.0344, 'grad_norm': 3.823090076446533, 'learning_rate': 7.643358070381144e-07, 'kl': -0.0002, 'entropy': -0.0654, 'ce_loss': 0.0454, 'epoch': 0.55}

 28%|██▊       | 390/1410 [1:15:58<3:16:08, 11.54s/it]
 28%|██▊       | 391/1410 [1:16:10<3:17:24, 11.62s/it]
                                                      
{'loss': 0.0179, 'grad_norm': 1.5790297985076904, 'learning_rate': 7.635949360904656e-07, 'kl': -0.006, 'entropy': 0.016, 'ce_loss': 0.1213, 'epoch': 0.55}

 28%|██▊       | 391/1410 [1:16:10<3:17:24, 11.62s/it]
 28%|██▊       | 392/1410 [1:16:22<3:18:13, 11.68s/it]
                                                      
{'loss': 0.025, 'grad_norm': 1.9152096509933472, 'learning_rate': 7.628524088711743e-07, 'kl': -0.0, 'entropy': -0.0201, 'ce_loss': 0.0075, 'epoch': 0.56}

 28%|██▊       | 392/1410 [1:16:22<3:18:13, 11.68s/it]
 28%|██▊       | 393/1410 [1:16:34<3:18:16, 11.70s/it]
                                                      
{'loss': 0.0145, 'grad_norm': 1.3110814094543457, 'learning_rate': 7.621082293019454e-07, 'kl': -0.0064, 'entropy': -0.0393, 'ce_loss': 0.0303, 'epoch': 0.56}

 28%|██▊       | 393/1410 [1:16:34<3:18:16, 11.70s/it]
 28%|██▊       | 394/1410 [1:16:45<3:17:24, 11.66s/it]
                                                      
{'loss': 0.0284, 'grad_norm': 1.7761447429656982, 'learning_rate': 7.613624013132108e-07, 'kl': -0.0001, 'entropy': -0.0596, 'ce_loss': 0.027, 'epoch': 0.56}

 28%|██▊       | 394/1410 [1:16:45<3:17:24, 11.66s/it]
 28%|██▊       | 395/1410 [1:16:57<3:17:49, 11.69s/it]
                                                      
{'loss': 0.0379, 'grad_norm': 2.6494300365448, 'learning_rate': 7.606149288441085e-07, 'kl': -0.009, 'entropy': -0.0388, 'ce_loss': 0.0535, 'epoch': 0.56}

 28%|██▊       | 395/1410 [1:16:57<3:17:49, 11.69s/it]
 28%|██▊       | 396/1410 [1:17:09<3:17:53, 11.71s/it]
                                                      
{'loss': 0.0317, 'grad_norm': 2.389150619506836, 'learning_rate': 7.59865815842462e-07, 'kl': -0.0036, 'entropy': -0.0649, 'ce_loss': 0.0519, 'epoch': 0.56}

 28%|██▊       | 396/1410 [1:17:09<3:17:53, 11.71s/it]
 28%|██▊       | 397/1410 [1:17:21<3:18:54, 11.78s/it]
                                                      
{'loss': 0.0446, 'grad_norm': 2.4487271308898926, 'learning_rate': 7.591150662647594e-07, 'kl': -0.0036, 'entropy': 0.0359, 'ce_loss': 0.0895, 'epoch': 0.56}

 28%|██▊       | 397/1410 [1:17:21<3:18:54, 11.78s/it]
 28%|██▊       | 398/1410 [1:17:33<3:19:14, 11.81s/it]
                                                      
{'loss': 0.0154, 'grad_norm': 1.702573537826538, 'learning_rate': 7.583626840761325e-07, 'kl': -0.0044, 'entropy': 0.0381, 'ce_loss': 0.107, 'epoch': 0.56}

 28%|██▊       | 398/1410 [1:17:33<3:19:14, 11.81s/it]
 28%|██▊       | 399/1410 [1:17:45<3:18:59, 11.81s/it]
                                                      
{'loss': 0.0132, 'grad_norm': 1.408104419708252, 'learning_rate': 7.576086732503355e-07, 'kl': -0.0067, 'entropy': -0.0366, 'ce_loss': 0.0648, 'epoch': 0.57}

 28%|██▊       | 399/1410 [1:17:45<3:18:59, 11.81s/it]
 28%|██▊       | 400/1410 [1:17:56<3:15:50, 11.63s/it]
                                                      
{'loss': 0.0199, 'grad_norm': 1.700273036956787, 'learning_rate': 7.56853037769725e-07, 'kl': -0.0027, 'entropy': -0.0173, 'ce_loss': 0.0144, 'epoch': 0.57}

 28%|██▊       | 400/1410 [1:17:56<3:15:50, 11.63s/it]
 28%|██▊       | 401/1410 [1:18:08<3:16:51, 11.71s/it]
                                                      
{'loss': 0.0198, 'grad_norm': 1.4835880994796753, 'learning_rate': 7.560957816252378e-07, 'kl': -0.0104, 'entropy': -0.0491, 'ce_loss': 0.0281, 'epoch': 0.57}

 28%|██▊       | 401/1410 [1:18:08<3:16:51, 11.71s/it]
 29%|██▊       | 402/1410 [1:18:20<3:17:33, 11.76s/it]
                                                      
{'loss': 0.0446, 'grad_norm': 2.2538840770721436, 'learning_rate': 7.553369088163703e-07, 'kl': -0.0044, 'entropy': 0.0261, 'ce_loss': 0.0916, 'epoch': 0.57}

 29%|██▊       | 402/1410 [1:18:20<3:17:33, 11.76s/it]
 29%|██▊       | 403/1410 [1:18:30<3:12:55, 11.49s/it]
                                                      
{'loss': 0.0291, 'grad_norm': 2.4046289920806885, 'learning_rate': 7.545764233511578e-07, 'kl': -0.0071, 'entropy': -0.0596, 'ce_loss': 0.0456, 'epoch': 0.57}

 29%|██▊       | 403/1410 [1:18:30<3:12:55, 11.49s/it]
 29%|██▊       | 404/1410 [1:18:42<3:13:24, 11.53s/it]
                                                      
{'loss': 0.0336, 'grad_norm': 2.8985395431518555, 'learning_rate': 7.538143292461528e-07, 'kl': 0.004, 'entropy': -0.0153, 'ce_loss': 0.0009, 'epoch': 0.57}

 29%|██▊       | 404/1410 [1:18:42<3:13:24, 11.53s/it]
 29%|██▊       | 405/1410 [1:18:53<3:10:23, 11.37s/it]
                                                      
{'loss': 0.042, 'grad_norm': 2.517591953277588, 'learning_rate': 7.530506305264039e-07, 'kl': -0.0066, 'entropy': -0.0427, 'ce_loss': 0.0417, 'epoch': 0.57}

 29%|██▊       | 405/1410 [1:18:53<3:10:23, 11.37s/it]
 29%|██▉       | 406/1410 [1:19:05<3:12:26, 11.50s/it]
                                                      
{'loss': 0.0117, 'grad_norm': 1.8086538314819336, 'learning_rate': 7.522853312254344e-07, 'kl': -0.004, 'entropy': 0.0461, 'ce_loss': 0.0651, 'epoch': 0.58}

 29%|██▉       | 406/1410 [1:19:05<3:12:26, 11.50s/it]
 29%|██▉       | 407/1410 [1:19:17<3:13:43, 11.59s/it]
                                                      
{'loss': 0.0265, 'grad_norm': 2.5316147804260254, 'learning_rate': 7.515184353852216e-07, 'kl': 0.011, 'entropy': -0.0618, 'ce_loss': 0.0199, 'epoch': 0.58}

 29%|██▉       | 407/1410 [1:19:17<3:13:43, 11.59s/it]
 29%|██▉       | 408/1410 [1:19:29<3:14:48, 11.67s/it]
                                                      
{'loss': 0.0311, 'grad_norm': 1.6711227893829346, 'learning_rate': 7.507499470561744e-07, 'kl': -0.001, 'entropy': 0.0381, 'ce_loss': 0.057, 'epoch': 0.58}

 29%|██▉       | 408/1410 [1:19:29<3:14:48, 11.67s/it]
 29%|██▉       | 409/1410 [1:19:40<3:12:12, 11.52s/it]
                                                      
{'loss': 0.0242, 'grad_norm': 2.721525192260742, 'learning_rate': 7.499798702971131e-07, 'kl': -0.0042, 'entropy': -0.0776, 'ce_loss': 0.0272, 'epoch': 0.58}

 29%|██▉       | 409/1410 [1:19:40<3:12:12, 11.52s/it]
 29%|██▉       | 410/1410 [1:19:51<3:13:03, 11.58s/it]
                                                      
{'loss': 0.0236, 'grad_norm': 1.9472200870513916, 'learning_rate': 7.492082091752472e-07, 'kl': -0.0075, 'entropy': 0.0143, 'ce_loss': 0.0706, 'epoch': 0.58}

 29%|██▉       | 410/1410 [1:19:51<3:13:03, 11.58s/it]
 29%|██▉       | 411/1410 [1:20:03<3:14:33, 11.69s/it]
                                                      
{'loss': 0.0435, 'grad_norm': 2.033396005630493, 'learning_rate': 7.48434967766154e-07, 'kl': -0.0057, 'entropy': 0.016, 'ce_loss': 0.1119, 'epoch': 0.58}

 29%|██▉       | 411/1410 [1:20:03<3:14:33, 11.69s/it]
 29%|██▉       | 412/1410 [1:20:15<3:14:10, 11.67s/it]
                                                      
{'loss': 0.0226, 'grad_norm': 1.464625597000122, 'learning_rate': 7.476601501537574e-07, 'kl': -0.0004, 'entropy': -0.0991, 'ce_loss': 0.0269, 'epoch': 0.58}

 29%|██▉       | 412/1410 [1:20:15<3:14:10, 11.67s/it]
 29%|██▉       | 413/1410 [1:20:26<3:11:25, 11.52s/it]
                                                      
{'loss': 0.0378, 'grad_norm': 2.1464288234710693, 'learning_rate': 7.46883760430306e-07, 'kl': -0.0029, 'entropy': 0.01, 'ce_loss': 0.0921, 'epoch': 0.59}

 29%|██▉       | 413/1410 [1:20:26<3:11:25, 11.52s/it]
 29%|██▉       | 414/1410 [1:20:38<3:12:25, 11.59s/it]
                                                      
{'loss': 0.0446, 'grad_norm': 2.5877485275268555, 'learning_rate': 7.461058026963513e-07, 'kl': -0.0068, 'entropy': -0.0645, 'ce_loss': 0.0411, 'epoch': 0.59}

 29%|██▉       | 414/1410 [1:20:38<3:12:25, 11.59s/it]
 29%|██▉       | 415/1410 [1:20:50<3:12:34, 11.61s/it]
                                                      
{'loss': 0.0374, 'grad_norm': 2.679800510406494, 'learning_rate': 7.453262810607273e-07, 'kl': -0.0044, 'entropy': -0.0302, 'ce_loss': 0.0175, 'epoch': 0.59}

 29%|██▉       | 415/1410 [1:20:50<3:12:34, 11.61s/it]
 30%|██▉       | 416/1410 [1:21:01<3:12:51, 11.64s/it]
                                                      
{'loss': 0.0081, 'grad_norm': 1.1835030317306519, 'learning_rate': 7.445451996405266e-07, 'kl': -0.0066, 'entropy': -0.0503, 'ce_loss': 0.0458, 'epoch': 0.59}

 30%|██▉       | 416/1410 [1:21:01<3:12:51, 11.64s/it]
 30%|██▉       | 417/1410 [1:21:13<3:13:14, 11.68s/it]
                                                      
{'loss': 0.0267, 'grad_norm': 1.9970979690551758, 'learning_rate': 7.437625625610807e-07, 'kl': 0.001, 'entropy': -0.0359, 'ce_loss': 0.0102, 'epoch': 0.59}

 30%|██▉       | 417/1410 [1:21:13<3:13:14, 11.68s/it]
 30%|██▉       | 418/1410 [1:21:24<3:10:03, 11.50s/it]
                                                      
{'loss': 0.0123, 'grad_norm': 1.3939279317855835, 'learning_rate': 7.429783739559373e-07, 'kl': -0.0005, 'entropy': -0.0688, 'ce_loss': 0.0147, 'epoch': 0.59}

 30%|██▉       | 418/1410 [1:21:24<3:10:03, 11.50s/it]
 30%|██▉       | 419/1410 [1:21:36<3:10:18, 11.52s/it]
                                                      
{'loss': 0.0174, 'grad_norm': 1.7200249433517456, 'learning_rate': 7.421926379668385e-07, 'kl': -0.0073, 'entropy': -0.064, 'ce_loss': 0.0278, 'epoch': 0.59}

 30%|██▉       | 419/1410 [1:21:36<3:10:18, 11.52s/it]
 30%|██▉       | 420/1410 [1:21:48<3:12:12, 11.65s/it]
                                                      
{'loss': 0.0169, 'grad_norm': 1.6951617002487183, 'learning_rate': 7.414053587436989e-07, 'kl': -0.0043, 'entropy': -0.0352, 'ce_loss': 0.0262, 'epoch': 0.6}

 30%|██▉       | 420/1410 [1:21:48<3:12:12, 11.65s/it]
 30%|██▉       | 421/1410 [1:22:00<3:13:55, 11.77s/it]
                                                      
{'loss': 0.0391, 'grad_norm': 1.9637943506240845, 'learning_rate': 7.406165404445839e-07, 'kl': -0.0088, 'entropy': 0.0581, 'ce_loss': 0.0846, 'epoch': 0.6}

 30%|██▉       | 421/1410 [1:22:00<3:13:55, 11.77s/it]
 30%|██▉       | 422/1410 [1:22:11<3:13:31, 11.75s/it]
                                                      
{'loss': 0.0247, 'grad_norm': 2.4510955810546875, 'learning_rate': 7.398261872356877e-07, 'kl': 0.0017, 'entropy': -0.0029, 'ce_loss': 0.0045, 'epoch': 0.6}

 30%|██▉       | 422/1410 [1:22:11<3:13:31, 11.75s/it]
 30%|███       | 423/1410 [1:22:23<3:12:53, 11.73s/it]
                                                      
{'loss': 0.0154, 'grad_norm': 1.555138349533081, 'learning_rate': 7.390343032913111e-07, 'kl': 0.0035, 'entropy': -0.026, 'ce_loss': 0.005, 'epoch': 0.6}

 30%|███       | 423/1410 [1:22:23<3:12:53, 11.73s/it]
 30%|███       | 424/1410 [1:22:35<3:12:51, 11.74s/it]
                                                      
{'loss': 0.0272, 'grad_norm': 1.6885591745376587, 'learning_rate': 7.382408927938395e-07, 'kl': -0.0088, 'entropy': 0.0096, 'ce_loss': 0.0969, 'epoch': 0.6}

 30%|███       | 424/1410 [1:22:35<3:12:51, 11.74s/it]
 30%|███       | 425/1410 [1:22:47<3:13:22, 11.78s/it]
                                                      
{'loss': 0.0252, 'grad_norm': 2.049530267715454, 'learning_rate': 7.374459599337212e-07, 'kl': -0.0053, 'entropy': -0.0131, 'ce_loss': 0.0668, 'epoch': 0.6}

 30%|███       | 425/1410 [1:22:47<3:13:22, 11.78s/it]
 30%|███       | 426/1410 [1:22:59<3:14:00, 11.83s/it]
                                                      
{'loss': 0.0216, 'grad_norm': 1.875067949295044, 'learning_rate': 7.36649508909445e-07, 'kl': -0.0001, 'entropy': -0.0762, 'ce_loss': 0.0035, 'epoch': 0.6}

 30%|███       | 426/1410 [1:22:59<3:14:00, 11.83s/it]
 30%|███       | 427/1410 [1:23:10<3:12:40, 11.76s/it]
                                                      
{'loss': 0.0488, 'grad_norm': 2.495471715927124, 'learning_rate': 7.358515439275172e-07, 'kl': -0.0024, 'entropy': -0.03, 'ce_loss': 0.0139, 'epoch': 0.61}

 30%|███       | 427/1410 [1:23:10<3:12:40, 11.76s/it]
 30%|███       | 428/1410 [1:23:22<3:13:03, 11.80s/it]
                                                      
{'loss': 0.0476, 'grad_norm': 2.9023728370666504, 'learning_rate': 7.350520692024409e-07, 'kl': -0.0062, 'entropy': -0.0015, 'ce_loss': 0.067, 'epoch': 0.61}

 30%|███       | 428/1410 [1:23:22<3:13:03, 11.80s/it]
 30%|███       | 429/1410 [1:23:34<3:12:39, 11.78s/it]
                                                      
{'loss': 0.0171, 'grad_norm': 1.5917972326278687, 'learning_rate': 7.34251088956693e-07, 'kl': -0.0087, 'entropy': -0.0258, 'ce_loss': 0.0586, 'epoch': 0.61}

 30%|███       | 429/1410 [1:23:34<3:12:39, 11.78s/it]
 30%|███       | 430/1410 [1:23:46<3:12:36, 11.79s/it]
                                                      
{'loss': 0.0214, 'grad_norm': 1.3719409704208374, 'learning_rate': 7.334486074207018e-07, 'kl': 0.0002, 'entropy': 0.0281, 'ce_loss': 0.1182, 'epoch': 0.61}

 30%|███       | 430/1410 [1:23:46<3:12:36, 11.79s/it]
 31%|███       | 431/1410 [1:23:57<3:12:11, 11.78s/it]
                                                      
{'loss': 0.0387, 'grad_norm': 1.844687819480896, 'learning_rate': 7.326446288328243e-07, 'kl': 0.0011, 'entropy': -0.0564, 'ce_loss': 0.019, 'epoch': 0.61}

 31%|███       | 431/1410 [1:23:57<3:12:11, 11.78s/it]
 31%|███       | 432/1410 [1:24:09<3:10:41, 11.70s/it]
                                                      
{'loss': 0.0249, 'grad_norm': 2.162864923477173, 'learning_rate': 7.318391574393248e-07, 'kl': 0.0161, 'entropy': -0.0234, 'ce_loss': 0.0075, 'epoch': 0.61}

 31%|███       | 432/1410 [1:24:09<3:10:41, 11.70s/it]
 31%|███       | 433/1410 [1:24:21<3:11:12, 11.74s/it]
                                                      
{'loss': 0.0404, 'grad_norm': 2.602679967880249, 'learning_rate': 7.310321974943519e-07, 'kl': -0.0002, 'entropy': -0.0117, 'ce_loss': 0.0002, 'epoch': 0.61}

 31%|███       | 433/1410 [1:24:21<3:11:12, 11.74s/it]
 31%|███       | 434/1410 [1:24:33<3:11:11, 11.75s/it]
                                                      
{'loss': 0.031, 'grad_norm': 2.553828716278076, 'learning_rate': 7.302237532599159e-07, 'kl': -0.0106, 'entropy': -0.0342, 'ce_loss': 0.0994, 'epoch': 0.62}

 31%|███       | 434/1410 [1:24:33<3:11:11, 11.75s/it]
 31%|███       | 435/1410 [1:24:44<3:09:00, 11.63s/it]
                                                      
{'loss': 0.0439, 'grad_norm': 2.1907601356506348, 'learning_rate': 7.294138290058667e-07, 'kl': -0.0015, 'entropy': -0.0396, 'ce_loss': 0.0024, 'epoch': 0.62}

 31%|███       | 435/1410 [1:24:44<3:09:00, 11.63s/it]
 31%|███       | 436/1410 [1:24:56<3:09:44, 11.69s/it]
                                                      
{'loss': 0.0149, 'grad_norm': 0.9040344953536987, 'learning_rate': 7.286024290098706e-07, 'kl': -0.0005, 'entropy': -0.05, 'ce_loss': 0.0121, 'epoch': 0.62}

 31%|███       | 436/1410 [1:24:56<3:09:44, 11.69s/it]
 31%|███       | 437/1410 [1:25:08<3:10:26, 11.74s/it]
                                                      
{'loss': 0.0262, 'grad_norm': 1.889780879020691, 'learning_rate': 7.277895575573886e-07, 'kl': -0.0041, 'entropy': -0.0527, 'ce_loss': 0.0307, 'epoch': 0.62}

 31%|███       | 437/1410 [1:25:08<3:10:26, 11.74s/it]
 31%|███       | 438/1410 [1:25:19<3:10:49, 11.78s/it]
                                                      
{'loss': 0.028, 'grad_norm': 1.7383602857589722, 'learning_rate': 7.269752189416531e-07, 'kl': -0.0037, 'entropy': -0.0535, 'ce_loss': 0.0135, 'epoch': 0.62}

 31%|███       | 438/1410 [1:25:19<3:10:49, 11.78s/it]
 31%|███       | 439/1410 [1:25:31<3:11:12, 11.82s/it]
                                                      
{'loss': 0.0336, 'grad_norm': 2.0620980262756348, 'learning_rate': 7.261594174636453e-07, 'kl': -0.0052, 'entropy': 0.0403, 'ce_loss': 0.0904, 'epoch': 0.62}

 31%|███       | 439/1410 [1:25:31<3:11:12, 11.82s/it]
 31%|███       | 440/1410 [1:25:43<3:10:24, 11.78s/it]
                                                      
{'loss': 0.0175, 'grad_norm': 1.7106395959854126, 'learning_rate': 7.253421574320726e-07, 'kl': 0.0015, 'entropy': -0.0089, 'ce_loss': 0.0009, 'epoch': 0.62}

 31%|███       | 440/1410 [1:25:43<3:10:24, 11.78s/it]
 31%|███▏      | 441/1410 [1:25:55<3:09:46, 11.75s/it]
                                                      
{'loss': 0.0208, 'grad_norm': 2.3172006607055664, 'learning_rate': 7.245234431633461e-07, 'kl': -0.0054, 'entropy': 0.0928, 'ce_loss': 0.1527, 'epoch': 0.63}

 31%|███▏      | 441/1410 [1:25:55<3:09:46, 11.75s/it]
 31%|███▏      | 442/1410 [1:26:06<3:09:00, 11.72s/it]
                                                      
{'loss': 0.0344, 'grad_norm': 2.5766446590423584, 'learning_rate': 7.237032789815575e-07, 'kl': -0.0009, 'entropy': -0.0496, 'ce_loss': 0.0225, 'epoch': 0.63}

 31%|███▏      | 442/1410 [1:26:06<3:09:00, 11.72s/it]
 31%|███▏      | 443/1410 [1:26:18<3:08:13, 11.68s/it]
                                                      
{'loss': 0.0293, 'grad_norm': 2.7834055423736572, 'learning_rate': 7.22881669218456e-07, 'kl': -0.0071, 'entropy': -0.0767, 'ce_loss': 0.036, 'epoch': 0.63}

 31%|███▏      | 443/1410 [1:26:18<3:08:13, 11.68s/it]
 31%|███▏      | 444/1410 [1:26:30<3:08:11, 11.69s/it]
                                                      
{'loss': 0.0244, 'grad_norm': 1.8335567712783813, 'learning_rate': 7.22058618213426e-07, 'kl': 0.0047, 'entropy': -0.016, 'ce_loss': 0.0021, 'epoch': 0.63}

 31%|███▏      | 444/1410 [1:26:30<3:08:11, 11.69s/it]
 32%|███▏      | 445/1410 [1:26:41<3:08:09, 11.70s/it]
                                                      
{'loss': 0.0236, 'grad_norm': 1.9216578006744385, 'learning_rate': 7.212341303134639e-07, 'kl': -0.0022, 'entropy': 0.0139, 'ce_loss': 0.084, 'epoch': 0.63}

 32%|███▏      | 445/1410 [1:26:41<3:08:09, 11.70s/it]
 32%|███▏      | 446/1410 [1:26:53<3:08:47, 11.75s/it]
                                                      
{'loss': 0.0194, 'grad_norm': 2.151425361633301, 'learning_rate': 7.204082098731551e-07, 'kl': -0.0095, 'entropy': -0.0053, 'ce_loss': 0.086, 'epoch': 0.63}

 32%|███▏      | 446/1410 [1:26:53<3:08:47, 11.75s/it]
 32%|███▏      | 447/1410 [1:27:04<3:05:43, 11.57s/it]
                                                      
{'loss': 0.0537, 'grad_norm': 2.7734432220458984, 'learning_rate': 7.195808612546508e-07, 'kl': 0.008, 'entropy': -0.0415, 'ce_loss': 0.006, 'epoch': 0.63}

 32%|███▏      | 447/1410 [1:27:04<3:05:43, 11.57s/it]
 32%|███▏      | 448/1410 [1:27:16<3:06:23, 11.63s/it]
                                                      
{'loss': 0.0166, 'grad_norm': 1.596106767654419, 'learning_rate': 7.187520888276457e-07, 'kl': 0.0065, 'entropy': -0.0388, 'ce_loss': 0.0115, 'epoch': 0.64}

 32%|███▏      | 448/1410 [1:27:16<3:06:23, 11.63s/it]
 32%|███▏      | 449/1410 [1:27:28<3:06:17, 11.63s/it]
                                                      
{'loss': 0.0265, 'grad_norm': 2.2252233028411865, 'learning_rate': 7.179218969693542e-07, 'kl': -0.0004, 'entropy': -0.0354, 'ce_loss': 0.0066, 'epoch': 0.64}

 32%|███▏      | 449/1410 [1:27:28<3:06:17, 11.63s/it]
 32%|███▏      | 450/1410 [1:27:40<3:07:03, 11.69s/it]
                                                      
{'loss': 0.018, 'grad_norm': 1.131973147392273, 'learning_rate': 7.170902900644876e-07, 'kl': -0.0002, 'entropy': 0.1128, 'ce_loss': 0.1149, 'epoch': 0.64}

 32%|███▏      | 450/1410 [1:27:40<3:07:03, 11.69s/it]
 32%|███▏      | 451/1410 [1:27:52<3:07:36, 11.74s/it]
                                                      
{'loss': 0.046, 'grad_norm': 2.057371139526367, 'learning_rate': 7.162572725052302e-07, 'kl': -0.0014, 'entropy': 0.0723, 'ce_loss': 0.0852, 'epoch': 0.64}

 32%|███▏      | 451/1410 [1:27:52<3:07:36, 11.74s/it]
 32%|███▏      | 452/1410 [1:28:03<3:08:16, 11.79s/it]
                                                      
{'loss': 0.0178, 'grad_norm': 1.3466570377349854, 'learning_rate': 7.154228486912181e-07, 'kl': -0.0052, 'entropy': 0.0095, 'ce_loss': 0.1024, 'epoch': 0.64}

 32%|███▏      | 452/1410 [1:28:03<3:08:16, 11.79s/it]
 32%|███▏      | 453/1410 [1:28:15<3:07:18, 11.74s/it]
                                                      
{'loss': 0.0404, 'grad_norm': 2.340275526046753, 'learning_rate': 7.145870230295135e-07, 'kl': -0.0009, 'entropy': -0.0471, 'ce_loss': 0.0215, 'epoch': 0.64}

 32%|███▏      | 453/1410 [1:28:15<3:07:18, 11.74s/it]
 32%|███▏      | 454/1410 [1:28:26<3:03:56, 11.54s/it]
                                                      
{'loss': 0.0205, 'grad_norm': 2.10520339012146, 'learning_rate': 7.137497999345828e-07, 'kl': 0.0104, 'entropy': -0.041, 'ce_loss': 0.0052, 'epoch': 0.64}

 32%|███▏      | 454/1410 [1:28:26<3:03:56, 11.54s/it]
 32%|███▏      | 455/1410 [1:28:37<2:59:37, 11.29s/it]
                                                      
{'loss': 0.0349, 'grad_norm': 2.2727506160736084, 'learning_rate': 7.129111838282733e-07, 'kl': -0.0116, 'entropy': -0.0593, 'ce_loss': 0.0535, 'epoch': 0.65}

 32%|███▏      | 455/1410 [1:28:37<2:59:37, 11.29s/it]
 32%|███▏      | 456/1410 [1:28:48<2:59:42, 11.30s/it]
                                                      
{'loss': 0.0291, 'grad_norm': 2.7535290718078613, 'learning_rate': 7.120711791397894e-07, 'kl': -0.0079, 'entropy': -0.0205, 'ce_loss': 0.0295, 'epoch': 0.65}

 32%|███▏      | 456/1410 [1:28:48<2:59:42, 11.30s/it]
 32%|███▏      | 457/1410 [1:29:00<3:01:43, 11.44s/it]
                                                      
{'loss': 0.0132, 'grad_norm': 1.3303486108779907, 'learning_rate': 7.112297903056694e-07, 'kl': -0.0031, 'entropy': -0.0776, 'ce_loss': 0.0243, 'epoch': 0.65}

 32%|███▏      | 457/1410 [1:29:00<3:01:43, 11.44s/it]
 32%|███▏      | 458/1410 [1:29:12<3:03:31, 11.57s/it]
                                                      
{'loss': 0.0212, 'grad_norm': 1.6422302722930908, 'learning_rate': 7.103870217697621e-07, 'kl': -0.008, 'entropy': 0.027, 'ce_loss': 0.0918, 'epoch': 0.65}

 32%|███▏      | 458/1410 [1:29:12<3:03:31, 11.57s/it]
 33%|███▎      | 459/1410 [1:29:24<3:04:56, 11.67s/it]
                                                      
{'loss': 0.0218, 'grad_norm': 1.906676173210144, 'learning_rate': 7.09542877983203e-07, 'kl': -0.0025, 'entropy': -0.0245, 'ce_loss': 0.0431, 'epoch': 0.65}

 33%|███▎      | 459/1410 [1:29:24<3:04:56, 11.67s/it]
 33%|███▎      | 460/1410 [1:29:35<3:01:25, 11.46s/it]
                                                      
{'loss': 0.0303, 'grad_norm': 2.425304412841797, 'learning_rate': 7.086973634043916e-07, 'kl': 0.0081, 'entropy': -0.0576, 'ce_loss': 0.0165, 'epoch': 0.65}

 33%|███▎      | 460/1410 [1:29:35<3:01:25, 11.46s/it]
 33%|███▎      | 461/1410 [1:29:46<3:02:21, 11.53s/it]
                                                      
{'loss': 0.0205, 'grad_norm': 2.109041452407837, 'learning_rate': 7.078504824989669e-07, 'kl': 0.0016, 'entropy': -0.0128, 'ce_loss': 0.0007, 'epoch': 0.65}

 33%|███▎      | 461/1410 [1:29:46<3:02:21, 11.53s/it]
 33%|███▎      | 462/1410 [1:29:57<2:59:17, 11.35s/it]
                                                      
{'loss': 0.0126, 'grad_norm': 1.757538914680481, 'learning_rate': 7.070022397397844e-07, 'kl': -0.0092, 'entropy': -0.0403, 'ce_loss': 0.0331, 'epoch': 0.66}

 33%|███▎      | 462/1410 [1:29:57<2:59:17, 11.35s/it]
 33%|███▎      | 463/1410 [1:30:09<3:00:07, 11.41s/it]
                                                      
{'loss': 0.0284, 'grad_norm': 2.5643093585968018, 'learning_rate': 7.061526396068921e-07, 'kl': 0.003, 'entropy': -0.0095, 'ce_loss': 0.0015, 'epoch': 0.66}

 33%|███▎      | 463/1410 [1:30:09<3:00:07, 11.41s/it]
 33%|███▎      | 464/1410 [1:30:20<2:59:32, 11.39s/it]
                                                      
{'loss': 0.0446, 'grad_norm': 2.7942826747894287, 'learning_rate': 7.053016865875077e-07, 'kl': 0.0013, 'entropy': -0.0513, 'ce_loss': 0.0285, 'epoch': 0.66}

 33%|███▎      | 464/1410 [1:30:20<2:59:32, 11.39s/it]
 33%|███▎      | 465/1410 [1:30:31<2:56:10, 11.19s/it]
                                                      
{'loss': 0.0361, 'grad_norm': 2.82483172416687, 'learning_rate': 7.044493851759934e-07, 'kl': 0.0115, 'entropy': -0.0815, 'ce_loss': 0.0156, 'epoch': 0.66}

 33%|███▎      | 465/1410 [1:30:31<2:56:10, 11.19s/it]
 33%|███▎      | 466/1410 [1:30:43<2:58:37, 11.35s/it]
                                                      
{'loss': 0.0259, 'grad_norm': 2.330594062805176, 'learning_rate': 7.035957398738336e-07, 'kl': -0.0005, 'entropy': -0.0608, 'ce_loss': 0.0215, 'epoch': 0.66}

 33%|███▎      | 466/1410 [1:30:43<2:58:37, 11.35s/it]
 33%|███▎      | 467/1410 [1:30:55<3:00:55, 11.51s/it]
                                                      
{'loss': 0.0336, 'grad_norm': 1.627484679222107, 'learning_rate': 7.027407551896105e-07, 'kl': -0.0003, 'entropy': -0.0166, 'ce_loss': 0.0024, 'epoch': 0.66}

 33%|███▎      | 467/1410 [1:30:55<3:00:55, 11.51s/it]
 33%|███▎      | 468/1410 [1:31:05<2:57:48, 11.32s/it]
                                                      
{'loss': 0.0281, 'grad_norm': 2.3063082695007324, 'learning_rate': 7.018844356389799e-07, 'kl': -0.0014, 'entropy': -0.0544, 'ce_loss': 0.0454, 'epoch': 0.66}

 33%|███▎      | 468/1410 [1:31:05<2:57:48, 11.32s/it]
 33%|███▎      | 469/1410 [1:31:17<2:59:45, 11.46s/it]
                                                      
{'loss': 0.0366, 'grad_norm': 2.2260916233062744, 'learning_rate': 7.010267857446484e-07, 'kl': -0.0074, 'entropy': -0.0535, 'ce_loss': 0.0371, 'epoch': 0.67}

 33%|███▎      | 469/1410 [1:31:17<2:59:45, 11.46s/it]
 33%|███▎      | 470/1410 [1:31:29<3:01:02, 11.56s/it]
                                                      
{'loss': 0.0333, 'grad_norm': 2.1696412563323975, 'learning_rate': 7.001678100363482e-07, 'kl': -0.0084, 'entropy': -0.015, 'ce_loss': 0.1147, 'epoch': 0.67}

 33%|███▎      | 470/1410 [1:31:29<3:01:02, 11.56s/it]
 33%|███▎      | 471/1410 [1:31:39<2:53:44, 11.10s/it]
                                                      
{'loss': 0.0293, 'grad_norm': 2.551046371459961, 'learning_rate': 6.993075130508145e-07, 'kl': -0.0019, 'entropy': -0.0728, 'ce_loss': 0.0342, 'epoch': 0.67}

 33%|███▎      | 471/1410 [1:31:39<2:53:44, 11.10s/it]
 33%|███▎      | 472/1410 [1:31:50<2:54:54, 11.19s/it]
                                                      
{'loss': 0.0321, 'grad_norm': 2.2865657806396484, 'learning_rate': 6.984458993317609e-07, 'kl': -0.0067, 'entropy': -0.0713, 'ce_loss': 0.0384, 'epoch': 0.67}

 33%|███▎      | 472/1410 [1:31:50<2:54:54, 11.19s/it]
 34%|███▎      | 473/1410 [1:32:02<2:57:39, 11.38s/it]
                                                      
{'loss': 0.0314, 'grad_norm': 1.9282550811767578, 'learning_rate': 6.975829734298548e-07, 'kl': -0.0028, 'entropy': -0.1079, 'ce_loss': 0.0245, 'epoch': 0.67}

 34%|███▎      | 473/1410 [1:32:02<2:57:39, 11.38s/it]
 34%|███▎      | 474/1410 [1:32:14<2:59:39, 11.52s/it]
                                                      
{'loss': 0.0224, 'grad_norm': 2.03281831741333, 'learning_rate': 6.967187399026943e-07, 'kl': -0.0031, 'entropy': -0.0537, 'ce_loss': 0.0438, 'epoch': 0.67}

 34%|███▎      | 474/1410 [1:32:14<2:59:39, 11.52s/it]
 34%|███▎      | 475/1410 [1:32:26<3:00:47, 11.60s/it]
                                                      
{'loss': 0.0469, 'grad_norm': 2.2547671794891357, 'learning_rate': 6.958532033147841e-07, 'kl': -0.0007, 'entropy': -0.0194, 'ce_loss': 0.0042, 'epoch': 0.67}

 34%|███▎      | 475/1410 [1:32:26<3:00:47, 11.60s/it]
 34%|███▍      | 476/1410 [1:32:38<3:01:55, 11.69s/it]
                                                      
{'loss': 0.0277, 'grad_norm': 2.18046498298645, 'learning_rate': 6.949863682375109e-07, 'kl': 0.0102, 'entropy': -0.0474, 'ce_loss': 0.0231, 'epoch': 0.68}

 34%|███▍      | 476/1410 [1:32:38<3:01:55, 11.69s/it]
 34%|███▍      | 477/1410 [1:32:50<3:02:28, 11.74s/it]
                                                      
{'loss': 0.0315, 'grad_norm': 2.5954220294952393, 'learning_rate': 6.941182392491191e-07, 'kl': 0.0004, 'entropy': -0.0742, 'ce_loss': 0.0111, 'epoch': 0.68}

 34%|███▍      | 477/1410 [1:32:50<3:02:28, 11.74s/it]
 34%|███▍      | 478/1410 [1:33:01<3:00:53, 11.65s/it]
                                                      
{'loss': 0.0345, 'grad_norm': 2.5036635398864746, 'learning_rate': 6.932488209346875e-07, 'kl': -0.0027, 'entropy': -0.0737, 'ce_loss': 0.0298, 'epoch': 0.68}

 34%|███▍      | 478/1410 [1:33:01<3:00:53, 11.65s/it]
 34%|███▍      | 479/1410 [1:33:13<3:01:46, 11.72s/it]
                                                      
{'loss': 0.0301, 'grad_norm': 2.169154405593872, 'learning_rate': 6.923781178861044e-07, 'kl': -0.0115, 'entropy': -0.0654, 'ce_loss': 0.0264, 'epoch': 0.68}

 34%|███▍      | 479/1410 [1:33:13<3:01:46, 11.72s/it]
 34%|███▍      | 480/1410 [1:33:25<3:03:23, 11.83s/it]
                                                      
{'loss': 0.0506, 'grad_norm': 2.3810040950775146, 'learning_rate': 6.915061347020433e-07, 'kl': -0.0095, 'entropy': -0.0049, 'ce_loss': 0.072, 'epoch': 0.68}

 34%|███▍      | 480/1410 [1:33:25<3:03:23, 11.83s/it]
 34%|███▍      | 481/1410 [1:33:37<3:03:18, 11.84s/it]
                                                      
{'loss': 0.0586, 'grad_norm': 2.334956169128418, 'learning_rate': 6.90632875987939e-07, 'kl': -0.0037, 'entropy': -0.0549, 'ce_loss': 0.0419, 'epoch': 0.68}

 34%|███▍      | 481/1410 [1:33:37<3:03:18, 11.84s/it]
 34%|███▍      | 482/1410 [1:33:48<2:59:48, 11.63s/it]
                                                      
{'loss': 0.0341, 'grad_norm': 2.467493772506714, 'learning_rate': 6.89758346355963e-07, 'kl': 0.0061, 'entropy': -0.0374, 'ce_loss': 0.008, 'epoch': 0.68}

 34%|███▍      | 482/1410 [1:33:48<2:59:48, 11.63s/it]
 34%|███▍      | 483/1410 [1:34:00<3:00:57, 11.71s/it]
                                                      
{'loss': 0.0575, 'grad_norm': 2.2830162048339844, 'learning_rate': 6.888825504249995e-07, 'kl': 0.0015, 'entropy': -0.0791, 'ce_loss': 0.0113, 'epoch': 0.69}

 34%|███▍      | 483/1410 [1:34:00<3:00:57, 11.71s/it]
 34%|███▍      | 484/1410 [1:34:11<2:57:33, 11.50s/it]
                                                      
{'loss': 0.0366, 'grad_norm': 2.27901029586792, 'learning_rate': 6.880054928206205e-07, 'kl': 0.0014, 'entropy': -0.0206, 'ce_loss': 0.0002, 'epoch': 0.69}

 34%|███▍      | 484/1410 [1:34:11<2:57:33, 11.50s/it]
 34%|███▍      | 485/1410 [1:34:23<2:59:17, 11.63s/it]
                                                      
{'loss': 0.03, 'grad_norm': 1.830018401145935, 'learning_rate': 6.871271781750613e-07, 'kl': -0.0047, 'entropy': -0.0325, 'ce_loss': 0.046, 'epoch': 0.69}

 34%|███▍      | 485/1410 [1:34:23<2:59:17, 11.63s/it]
 34%|███▍      | 486/1410 [1:34:34<2:58:53, 11.62s/it]
                                                      
{'loss': 0.0246, 'grad_norm': 2.1868889331817627, 'learning_rate': 6.86247611127197e-07, 'kl': -0.0042, 'entropy': -0.025, 'ce_loss': 0.0132, 'epoch': 0.69}

 34%|███▍      | 486/1410 [1:34:34<2:58:53, 11.62s/it]
 35%|███▍      | 487/1410 [1:34:46<2:58:39, 11.61s/it]
                                                      
{'loss': 0.0249, 'grad_norm': 1.8760014772415161, 'learning_rate': 6.85366796322517e-07, 'kl': -0.0073, 'entropy': -0.0315, 'ce_loss': 0.0348, 'epoch': 0.69}

 35%|███▍      | 487/1410 [1:34:46<2:58:39, 11.61s/it]
 35%|███▍      | 488/1410 [1:34:58<2:59:07, 11.66s/it]
                                                      
{'loss': 0.0212, 'grad_norm': 2.1378509998321533, 'learning_rate': 6.844847384131007e-07, 'kl': 0.0018, 'entropy': -0.0618, 'ce_loss': 0.0224, 'epoch': 0.69}

 35%|███▍      | 488/1410 [1:34:58<2:59:07, 11.66s/it]
 35%|███▍      | 489/1410 [1:35:10<2:59:50, 11.72s/it]
                                                      
{'loss': 0.0233, 'grad_norm': 2.0261619091033936, 'learning_rate': 6.836014420575933e-07, 'kl': -0.0088, 'entropy': 0.012, 'ce_loss': 0.074, 'epoch': 0.69}

 35%|███▍      | 489/1410 [1:35:10<2:59:50, 11.72s/it]
 35%|███▍      | 490/1410 [1:35:21<2:59:34, 11.71s/it]
                                                      
{'loss': 0.0112, 'grad_norm': 1.5037927627563477, 'learning_rate': 6.827169119211807e-07, 'kl': -0.0012, 'entropy': -0.0674, 'ce_loss': 0.0395, 'epoch': 0.7}

 35%|███▍      | 490/1410 [1:35:21<2:59:34, 11.71s/it]
 35%|███▍      | 491/1410 [1:35:33<3:00:07, 11.76s/it]
                                                      
{'loss': 0.05, 'grad_norm': 2.8132240772247314, 'learning_rate': 6.818311526755651e-07, 'kl': 0.0074, 'entropy': -0.0354, 'ce_loss': 0.0141, 'epoch': 0.7}

 35%|███▍      | 491/1410 [1:35:33<3:00:07, 11.76s/it]
 35%|███▍      | 492/1410 [1:35:45<3:00:06, 11.77s/it]
                                                      
{'loss': 0.0267, 'grad_norm': 1.8065080642700195, 'learning_rate': 6.809441689989408e-07, 'kl': 0.0013, 'entropy': -0.0126, 'ce_loss': 0.0007, 'epoch': 0.7}

 35%|███▍      | 492/1410 [1:35:45<3:00:06, 11.77s/it]
 35%|███▍      | 493/1410 [1:35:57<2:59:53, 11.77s/it]
                                                      
{'loss': 0.0351, 'grad_norm': 2.775426149368286, 'learning_rate': 6.800559655759681e-07, 'kl': -0.0023, 'entropy': -0.0299, 'ce_loss': 0.0016, 'epoch': 0.7}

 35%|███▍      | 493/1410 [1:35:57<2:59:53, 11.77s/it]
 35%|███▌      | 494/1410 [1:36:08<2:59:06, 11.73s/it]
                                                      
{'loss': 0.0231, 'grad_norm': 2.3288891315460205, 'learning_rate': 6.791665470977503e-07, 'kl': -0.0006, 'entropy': -0.0496, 'ce_loss': 0.0084, 'epoch': 0.7}

 35%|███▌      | 494/1410 [1:36:08<2:59:06, 11.73s/it]
 35%|███▌      | 495/1410 [1:36:20<2:58:49, 11.73s/it]
                                                      
{'loss': 0.058, 'grad_norm': 1.925110936164856, 'learning_rate': 6.78275918261808e-07, 'kl': -0.0013, 'entropy': -0.0693, 'ce_loss': 0.017, 'epoch': 0.7}

 35%|███▌      | 495/1410 [1:36:20<2:58:49, 11.73s/it]
 35%|███▌      | 496/1410 [1:36:32<2:57:29, 11.65s/it]
                                                      
{'loss': 0.0272, 'grad_norm': 2.2979507446289062, 'learning_rate': 6.773840837720537e-07, 'kl': 0.0003, 'entropy': -0.0242, 'ce_loss': 0.0011, 'epoch': 0.7}

 35%|███▌      | 496/1410 [1:36:32<2:57:29, 11.65s/it]
 35%|███▌      | 497/1410 [1:36:44<2:58:26, 11.73s/it]
                                                      
{'loss': 0.0338, 'grad_norm': 2.3894221782684326, 'learning_rate': 6.764910483387682e-07, 'kl': 0.0013, 'entropy': -0.0359, 'ce_loss': 0.0051, 'epoch': 0.7}

 35%|███▌      | 497/1410 [1:36:44<2:58:26, 11.73s/it]
 35%|███▌      | 498/1410 [1:36:55<2:58:07, 11.72s/it]
                                                      
{'loss': 0.0161, 'grad_norm': 1.736132264137268, 'learning_rate': 6.755968166785753e-07, 'kl': -0.0006, 'entropy': -0.0608, 'ce_loss': 0.0214, 'epoch': 0.71}

 35%|███▌      | 498/1410 [1:36:55<2:58:07, 11.72s/it]
 35%|███▌      | 499/1410 [1:37:07<2:57:41, 11.70s/it]
                                                      
{'loss': 0.0189, 'grad_norm': 1.8899191617965698, 'learning_rate': 6.747013935144163e-07, 'kl': -0.0035, 'entropy': -0.0674, 'ce_loss': 0.0412, 'epoch': 0.71}

 35%|███▌      | 499/1410 [1:37:07<2:57:41, 11.70s/it]
 35%|███▌      | 500/1410 [1:37:19<2:57:33, 11.71s/it]
                                                      
{'loss': 0.0274, 'grad_norm': 2.0919930934906006, 'learning_rate': 6.738047835755257e-07, 'kl': -0.0031, 'entropy': -0.0447, 'ce_loss': 0.0201, 'epoch': 0.71}

 35%|███▌      | 500/1410 [1:37:19<2:57:33, 11.71s/it]
 36%|███▌      | 501/1410 [1:37:30<2:56:52, 11.68s/it]
                                                      
{'loss': 0.0213, 'grad_norm': 2.5028679370880127, 'learning_rate': 6.729069915974062e-07, 'kl': 0.0025, 'entropy': -0.0238, 'ce_loss': 0.0033, 'epoch': 0.71}

 36%|███▌      | 501/1410 [1:37:30<2:56:52, 11.68s/it]
 36%|███▌      | 502/1410 [1:37:42<2:55:45, 11.61s/it]
                                                      
{'loss': 0.0392, 'grad_norm': 2.5599987506866455, 'learning_rate': 6.720080223218033e-07, 'kl': -0.0012, 'entropy': -0.0613, 'ce_loss': 0.0143, 'epoch': 0.71}

 36%|███▌      | 502/1410 [1:37:42<2:55:45, 11.61s/it]
 36%|███▌      | 503/1410 [1:37:53<2:55:42, 11.62s/it]
                                                      
{'loss': 0.0095, 'grad_norm': 1.0909353494644165, 'learning_rate': 6.711078804966806e-07, 'kl': -0.0096, 'entropy': -0.0422, 'ce_loss': 0.034, 'epoch': 0.71}

 36%|███▌      | 503/1410 [1:37:53<2:55:42, 11.62s/it]
 36%|███▌      | 504/1410 [1:38:05<2:55:32, 11.62s/it]
                                                      
{'loss': 0.0304, 'grad_norm': 2.0937304496765137, 'learning_rate': 6.702065708761941e-07, 'kl': -0.0089, 'entropy': 0.1143, 'ce_loss': 0.1402, 'epoch': 0.71}

 36%|███▌      | 504/1410 [1:38:05<2:55:32, 11.62s/it]
 36%|███▌      | 505/1410 [1:38:17<2:55:42, 11.65s/it]
                                                      
{'loss': 0.0217, 'grad_norm': 1.806587815284729, 'learning_rate': 6.693040982206685e-07, 'kl': -0.0028, 'entropy': -0.0549, 'ce_loss': 0.0457, 'epoch': 0.72}

 36%|███▌      | 505/1410 [1:38:17<2:55:42, 11.65s/it]
 36%|███▌      | 506/1410 [1:38:28<2:55:38, 11.66s/it]
                                                      
{'loss': 0.0204, 'grad_norm': 2.34892201423645, 'learning_rate': 6.684004672965705e-07, 'kl': 0.0044, 'entropy': -0.0282, 'ce_loss': 0.0201, 'epoch': 0.72}

 36%|███▌      | 506/1410 [1:38:28<2:55:38, 11.66s/it]
 36%|███▌      | 507/1410 [1:38:40<2:55:11, 11.64s/it]
                                                      
{'loss': 0.0388, 'grad_norm': 2.2986598014831543, 'learning_rate': 6.674956828764845e-07, 'kl': -0.0005, 'entropy': -0.0576, 'ce_loss': 0.0319, 'epoch': 0.72}

 36%|███▌      | 507/1410 [1:38:40<2:55:11, 11.64s/it]
 36%|███▌      | 508/1410 [1:38:52<2:55:37, 11.68s/it]
                                                      
{'loss': 0.0276, 'grad_norm': 1.942150592803955, 'learning_rate': 6.665897497390868e-07, 'kl': -0.0016, 'entropy': -0.0295, 'ce_loss': 0.0017, 'epoch': 0.72}

 36%|███▌      | 508/1410 [1:38:52<2:55:37, 11.68s/it]
 36%|███▌      | 509/1410 [1:39:03<2:55:49, 11.71s/it]
                                                      
{'loss': 0.0323, 'grad_norm': 2.001157283782959, 'learning_rate': 6.656826726691212e-07, 'kl': -0.0034, 'entropy': -0.0525, 'ce_loss': 0.0227, 'epoch': 0.72}

 36%|███▌      | 509/1410 [1:39:04<2:55:49, 11.71s/it]
 36%|███▌      | 510/1410 [1:39:15<2:55:14, 11.68s/it]
                                                      
{'loss': 0.0304, 'grad_norm': 2.569131851196289, 'learning_rate': 6.647744564573731e-07, 'kl': -0.0037, 'entropy': 0.0723, 'ce_loss': 0.1433, 'epoch': 0.72}

 36%|███▌      | 510/1410 [1:39:15<2:55:14, 11.68s/it]
 36%|███▌      | 511/1410 [1:39:27<2:55:31, 11.72s/it]
                                                      
{'loss': 0.0171, 'grad_norm': 1.719732642173767, 'learning_rate': 6.638651059006443e-07, 'kl': -0.0034, 'entropy': 0.0669, 'ce_loss': 0.16, 'epoch': 0.72}

 36%|███▌      | 511/1410 [1:39:27<2:55:31, 11.72s/it]
 36%|███▋      | 512/1410 [1:39:39<2:56:13, 11.77s/it]
                                                      
{'loss': 0.0252, 'grad_norm': 1.9206371307373047, 'learning_rate': 6.629546258017276e-07, 'kl': -0.007, 'entropy': -0.042, 'ce_loss': 0.0296, 'epoch': 0.73}

 36%|███▋      | 512/1410 [1:39:39<2:56:13, 11.77s/it]
 36%|███▋      | 513/1410 [1:39:51<2:56:31, 11.81s/it]
                                                      
{'loss': 0.0263, 'grad_norm': 2.1810569763183594, 'learning_rate': 6.620430209693816e-07, 'kl': 0.003, 'entropy': -0.0737, 'ce_loss': 0.0225, 'epoch': 0.73}

 36%|███▋      | 513/1410 [1:39:51<2:56:31, 11.81s/it]
 36%|███▋      | 514/1410 [1:40:03<2:56:39, 11.83s/it]
                                                      
{'loss': 0.0228, 'grad_norm': 1.7998206615447998, 'learning_rate': 6.611302962183057e-07, 'kl': -0.0007, 'entropy': 0.0825, 'ce_loss': 0.0868, 'epoch': 0.73}

 36%|███▋      | 514/1410 [1:40:03<2:56:39, 11.83s/it]
 37%|███▋      | 515/1410 [1:40:14<2:56:15, 11.82s/it]
                                                      
{'loss': 0.0209, 'grad_norm': 1.4100327491760254, 'learning_rate': 6.602164563691136e-07, 'kl': -0.0056, 'entropy': 0.0051, 'ce_loss': 0.0881, 'epoch': 0.73}

 37%|███▋      | 515/1410 [1:40:14<2:56:15, 11.82s/it]
 37%|███▋      | 516/1410 [1:40:26<2:54:46, 11.73s/it]
                                                      
{'loss': 0.0266, 'grad_norm': 1.944717526435852, 'learning_rate': 6.593015062483086e-07, 'kl': -0.006, 'entropy': 0.0165, 'ce_loss': 0.0863, 'epoch': 0.73}

 37%|███▋      | 516/1410 [1:40:26<2:54:46, 11.73s/it]
 37%|███▋      | 517/1410 [1:40:38<2:54:09, 11.70s/it]
                                                      
{'loss': 0.0268, 'grad_norm': 1.9371775388717651, 'learning_rate': 6.583854506882584e-07, 'kl': -0.0062, 'entropy': -0.0139, 'ce_loss': 0.0456, 'epoch': 0.73}

 37%|███▋      | 517/1410 [1:40:38<2:54:09, 11.70s/it]
 37%|███▋      | 518/1410 [1:40:49<2:52:31, 11.60s/it]
                                                      
{'loss': 0.0276, 'grad_norm': 2.1519176959991455, 'learning_rate': 6.574682945271683e-07, 'kl': -0.0036, 'entropy': -0.0291, 'ce_loss': 0.014, 'epoch': 0.73}

 37%|███▋      | 518/1410 [1:40:49<2:52:31, 11.60s/it]
 37%|███▋      | 519/1410 [1:41:00<2:50:18, 11.47s/it]
                                                      
{'loss': 0.0227, 'grad_norm': 2.130770683288574, 'learning_rate': 6.565500426090577e-07, 'kl': -0.0072, 'entropy': -0.0515, 'ce_loss': 0.0179, 'epoch': 0.74}

 37%|███▋      | 519/1410 [1:41:00<2:50:18, 11.47s/it]
 37%|███▋      | 520/1410 [1:41:12<2:51:10, 11.54s/it]
                                                      
{'loss': 0.0414, 'grad_norm': 2.4019014835357666, 'learning_rate': 6.556306997837322e-07, 'kl': -0.0069, 'entropy': -0.0026, 'ce_loss': 0.0964, 'epoch': 0.74}

 37%|███▋      | 520/1410 [1:41:12<2:51:10, 11.54s/it]
 37%|███▋      | 521/1410 [1:41:24<2:52:10, 11.62s/it]
                                                      
{'loss': 0.0298, 'grad_norm': 2.266098737716675, 'learning_rate': 6.547102709067594e-07, 'kl': -0.004, 'entropy': 0.0146, 'ce_loss': 0.0536, 'epoch': 0.74}

 37%|███▋      | 521/1410 [1:41:24<2:52:10, 11.62s/it]
 37%|███▋      | 522/1410 [1:41:35<2:50:28, 11.52s/it]
                                                      
{'loss': 0.024, 'grad_norm': 2.3482227325439453, 'learning_rate': 6.537887608394435e-07, 'kl': -0.0092, 'entropy': 0.0299, 'ce_loss': 0.1158, 'epoch': 0.74}

 37%|███▋      | 522/1410 [1:41:35<2:50:28, 11.52s/it]
 37%|███▋      | 523/1410 [1:41:47<2:51:28, 11.60s/it]
                                                      
{'loss': 0.0267, 'grad_norm': 1.7567193508148193, 'learning_rate': 6.528661744487984e-07, 'kl': -0.0032, 'entropy': -0.0422, 'ce_loss': 0.0175, 'epoch': 0.74}

 37%|███▋      | 523/1410 [1:41:47<2:51:28, 11.60s/it]
 37%|███▋      | 524/1410 [1:41:58<2:52:06, 11.65s/it]
                                                      
{'loss': 0.0343, 'grad_norm': 2.5552854537963867, 'learning_rate': 6.51942516607523e-07, 'kl': -0.0082, 'entropy': 0.1206, 'ce_loss': 0.1859, 'epoch': 0.74}

 37%|███▋      | 524/1410 [1:41:58<2:52:06, 11.65s/it]
 37%|███▋      | 525/1410 [1:42:10<2:50:26, 11.55s/it]
                                                      
{'loss': 0.0266, 'grad_norm': 1.7570756673812866, 'learning_rate': 6.510177921939749e-07, 'kl': -0.0005, 'entropy': -0.0085, 'ce_loss': 0.0004, 'epoch': 0.74}

 37%|███▋      | 525/1410 [1:42:10<2:50:26, 11.55s/it]
 37%|███▋      | 526/1410 [1:42:21<2:51:01, 11.61s/it]
                                                      
{'loss': 0.0403, 'grad_norm': 2.8151965141296387, 'learning_rate': 6.500920060921454e-07, 'kl': 0.0014, 'entropy': -0.0938, 'ce_loss': 0.0312, 'epoch': 0.75}

 37%|███▋      | 526/1410 [1:42:21<2:51:01, 11.61s/it]
 37%|███▋      | 527/1410 [1:42:33<2:49:08, 11.49s/it]
                                                      
{'loss': 0.0378, 'grad_norm': 2.712097644805908, 'learning_rate': 6.491651631916324e-07, 'kl': 0.0058, 'entropy': -0.0391, 'ce_loss': 0.0196, 'epoch': 0.75}

 37%|███▋      | 527/1410 [1:42:33<2:49:08, 11.49s/it]
 37%|███▋      | 528/1410 [1:42:44<2:47:25, 11.39s/it]
                                                      
{'loss': 0.0233, 'grad_norm': 2.2010343074798584, 'learning_rate': 6.482372683876158e-07, 'kl': 0.0067, 'entropy': -0.0439, 'ce_loss': 0.0087, 'epoch': 0.75}

 37%|███▋      | 528/1410 [1:42:44<2:47:25, 11.39s/it]
 38%|███▊      | 529/1410 [1:42:56<2:49:27, 11.54s/it]
                                                      
{'loss': 0.0288, 'grad_norm': 2.5631778240203857, 'learning_rate': 6.473083265808315e-07, 'kl': -0.0085, 'entropy': -0.0488, 'ce_loss': 0.0379, 'epoch': 0.75}

 38%|███▊      | 529/1410 [1:42:56<2:49:27, 11.54s/it]
 38%|███▊      | 530/1410 [1:43:08<2:50:57, 11.66s/it]
                                                      
{'loss': 0.0222, 'grad_norm': 1.6199802160263062, 'learning_rate': 6.463783426775446e-07, 'kl': -0.0099, 'entropy': 0.0569, 'ce_loss': 0.096, 'epoch': 0.75}

 38%|███▊      | 530/1410 [1:43:08<2:50:57, 11.66s/it]
 38%|███▊      | 531/1410 [1:43:19<2:49:57, 11.60s/it]
                                                      
{'loss': 0.0277, 'grad_norm': 1.9183859825134277, 'learning_rate': 6.454473215895243e-07, 'kl': -0.0011, 'entropy': -0.0554, 'ce_loss': 0.0162, 'epoch': 0.75}

 38%|███▊      | 531/1410 [1:43:19<2:49:57, 11.60s/it]
 38%|███▊      | 532/1410 [1:43:31<2:50:27, 11.65s/it]
                                                      
{'loss': 0.0328, 'grad_norm': 2.2873716354370117, 'learning_rate': 6.445152682340178e-07, 'kl': 0.0205, 'entropy': -0.0801, 'ce_loss': 0.0191, 'epoch': 0.75}

 38%|███▊      | 532/1410 [1:43:31<2:50:27, 11.65s/it]
 38%|███▊      | 533/1410 [1:43:43<2:51:03, 11.70s/it]
                                                      
{'loss': 0.0256, 'grad_norm': 1.9595588445663452, 'learning_rate': 6.435821875337245e-07, 'kl': -0.0099, 'entropy': -0.019, 'ce_loss': 0.0902, 'epoch': 0.76}

 38%|███▊      | 533/1410 [1:43:43<2:51:03, 11.70s/it]
 38%|███▊      | 534/1410 [1:43:55<2:52:35, 11.82s/it]
                                                      
{'loss': 0.0286, 'grad_norm': 1.1226797103881836, 'learning_rate': 6.426480844167696e-07, 'kl': -0.0003, 'entropy': -0.0137, 'ce_loss': 0.0005, 'epoch': 0.76}

 38%|███▊      | 534/1410 [1:43:55<2:52:35, 11.82s/it]
 38%|███▊      | 535/1410 [1:44:07<2:51:59, 11.79s/it]
                                                      
{'loss': 0.0243, 'grad_norm': 2.3692386150360107, 'learning_rate': 6.41712963816678e-07, 'kl': -0.0033, 'entropy': 0.0493, 'ce_loss': 0.0987, 'epoch': 0.76}

 38%|███▊      | 535/1410 [1:44:07<2:51:59, 11.79s/it]
 38%|███▊      | 536/1410 [1:44:18<2:50:14, 11.69s/it]
                                                      
{'loss': 0.0293, 'grad_norm': 2.1379306316375732, 'learning_rate': 6.407768306723489e-07, 'kl': -0.0057, 'entropy': 0.0171, 'ce_loss': 0.0919, 'epoch': 0.76}

 38%|███▊      | 536/1410 [1:44:18<2:50:14, 11.69s/it]
 38%|███▊      | 537/1410 [1:44:30<2:51:03, 11.76s/it]
                                                      
{'loss': 0.0441, 'grad_norm': 2.4283716678619385, 'learning_rate': 6.398396899280293e-07, 'kl': -0.0106, 'entropy': 0.0996, 'ce_loss': 0.1349, 'epoch': 0.76}

 38%|███▊      | 537/1410 [1:44:30<2:51:03, 11.76s/it]
 38%|███▊      | 538/1410 [1:44:42<2:50:48, 11.75s/it]
                                                      
{'loss': 0.0223, 'grad_norm': 1.8983925580978394, 'learning_rate': 6.389015465332874e-07, 'kl': 0.0157, 'entropy': -0.0422, 'ce_loss': 0.0083, 'epoch': 0.76}

 38%|███▊      | 538/1410 [1:44:42<2:50:48, 11.75s/it]
 38%|███▊      | 539/1410 [1:44:53<2:50:25, 11.74s/it]
                                                      
{'loss': 0.0228, 'grad_norm': 2.0490517616271973, 'learning_rate': 6.379624054429876e-07, 'kl': -0.0033, 'entropy': 0.0422, 'ce_loss': 0.1005, 'epoch': 0.76}

 38%|███▊      | 539/1410 [1:44:53<2:50:25, 11.74s/it]
 38%|███▊      | 540/1410 [1:45:05<2:49:57, 11.72s/it]
                                                      
{'loss': 0.0295, 'grad_norm': 2.3664391040802, 'learning_rate': 6.370222716172635e-07, 'kl': 0.0009, 'entropy': 0.1011, 'ce_loss': 0.0896, 'epoch': 0.77}

 38%|███▊      | 540/1410 [1:45:05<2:49:57, 11.72s/it]
 38%|███▊      | 541/1410 [1:45:17<2:49:35, 11.71s/it]
                                                      
{'loss': 0.0218, 'grad_norm': 2.2613813877105713, 'learning_rate': 6.360811500214917e-07, 'kl': -0.0025, 'entropy': -0.0596, 'ce_loss': 0.0301, 'epoch': 0.77}

 38%|███▊      | 541/1410 [1:45:17<2:49:35, 11.71s/it]
 38%|███▊      | 542/1410 [1:45:28<2:49:16, 11.70s/it]
                                                      
{'loss': 0.0279, 'grad_norm': 2.259427309036255, 'learning_rate': 6.35139045626266e-07, 'kl': -0.0051, 'entropy': -0.0645, 'ce_loss': 0.0279, 'epoch': 0.77}

 38%|███▊      | 542/1410 [1:45:28<2:49:16, 11.70s/it]
 39%|███▊      | 543/1410 [1:45:40<2:49:40, 11.74s/it]
                                                      
{'loss': 0.023, 'grad_norm': 1.559601902961731, 'learning_rate': 6.341959634073705e-07, 'kl': 0.0008, 'entropy': -0.1035, 'ce_loss': 0.0222, 'epoch': 0.77}

 39%|███▊      | 543/1410 [1:45:40<2:49:40, 11.74s/it]
 39%|███▊      | 544/1410 [1:45:52<2:50:41, 11.83s/it]
                                                      
{'loss': 0.0176, 'grad_norm': 1.6391867399215698, 'learning_rate': 6.332519083457545e-07, 'kl': -0.0101, 'entropy': -0.011, 'ce_loss': 0.0546, 'epoch': 0.77}

 39%|███▊      | 544/1410 [1:45:52<2:50:41, 11.83s/it]
 39%|███▊      | 545/1410 [1:46:04<2:50:12, 11.81s/it]
                                                      
{'loss': 0.0252, 'grad_norm': 1.5913337469100952, 'learning_rate': 6.323068854275045e-07, 'kl': 0.0006, 'entropy': -0.0242, 'ce_loss': 0.0006, 'epoch': 0.77}

 39%|███▊      | 545/1410 [1:46:04<2:50:12, 11.81s/it]
 39%|███▊      | 546/1410 [1:46:16<2:50:19, 11.83s/it]
                                                      
{'loss': 0.0324, 'grad_norm': 2.406446933746338, 'learning_rate': 6.313608996438196e-07, 'kl': -0.0009, 'entropy': 0.0364, 'ce_loss': 0.0994, 'epoch': 0.77}

 39%|███▊      | 546/1410 [1:46:16<2:50:19, 11.83s/it]
 39%|███▉      | 547/1410 [1:46:28<2:50:47, 11.87s/it]
                                                      
{'loss': 0.0254, 'grad_norm': 1.9924955368041992, 'learning_rate': 6.304139559909839e-07, 'kl': -0.0063, 'entropy': -0.0253, 'ce_loss': 0.0215, 'epoch': 0.78}

 39%|███▉      | 547/1410 [1:46:28<2:50:47, 11.87s/it]
 39%|███▉      | 548/1410 [1:46:39<2:48:21, 11.72s/it]
                                                      
{'loss': 0.0189, 'grad_norm': 1.6608295440673828, 'learning_rate': 6.294660594703406e-07, 'kl': -0.0032, 'entropy': -0.0364, 'ce_loss': 0.0126, 'epoch': 0.78}

 39%|███▉      | 548/1410 [1:46:39<2:48:21, 11.72s/it]
 39%|███▉      | 549/1410 [1:46:51<2:49:11, 11.79s/it]
                                                      
{'loss': 0.0449, 'grad_norm': 2.349205255508423, 'learning_rate': 6.285172150882659e-07, 'kl': -0.0084, 'entropy': -0.0649, 'ce_loss': 0.0339, 'epoch': 0.78}

 39%|███▉      | 549/1410 [1:46:51<2:49:11, 11.79s/it]
 39%|███▉      | 550/1410 [1:47:03<2:49:22, 11.82s/it]
                                                      
{'loss': 0.0397, 'grad_norm': 2.048190116882324, 'learning_rate': 6.275674278561416e-07, 'kl': -0.0008, 'entropy': -0.05, 'ce_loss': 0.0243, 'epoch': 0.78}

 39%|███▉      | 550/1410 [1:47:03<2:49:22, 11.82s/it]
 39%|███▉      | 551/1410 [1:47:15<2:48:58, 11.80s/it]
                                                      
{'loss': 0.022, 'grad_norm': 1.877851963043213, 'learning_rate': 6.266167027903295e-07, 'kl': 0.0028, 'entropy': -0.0908, 'ce_loss': 0.0409, 'epoch': 0.78}

 39%|███▉      | 551/1410 [1:47:15<2:48:58, 11.80s/it]
 39%|███▉      | 552/1410 [1:47:27<2:48:19, 11.77s/it]
                                                      
{'loss': 0.0292, 'grad_norm': 2.522261619567871, 'learning_rate': 6.256650449121446e-07, 'kl': -0.0023, 'entropy': 0.1201, 'ce_loss': 0.1751, 'epoch': 0.78}

 39%|███▉      | 552/1410 [1:47:27<2:48:19, 11.77s/it]
 39%|███▉      | 553/1410 [1:47:38<2:44:59, 11.55s/it]
                                                      
{'loss': 0.0326, 'grad_norm': 2.4258172512054443, 'learning_rate': 6.247124592478288e-07, 'kl': -0.0054, 'entropy': -0.0645, 'ce_loss': 0.0418, 'epoch': 0.78}

 39%|███▉      | 553/1410 [1:47:38<2:44:59, 11.55s/it]
 39%|███▉      | 554/1410 [1:47:49<2:45:08, 11.58s/it]
                                                      
{'loss': 0.0164, 'grad_norm': 1.7609374523162842, 'learning_rate': 6.23758950828524e-07, 'kl': -0.0012, 'entropy': -0.0154, 'ce_loss': 0.0035, 'epoch': 0.79}

 39%|███▉      | 554/1410 [1:47:49<2:45:08, 11.58s/it]
 39%|███▉      | 555/1410 [1:48:01<2:45:59, 11.65s/it]
                                                      
{'loss': 0.0322, 'grad_norm': 2.196741819381714, 'learning_rate': 6.228045246902454e-07, 'kl': -0.0077, 'entropy': -0.0569, 'ce_loss': 0.0451, 'epoch': 0.79}

 39%|███▉      | 555/1410 [1:48:01<2:45:59, 11.65s/it]
 39%|███▉      | 556/1410 [1:48:13<2:46:30, 11.70s/it]
                                                      
{'loss': 0.0264, 'grad_norm': 2.065279006958008, 'learning_rate': 6.218491858738559e-07, 'kl': -0.0018, 'entropy': -0.0403, 'ce_loss': 0.0172, 'epoch': 0.79}

 39%|███▉      | 556/1410 [1:48:13<2:46:30, 11.70s/it]
 40%|███▉      | 557/1410 [1:48:24<2:45:20, 11.63s/it]
                                                      
{'loss': 0.0281, 'grad_norm': 2.1368296146392822, 'learning_rate': 6.20892939425038e-07, 'kl': -0.0014, 'entropy': 0.0179, 'ce_loss': 0.093, 'epoch': 0.79}

 40%|███▉      | 557/1410 [1:48:24<2:45:20, 11.63s/it]
 40%|███▉      | 558/1410 [1:48:36<2:45:27, 11.65s/it]
                                                      
{'loss': 0.0483, 'grad_norm': 2.1733498573303223, 'learning_rate': 6.199357903942685e-07, 'kl': 0.0002, 'entropy': -0.0143, 'ce_loss': 0.0024, 'epoch': 0.79}

 40%|███▉      | 558/1410 [1:48:36<2:45:27, 11.65s/it]
 40%|███▉      | 559/1410 [1:48:48<2:45:49, 11.69s/it]
                                                      
{'loss': 0.0141, 'grad_norm': 1.4313536882400513, 'learning_rate': 6.189777438367909e-07, 'kl': -0.0017, 'entropy': 0.1475, 'ce_loss': 0.161, 'epoch': 0.79}

 40%|███▉      | 559/1410 [1:48:48<2:45:49, 11.69s/it]
 40%|███▉      | 560/1410 [1:48:59<2:44:52, 11.64s/it]
                                                      
{'loss': 0.0218, 'grad_norm': 1.7407170534133911, 'learning_rate': 6.180188048125893e-07, 'kl': -0.0061, 'entropy': -0.0388, 'ce_loss': 0.0258, 'epoch': 0.79}

 40%|███▉      | 560/1410 [1:48:59<2:44:52, 11.64s/it]
 40%|███▉      | 561/1410 [1:49:11<2:44:13, 11.61s/it]
                                                      
{'loss': 0.041, 'grad_norm': 2.306109666824341, 'learning_rate': 6.170589783863609e-07, 'kl': 0.0027, 'entropy': -0.084, 'ce_loss': 0.0131, 'epoch': 0.8}

 40%|███▉      | 561/1410 [1:49:11<2:44:13, 11.61s/it]
 40%|███▉      | 562/1410 [1:49:22<2:43:55, 11.60s/it]
                                                      
{'loss': 0.0301, 'grad_norm': 2.453903913497925, 'learning_rate': 6.160982696274906e-07, 'kl': 0.0011, 'entropy': -0.0152, 'ce_loss': 0.0012, 'epoch': 0.8}

 40%|███▉      | 562/1410 [1:49:22<2:43:55, 11.60s/it]
 40%|███▉      | 563/1410 [1:49:34<2:44:29, 11.65s/it]
                                                      
{'loss': 0.0266, 'grad_norm': 2.54028058052063, 'learning_rate': 6.151366836100227e-07, 'kl': -0.0017, 'entropy': -0.0952, 'ce_loss': 0.0205, 'epoch': 0.8}

 40%|███▉      | 563/1410 [1:49:34<2:44:29, 11.65s/it]
 40%|████      | 564/1410 [1:49:45<2:41:47, 11.47s/it]
                                                      
{'loss': 0.0373, 'grad_norm': 2.653311252593994, 'learning_rate': 6.14174225412635e-07, 'kl': -0.004, 'entropy': -0.0432, 'ce_loss': 0.037, 'epoch': 0.8}

 40%|████      | 564/1410 [1:49:45<2:41:47, 11.47s/it]
 40%|████      | 565/1410 [1:49:57<2:41:16, 11.45s/it]
                                                      
{'loss': 0.0287, 'grad_norm': 1.7778351306915283, 'learning_rate': 6.13210900118612e-07, 'kl': 0.0004, 'entropy': -0.0732, 'ce_loss': 0.0316, 'epoch': 0.8}

 40%|████      | 565/1410 [1:49:57<2:41:16, 11.45s/it]
 40%|████      | 566/1410 [1:50:09<2:42:53, 11.58s/it]
                                                      
{'loss': 0.0365, 'grad_norm': 2.048731803894043, 'learning_rate': 6.122467128158174e-07, 'kl': -0.0031, 'entropy': 0.0425, 'ce_loss': 0.09, 'epoch': 0.8}

 40%|████      | 566/1410 [1:50:09<2:42:53, 11.58s/it]
 40%|████      | 567/1410 [1:50:20<2:42:16, 11.55s/it]
                                                      
{'loss': 0.0188, 'grad_norm': 1.9849278926849365, 'learning_rate': 6.112816685966682e-07, 'kl': -0.0102, 'entropy': -0.0021, 'ce_loss': 0.08, 'epoch': 0.8}

 40%|████      | 567/1410 [1:50:20<2:42:16, 11.55s/it]
 40%|████      | 568/1410 [1:50:32<2:43:08, 11.63s/it]
                                                      
{'loss': 0.0236, 'grad_norm': 1.7907172441482544, 'learning_rate': 6.103157725581067e-07, 'kl': -0.0107, 'entropy': -0.0635, 'ce_loss': 0.0445, 'epoch': 0.81}

 40%|████      | 568/1410 [1:50:32<2:43:08, 11.63s/it]
 40%|████      | 569/1410 [1:50:44<2:43:53, 11.69s/it]
                                                      
{'loss': 0.0218, 'grad_norm': 1.4834519624710083, 'learning_rate': 6.093490298015744e-07, 'kl': -0.0078, 'entropy': 0.1099, 'ce_loss': 0.1091, 'epoch': 0.81}

 40%|████      | 569/1410 [1:50:44<2:43:53, 11.69s/it]
 40%|████      | 570/1410 [1:50:55<2:43:14, 11.66s/it]
                                                      
{'loss': 0.0203, 'grad_norm': 1.7853785753250122, 'learning_rate': 6.083814454329849e-07, 'kl': -0.0074, 'entropy': -0.0457, 'ce_loss': 0.0567, 'epoch': 0.81}

 40%|████      | 570/1410 [1:50:55<2:43:14, 11.66s/it]
 40%|████      | 571/1410 [1:51:07<2:42:39, 11.63s/it]
                                                      
{'loss': 0.0234, 'grad_norm': 2.141012191772461, 'learning_rate': 6.074130245626967e-07, 'kl': -0.0076, 'entropy': -0.0505, 'ce_loss': 0.026, 'epoch': 0.81}

 40%|████      | 571/1410 [1:51:07<2:42:39, 11.63s/it]
 41%|████      | 572/1410 [1:51:18<2:41:09, 11.54s/it]
                                                      
{'loss': 0.0388, 'grad_norm': 2.2987003326416016, 'learning_rate': 6.064437723054863e-07, 'kl': -0.0065, 'entropy': -0.0188, 'ce_loss': 0.0686, 'epoch': 0.81}

 41%|████      | 572/1410 [1:51:18<2:41:09, 11.54s/it]
 41%|████      | 573/1410 [1:51:30<2:43:16, 11.70s/it]
                                                      
{'loss': 0.0461, 'grad_norm': 2.386967182159424, 'learning_rate': 6.054736937805213e-07, 'kl': -0.0041, 'entropy': -0.0179, 'ce_loss': 0.0068, 'epoch': 0.81}

 41%|████      | 573/1410 [1:51:30<2:43:16, 11.70s/it]
 41%|████      | 574/1410 [1:51:42<2:43:38, 11.75s/it]
                                                      
{'loss': 0.0263, 'grad_norm': 1.662050485610962, 'learning_rate': 6.045027941113332e-07, 'kl': -0.0029, 'entropy': -0.0564, 'ce_loss': 0.0109, 'epoch': 0.81}

 41%|████      | 574/1410 [1:51:42<2:43:38, 11.75s/it]
 41%|████      | 575/1410 [1:51:54<2:43:27, 11.75s/it]
                                                      
{'loss': 0.0324, 'grad_norm': 2.068293333053589, 'learning_rate': 6.035310784257906e-07, 'kl': -0.0091, 'entropy': 0.0535, 'ce_loss': 0.1001, 'epoch': 0.82}

 41%|████      | 575/1410 [1:51:54<2:43:27, 11.75s/it]
 41%|████      | 576/1410 [1:52:05<2:42:43, 11.71s/it]
                                                      
{'loss': 0.0432, 'grad_norm': 3.0051379203796387, 'learning_rate': 6.025585518560719e-07, 'kl': 0.0126, 'entropy': -0.0439, 'ce_loss': 0.0351, 'epoch': 0.82}

 41%|████      | 576/1410 [1:52:05<2:42:43, 11.71s/it]
 41%|████      | 577/1410 [1:52:17<2:43:45, 11.80s/it]
                                                      
{'loss': 0.0461, 'grad_norm': 2.1602156162261963, 'learning_rate': 6.015852195386379e-07, 'kl': -0.01, 'entropy': -0.0569, 'ce_loss': 0.0326, 'epoch': 0.82}

 41%|████      | 577/1410 [1:52:17<2:43:45, 11.80s/it]
 41%|████      | 578/1410 [1:52:29<2:43:31, 11.79s/it]
                                                      
{'loss': 0.0078, 'grad_norm': 1.0576986074447632, 'learning_rate': 6.006110866142053e-07, 'kl': -0.0082, 'entropy': 0.0432, 'ce_loss': 0.1211, 'epoch': 0.82}

 41%|████      | 578/1410 [1:52:29<2:43:31, 11.79s/it]
 41%|████      | 579/1410 [1:52:41<2:42:24, 11.73s/it]
                                                      
{'loss': 0.0165, 'grad_norm': 1.490641713142395, 'learning_rate': 5.996361582277193e-07, 'kl': 0.0031, 'entropy': -0.0693, 'ce_loss': 0.0204, 'epoch': 0.82}

 41%|████      | 579/1410 [1:52:41<2:42:24, 11.73s/it]
 41%|████      | 580/1410 [1:52:53<2:42:48, 11.77s/it]
                                                      
{'loss': 0.0298, 'grad_norm': 1.4631919860839844, 'learning_rate': 5.986604395283265e-07, 'kl': -0.0075, 'entropy': 0.0091, 'ce_loss': 0.0872, 'epoch': 0.82}

 41%|████      | 580/1410 [1:52:53<2:42:48, 11.77s/it]
 41%|████      | 581/1410 [1:53:05<2:43:00, 11.80s/it]
                                                      
{'loss': 0.0403, 'grad_norm': 2.3061435222625732, 'learning_rate': 5.976839356693471e-07, 'kl': 0.007, 'entropy': -0.0413, 'ce_loss': 0.0096, 'epoch': 0.82}

 41%|████      | 581/1410 [1:53:05<2:43:00, 11.80s/it]
 41%|████▏     | 582/1410 [1:53:16<2:42:27, 11.77s/it]
                                                      
{'loss': 0.0249, 'grad_norm': 2.136716604232788, 'learning_rate': 5.967066518082486e-07, 'kl': -0.002, 'entropy': -0.0923, 'ce_loss': 0.0265, 'epoch': 0.83}

 41%|████▏     | 582/1410 [1:53:16<2:42:27, 11.77s/it]
 41%|████▏     | 583/1410 [1:53:28<2:42:32, 11.79s/it]
                                                      
{'loss': 0.0346, 'grad_norm': 1.7945911884307861, 'learning_rate': 5.957285931066183e-07, 'kl': -0.001, 'entropy': -0.0216, 'ce_loss': 0.0038, 'epoch': 0.83}

 41%|████▏     | 583/1410 [1:53:28<2:42:32, 11.79s/it]
 41%|████▏     | 584/1410 [1:53:40<2:42:30, 11.80s/it]
                                                      
{'loss': 0.0333, 'grad_norm': 2.075713872909546, 'learning_rate': 5.94749764730135e-07, 'kl': -0.0028, 'entropy': 0.0168, 'ce_loss': 0.1057, 'epoch': 0.83}

 41%|████▏     | 584/1410 [1:53:40<2:42:30, 11.80s/it]
 41%|████▏     | 585/1410 [1:53:52<2:42:12, 11.80s/it]
                                                      
{'loss': 0.0425, 'grad_norm': 2.4830615520477295, 'learning_rate': 5.937701718485435e-07, 'kl': -0.0029, 'entropy': -0.0479, 'ce_loss': 0.0088, 'epoch': 0.83}

 41%|████▏     | 585/1410 [1:53:52<2:42:12, 11.80s/it]
 42%|████▏     | 586/1410 [1:54:03<2:41:45, 11.78s/it]
                                                      
{'loss': 0.016, 'grad_norm': 1.6952855587005615, 'learning_rate': 5.927898196356259e-07, 'kl': -0.0079, 'entropy': -0.05, 'ce_loss': 0.0346, 'epoch': 0.83}

 42%|████▏     | 586/1410 [1:54:03<2:41:45, 11.78s/it]
 42%|████▏     | 587/1410 [1:54:15<2:41:35, 11.78s/it]
                                                      
{'loss': 0.0179, 'grad_norm': 2.1048405170440674, 'learning_rate': 5.91808713269175e-07, 'kl': -0.0005, 'entropy': -0.0356, 'ce_loss': 0.0021, 'epoch': 0.83}

 42%|████▏     | 587/1410 [1:54:15<2:41:35, 11.78s/it]
 42%|████▏     | 588/1410 [1:54:27<2:40:31, 11.72s/it]
                                                      
{'loss': 0.0265, 'grad_norm': 2.1970531940460205, 'learning_rate': 5.908268579309666e-07, 'kl': 0.0103, 'entropy': -0.0282, 'ce_loss': 0.0047, 'epoch': 0.83}

 42%|████▏     | 588/1410 [1:54:27<2:40:31, 11.72s/it]
 42%|████▏     | 589/1410 [1:54:38<2:39:52, 11.68s/it]
                                                      
{'loss': 0.0154, 'grad_norm': 1.4944103956222534, 'learning_rate': 5.898442588067323e-07, 'kl': 0.0027, 'entropy': -0.014, 'ce_loss': 0.0001, 'epoch': 0.84}

 42%|████▏     | 589/1410 [1:54:38<2:39:52, 11.68s/it]
 42%|████▏     | 590/1410 [1:54:50<2:40:13, 11.72s/it]
                                                      
{'loss': 0.0262, 'grad_norm': 1.8182913064956665, 'learning_rate': 5.888609210861319e-07, 'kl': -0.0004, 'entropy': -0.0796, 'ce_loss': 0.0174, 'epoch': 0.84}

 42%|████▏     | 590/1410 [1:54:50<2:40:13, 11.72s/it]
 42%|████▏     | 591/1410 [1:55:02<2:39:39, 11.70s/it]
                                                      
{'loss': 0.0163, 'grad_norm': 1.6029361486434937, 'learning_rate': 5.878768499627263e-07, 'kl': -0.0049, 'entropy': -0.0757, 'ce_loss': 0.0362, 'epoch': 0.84}

 42%|████▏     | 591/1410 [1:55:02<2:39:39, 11.70s/it]
 42%|████▏     | 592/1410 [1:55:13<2:38:34, 11.63s/it]
                                                      
{'loss': 0.0265, 'grad_norm': 2.4055299758911133, 'learning_rate': 5.868920506339497e-07, 'kl': -0.0124, 'entropy': -0.0, 'ce_loss': 0.0676, 'epoch': 0.84}

 42%|████▏     | 592/1410 [1:55:13<2:38:34, 11.63s/it]
 42%|████▏     | 593/1410 [1:55:25<2:39:02, 11.68s/it]
                                                      
{'loss': 0.0167, 'grad_norm': 1.791098713874817, 'learning_rate': 5.859065283010828e-07, 'kl': -0.0087, 'entropy': 0.0918, 'ce_loss': 0.1779, 'epoch': 0.84}

 42%|████▏     | 593/1410 [1:55:25<2:39:02, 11.68s/it]
 42%|████▏     | 594/1410 [1:55:37<2:39:45, 11.75s/it]
                                                      
{'loss': 0.0348, 'grad_norm': 2.1117031574249268, 'learning_rate': 5.849202881692245e-07, 'kl': -0.0057, 'entropy': 0.0874, 'ce_loss': 0.1158, 'epoch': 0.84}

 42%|████▏     | 594/1410 [1:55:37<2:39:45, 11.75s/it]
 42%|████▏     | 595/1410 [1:55:49<2:40:35, 11.82s/it]
                                                      
{'loss': 0.0347, 'grad_norm': 2.0887856483459473, 'learning_rate': 5.839333354472648e-07, 'kl': -0.0032, 'entropy': -0.0003, 'ce_loss': 0.0295, 'epoch': 0.84}

 42%|████▏     | 595/1410 [1:55:49<2:40:35, 11.82s/it]
 42%|████▏     | 596/1410 [1:56:01<2:40:17, 11.81s/it]
                                                      
{'loss': 0.0221, 'grad_norm': 1.5122356414794922, 'learning_rate': 5.829456753478574e-07, 'kl': -0.0055, 'entropy': 0.0483, 'ce_loss': 0.0915, 'epoch': 0.85}

 42%|████▏     | 596/1410 [1:56:01<2:40:17, 11.81s/it]
 42%|████▏     | 597/1410 [1:56:12<2:39:26, 11.77s/it]
                                                      
{'loss': 0.0292, 'grad_norm': 1.7365453243255615, 'learning_rate': 5.819573130873924e-07, 'kl': -0.0068, 'entropy': -0.0684, 'ce_loss': 0.0412, 'epoch': 0.85}

 42%|████▏     | 597/1410 [1:56:12<2:39:26, 11.77s/it]
 42%|████▏     | 598/1410 [1:56:24<2:38:11, 11.69s/it]
                                                      
{'loss': 0.0161, 'grad_norm': 1.583348035812378, 'learning_rate': 5.809682538859677e-07, 'kl': -0.0014, 'entropy': -0.0613, 'ce_loss': 0.0608, 'epoch': 0.85}

 42%|████▏     | 598/1410 [1:56:24<2:38:11, 11.69s/it]
 42%|████▏     | 599/1410 [1:56:36<2:37:41, 11.67s/it]
                                                      
{'loss': 0.0251, 'grad_norm': 1.8823504447937012, 'learning_rate': 5.799785029673627e-07, 'kl': 0.0037, 'entropy': -0.0038, 'ce_loss': 0.0002, 'epoch': 0.85}

 42%|████▏     | 599/1410 [1:56:36<2:37:41, 11.67s/it]
 43%|████▎     | 600/1410 [1:56:47<2:37:53, 11.70s/it]
                                                      
{'loss': 0.0252, 'grad_norm': 1.880802035331726, 'learning_rate': 5.789880655590098e-07, 'kl': 0.0046, 'entropy': -0.053, 'ce_loss': 0.0121, 'epoch': 0.85}

 43%|████▎     | 600/1410 [1:56:47<2:37:53, 11.70s/it]
 43%|████▎     | 601/1410 [1:56:59<2:38:11, 11.73s/it]
                                                      
{'loss': 0.0462, 'grad_norm': 2.4831485748291016, 'learning_rate': 5.779969468919674e-07, 'kl': -0.0081, 'entropy': -0.0071, 'ce_loss': 0.1315, 'epoch': 0.85}

 43%|████▎     | 601/1410 [1:56:59<2:38:11, 11.73s/it]
 43%|████▎     | 602/1410 [1:57:11<2:36:28, 11.62s/it]
                                                      
{'loss': 0.0292, 'grad_norm': 2.4062061309814453, 'learning_rate': 5.77005152200892e-07, 'kl': -0.0103, 'entropy': -0.0728, 'ce_loss': 0.0285, 'epoch': 0.85}

 43%|████▎     | 602/1410 [1:57:11<2:36:28, 11.62s/it]
 43%|████▎     | 603/1410 [1:57:23<2:37:39, 11.72s/it]
                                                      
{'loss': 0.014, 'grad_norm': 1.0801901817321777, 'learning_rate': 5.760126867240103e-07, 'kl': -0.0136, 'entropy': -0.011, 'ce_loss': 0.1058, 'epoch': 0.86}

 43%|████▎     | 603/1410 [1:57:23<2:37:39, 11.72s/it]
 43%|████▎     | 604/1410 [1:57:34<2:37:46, 11.75s/it]
                                                      
{'loss': 0.0224, 'grad_norm': 1.565490961074829, 'learning_rate': 5.750195557030921e-07, 'kl': 0.0008, 'entropy': -0.0264, 'ce_loss': 0.0011, 'epoch': 0.86}

 43%|████▎     | 604/1410 [1:57:34<2:37:46, 11.75s/it]
 43%|████▎     | 605/1410 [1:57:46<2:36:32, 11.67s/it]
                                                      
{'loss': 0.0222, 'grad_norm': 1.9822520017623901, 'learning_rate': 5.740257643834222e-07, 'kl': 0.0025, 'entropy': -0.0369, 'ce_loss': 0.0196, 'epoch': 0.86}

 43%|████▎     | 605/1410 [1:57:46<2:36:32, 11.67s/it]
 43%|████▎     | 606/1410 [1:57:58<2:37:49, 11.78s/it]
                                                      
{'loss': 0.0482, 'grad_norm': 2.527941942214966, 'learning_rate': 5.730313180137727e-07, 'kl': -0.0064, 'entropy': 0.082, 'ce_loss': 0.1103, 'epoch': 0.86}

 43%|████▎     | 606/1410 [1:57:58<2:37:49, 11.78s/it]
 43%|████▎     | 607/1410 [1:58:09<2:36:41, 11.71s/it]
                                                      
{'loss': 0.0258, 'grad_norm': 2.0539844036102295, 'learning_rate': 5.720362218463755e-07, 'kl': 0.0014, 'entropy': -0.0188, 'ce_loss': 0.0021, 'epoch': 0.86}

 43%|████▎     | 607/1410 [1:58:09<2:36:41, 11.71s/it]
 43%|████▎     | 608/1410 [1:58:21<2:35:40, 11.65s/it]
                                                      
{'loss': 0.0192, 'grad_norm': 1.5815198421478271, 'learning_rate': 5.710404811368945e-07, 'kl': -0.0025, 'entropy': -0.0469, 'ce_loss': 0.0262, 'epoch': 0.86}

 43%|████▎     | 608/1410 [1:58:21<2:35:40, 11.65s/it]
 43%|████▎     | 609/1410 [1:58:33<2:35:30, 11.65s/it]
                                                      
{'loss': 0.0138, 'grad_norm': 1.2807446718215942, 'learning_rate': 5.700441011443977e-07, 'kl': 0.0036, 'entropy': -0.0918, 'ce_loss': 0.0396, 'epoch': 0.86}

 43%|████▎     | 609/1410 [1:58:33<2:35:30, 11.65s/it]
 43%|████▎     | 610/1410 [1:58:44<2:36:24, 11.73s/it]
                                                      
{'loss': 0.0237, 'grad_norm': 2.086012601852417, 'learning_rate': 5.690470871313295e-07, 'kl': -0.0063, 'entropy': 0.012, 'ce_loss': 0.0792, 'epoch': 0.87}

 43%|████▎     | 610/1410 [1:58:44<2:36:24, 11.73s/it]
 43%|████▎     | 611/1410 [1:58:56<2:35:12, 11.66s/it]
                                                      
{'loss': 0.0385, 'grad_norm': 2.634516716003418, 'learning_rate': 5.68049444363483e-07, 'kl': -0.0022, 'entropy': -0.0349, 'ce_loss': 0.0259, 'epoch': 0.87}

 43%|████▎     | 611/1410 [1:58:56<2:35:12, 11.66s/it]
 43%|████▎     | 612/1410 [1:59:07<2:33:36, 11.55s/it]
                                                      
{'loss': 0.0267, 'grad_norm': 2.284311532974243, 'learning_rate': 5.67051178109972e-07, 'kl': -0.0035, 'entropy': -0.0479, 'ce_loss': 0.0355, 'epoch': 0.87}

 43%|████▎     | 612/1410 [1:59:07<2:33:36, 11.55s/it]
 43%|████▎     | 613/1410 [1:59:19<2:34:23, 11.62s/it]
                                                      
{'loss': 0.0272, 'grad_norm': 1.8127756118774414, 'learning_rate': 5.660522936432033e-07, 'kl': 0.005, 'entropy': -0.0178, 'ce_loss': 0.0006, 'epoch': 0.87}

 43%|████▎     | 613/1410 [1:59:19<2:34:23, 11.62s/it]
 44%|████▎     | 614/1410 [1:59:31<2:33:49, 11.59s/it]
                                                      
{'loss': 0.0202, 'grad_norm': 1.900926113128662, 'learning_rate': 5.650527962388491e-07, 'kl': -0.0021, 'entropy': -0.0569, 'ce_loss': 0.0271, 'epoch': 0.87}

 44%|████▎     | 614/1410 [1:59:31<2:33:49, 11.59s/it]
 44%|████▎     | 615/1410 [1:59:42<2:34:32, 11.66s/it]
                                                      
{'loss': 0.0511, 'grad_norm': 2.6691641807556152, 'learning_rate': 5.640526911758184e-07, 'kl': -0.0006, 'entropy': -0.05, 'ce_loss': 0.0136, 'epoch': 0.87}

 44%|████▎     | 615/1410 [1:59:42<2:34:32, 11.66s/it]
 44%|████▎     | 616/1410 [1:59:54<2:34:29, 11.67s/it]
                                                      
{'loss': 0.0495, 'grad_norm': 2.133441209793091, 'learning_rate': 5.630519837362299e-07, 'kl': -0.0069, 'entropy': -0.05, 'ce_loss': 0.0235, 'epoch': 0.87}

 44%|████▎     | 616/1410 [1:59:54<2:34:29, 11.67s/it]
 44%|████▍     | 617/1410 [2:00:06<2:35:10, 11.74s/it]
                                                      
{'loss': 0.0403, 'grad_norm': 2.55595326423645, 'learning_rate': 5.620506792053839e-07, 'kl': -0.0073, 'entropy': 0.0047, 'ce_loss': 0.1098, 'epoch': 0.88}

 44%|████▍     | 617/1410 [2:00:06<2:35:10, 11.74s/it]
 44%|████▍     | 618/1410 [2:00:17<2:32:12, 11.53s/it]
                                                      
{'loss': 0.0515, 'grad_norm': 3.3934054374694824, 'learning_rate': 5.610487828717338e-07, 'kl': -0.0002, 'entropy': -0.0693, 'ce_loss': 0.0215, 'epoch': 0.88}

 44%|████▍     | 618/1410 [2:00:17<2:32:12, 11.53s/it]
 44%|████▍     | 619/1410 [2:00:29<2:32:46, 11.59s/it]
                                                      
{'loss': 0.0358, 'grad_norm': 1.7285938262939453, 'learning_rate': 5.600463000268594e-07, 'kl': 0.0075, 'entropy': -0.0947, 'ce_loss': 0.0435, 'epoch': 0.88}

 44%|████▍     | 619/1410 [2:00:29<2:32:46, 11.59s/it]
 44%|████▍     | 620/1410 [2:00:40<2:32:49, 11.61s/it]
                                                      
{'loss': 0.0358, 'grad_norm': 2.6884078979492188, 'learning_rate': 5.590432359654371e-07, 'kl': 0.0026, 'entropy': -0.0391, 'ce_loss': 0.0074, 'epoch': 0.88}

 44%|████▍     | 620/1410 [2:00:40<2:32:49, 11.61s/it]
 44%|████▍     | 621/1410 [2:00:51<2:29:16, 11.35s/it]
                                                      
{'loss': 0.0404, 'grad_norm': 2.814727544784546, 'learning_rate': 5.580395959852142e-07, 'kl': -0.0022, 'entropy': -0.0786, 'ce_loss': 0.0351, 'epoch': 0.88}

 44%|████▍     | 621/1410 [2:00:51<2:29:16, 11.35s/it]
 44%|████▍     | 622/1410 [2:01:02<2:27:36, 11.24s/it]
                                                      
{'loss': 0.0324, 'grad_norm': 2.418732166290283, 'learning_rate': 5.57035385386979e-07, 'kl': -0.0061, 'entropy': -0.0674, 'ce_loss': 0.0141, 'epoch': 0.88}

 44%|████▍     | 622/1410 [2:01:02<2:27:36, 11.24s/it]
 44%|████▍     | 623/1410 [2:01:14<2:29:53, 11.43s/it]
                                                      
{'loss': 0.037, 'grad_norm': 2.3175148963928223, 'learning_rate': 5.560306094745338e-07, 'kl': -0.0061, 'entropy': 0.0091, 'ce_loss': 0.0769, 'epoch': 0.88}

 44%|████▍     | 623/1410 [2:01:14<2:29:53, 11.43s/it]
 44%|████▍     | 624/1410 [2:01:25<2:29:00, 11.37s/it]
                                                      
{'loss': 0.0461, 'grad_norm': 3.135500192642212, 'learning_rate': 5.550252735546667e-07, 'kl': -0.007, 'entropy': -0.0049, 'ce_loss': 0.1169, 'epoch': 0.89}

 44%|████▍     | 624/1410 [2:01:25<2:29:00, 11.37s/it]
 44%|████▍     | 625/1410 [2:01:36<2:27:30, 11.27s/it]
                                                      
{'loss': 0.0322, 'grad_norm': 2.6008338928222656, 'learning_rate': 5.540193829371234e-07, 'kl': -0.001, 'entropy': -0.0231, 'ce_loss': 0.0025, 'epoch': 0.89}

 44%|████▍     | 625/1410 [2:01:36<2:27:30, 11.27s/it]
 44%|████▍     | 626/1410 [2:01:48<2:28:42, 11.38s/it]
                                                      
{'loss': 0.0283, 'grad_norm': 2.3488872051239014, 'learning_rate': 5.530129429345791e-07, 'kl': -0.0054, 'entropy': -0.0703, 'ce_loss': 0.0361, 'epoch': 0.89}

 44%|████▍     | 626/1410 [2:01:48<2:28:42, 11.38s/it]
 44%|████▍     | 627/1410 [2:01:59<2:27:24, 11.30s/it]
                                                      
{'loss': 0.0295, 'grad_norm': 2.006892681121826, 'learning_rate': 5.52005958862611e-07, 'kl': -0.0071, 'entropy': -0.0591, 'ce_loss': 0.0499, 'epoch': 0.89}

 44%|████▍     | 627/1410 [2:01:59<2:27:24, 11.30s/it]
 45%|████▍     | 628/1410 [2:02:11<2:28:38, 11.40s/it]
                                                      
{'loss': 0.0356, 'grad_norm': 2.742338180541992, 'learning_rate': 5.5099843603967e-07, 'kl': -0.0067, 'entropy': -0.0137, 'ce_loss': 0.0627, 'epoch': 0.89}

 45%|████▍     | 628/1410 [2:02:11<2:28:38, 11.40s/it]
 45%|████▍     | 629/1410 [2:02:22<2:28:18, 11.39s/it]
                                                      
{'loss': 0.0294, 'grad_norm': 2.6782209873199463, 'learning_rate': 5.499903797870514e-07, 'kl': -0.0064, 'entropy': -0.0574, 'ce_loss': 0.0406, 'epoch': 0.89}

 45%|████▍     | 629/1410 [2:02:22<2:28:18, 11.39s/it]
 45%|████▍     | 630/1410 [2:02:34<2:29:20, 11.49s/it]
                                                      
{'loss': 0.0362, 'grad_norm': 2.3707706928253174, 'learning_rate': 5.489817954288692e-07, 'kl': 0.0004, 'entropy': -0.0483, 'ce_loss': 0.0116, 'epoch': 0.89}

 45%|████▍     | 630/1410 [2:02:34<2:29:20, 11.49s/it]
 45%|████▍     | 631/1410 [2:02:44<2:23:54, 11.08s/it]
                                                      
{'loss': 0.0244, 'grad_norm': 2.3564324378967285, 'learning_rate': 5.479726882920256e-07, 'kl': 0.0041, 'entropy': -0.0693, 'ce_loss': 0.0505, 'epoch': 0.9}

 45%|████▍     | 631/1410 [2:02:44<2:23:54, 11.08s/it]
 45%|████▍     | 632/1410 [2:02:55<2:22:16, 10.97s/it]
                                                      
{'loss': 0.0253, 'grad_norm': 2.138780117034912, 'learning_rate': 5.469630637061844e-07, 'kl': 0.006, 'entropy': -0.0654, 'ce_loss': 0.0379, 'epoch': 0.9}

 45%|████▍     | 632/1410 [2:02:55<2:22:16, 10.97s/it]
 45%|████▍     | 633/1410 [2:03:05<2:21:08, 10.90s/it]
                                                      
{'loss': 0.0194, 'grad_norm': 1.8955565690994263, 'learning_rate': 5.459529270037419e-07, 'kl': 0.0, 'entropy': -0.0815, 'ce_loss': 0.0354, 'epoch': 0.9}

 45%|████▍     | 633/1410 [2:03:05<2:21:08, 10.90s/it]
 45%|████▍     | 634/1410 [2:03:17<2:24:50, 11.20s/it]
                                                      
{'loss': 0.0125, 'grad_norm': 1.0602673292160034, 'learning_rate': 5.449422835198e-07, 'kl': -0.0068, 'entropy': -0.0776, 'ce_loss': 0.0415, 'epoch': 0.9}

 45%|████▍     | 634/1410 [2:03:17<2:24:50, 11.20s/it]
 45%|████▌     | 635/1410 [2:03:29<2:25:52, 11.29s/it]
                                                      
{'loss': 0.0187, 'grad_norm': 1.915784478187561, 'learning_rate': 5.439311385921362e-07, 'kl': 0.0024, 'entropy': -0.0466, 'ce_loss': 0.0204, 'epoch': 0.9}

 45%|████▌     | 635/1410 [2:03:29<2:25:52, 11.29s/it]
 45%|████▌     | 636/1410 [2:03:41<2:27:53, 11.46s/it]
                                                      
{'loss': 0.0399, 'grad_norm': 2.2146084308624268, 'learning_rate': 5.429194975611772e-07, 'kl': -0.0041, 'entropy': -0.028, 'ce_loss': 0.0173, 'epoch': 0.9}

 45%|████▌     | 636/1410 [2:03:41<2:27:53, 11.46s/it]
 45%|████▌     | 637/1410 [2:03:52<2:26:30, 11.37s/it]
                                                      
{'loss': 0.0177, 'grad_norm': 1.6641236543655396, 'learning_rate': 5.419073657699696e-07, 'kl': 0.0007, 'entropy': 0.0007, 'ce_loss': 0.0053, 'epoch': 0.9}

 45%|████▌     | 637/1410 [2:03:52<2:26:30, 11.37s/it]
 45%|████▌     | 638/1410 [2:04:03<2:27:26, 11.46s/it]
                                                      
{'loss': 0.023, 'grad_norm': 2.3760697841644287, 'learning_rate': 5.408947485641517e-07, 'kl': -0.0027, 'entropy': -0.0237, 'ce_loss': 0.0776, 'epoch': 0.9}

 45%|████▌     | 638/1410 [2:04:03<2:27:26, 11.46s/it]
 45%|████▌     | 639/1410 [2:04:15<2:28:22, 11.55s/it]
                                                      
{'loss': 0.0236, 'grad_norm': 2.37418794631958, 'learning_rate': 5.398816512919257e-07, 'kl': -0.0074, 'entropy': -0.0903, 'ce_loss': 0.045, 'epoch': 0.91}

 45%|████▌     | 639/1410 [2:04:15<2:28:22, 11.55s/it]
 45%|████▌     | 640/1410 [2:04:27<2:28:26, 11.57s/it]
                                                      
{'loss': 0.0327, 'grad_norm': 1.8160148859024048, 'learning_rate': 5.388680793040301e-07, 'kl': -0.0035, 'entropy': -0.0762, 'ce_loss': 0.033, 'epoch': 0.91}

 45%|████▌     | 640/1410 [2:04:27<2:28:26, 11.57s/it]
 45%|████▌     | 641/1410 [2:04:38<2:28:25, 11.58s/it]
                                                      
{'loss': 0.0188, 'grad_norm': 1.5713584423065186, 'learning_rate': 5.378540379537093e-07, 'kl': -0.003, 'entropy': -0.0476, 'ce_loss': 0.0322, 'epoch': 0.91}

 45%|████▌     | 641/1410 [2:04:38<2:28:25, 11.58s/it]
 46%|████▌     | 642/1410 [2:04:50<2:29:17, 11.66s/it]
                                                      
{'loss': 0.0452, 'grad_norm': 1.811855673789978, 'learning_rate': 5.368395325966876e-07, 'kl': -0.0039, 'entropy': 0.0137, 'ce_loss': 0.0719, 'epoch': 0.91}

 46%|████▌     | 642/1410 [2:04:50<2:29:17, 11.66s/it]
 46%|████▌     | 643/1410 [2:05:02<2:29:25, 11.69s/it]
                                                      
{'loss': 0.0329, 'grad_norm': 2.178006649017334, 'learning_rate': 5.358245685911394e-07, 'kl': -0.0028, 'entropy': -0.0498, 'ce_loss': 0.0365, 'epoch': 0.91}

 46%|████▌     | 643/1410 [2:05:02<2:29:25, 11.69s/it]
 46%|████▌     | 644/1410 [2:05:13<2:27:58, 11.59s/it]
                                                      
{'loss': 0.0324, 'grad_norm': 2.5815858840942383, 'learning_rate': 5.348091512976624e-07, 'kl': -0.0133, 'entropy': -0.0277, 'ce_loss': 0.0085, 'epoch': 0.91}

 46%|████▌     | 644/1410 [2:05:13<2:27:58, 11.59s/it]
 46%|████▌     | 645/1410 [2:05:25<2:27:58, 11.61s/it]
                                                      
{'loss': 0.0203, 'grad_norm': 1.7623604536056519, 'learning_rate': 5.337932860792472e-07, 'kl': -0.0057, 'entropy': 0.0347, 'ce_loss': 0.0999, 'epoch': 0.91}

 46%|████▌     | 645/1410 [2:05:25<2:27:58, 11.61s/it]
 46%|████▌     | 646/1410 [2:05:37<2:28:41, 11.68s/it]
                                                      
{'loss': 0.0442, 'grad_norm': 2.3680241107940674, 'learning_rate': 5.327769783012509e-07, 'kl': 0.0007, 'entropy': -0.0168, 'ce_loss': 0.0013, 'epoch': 0.92}

 46%|████▌     | 646/1410 [2:05:37<2:28:41, 11.68s/it]
 46%|████▌     | 647/1410 [2:05:49<2:29:02, 11.72s/it]
                                                      
{'loss': 0.0272, 'grad_norm': 1.8400912284851074, 'learning_rate': 5.317602333313677e-07, 'kl': -0.0012, 'entropy': -0.0166, 'ce_loss': 0.0008, 'epoch': 0.92}

 46%|████▌     | 647/1410 [2:05:49<2:29:02, 11.72s/it]
 46%|████▌     | 648/1410 [2:06:00<2:28:38, 11.70s/it]
                                                      
{'loss': 0.0433, 'grad_norm': 1.9117997884750366, 'learning_rate': 5.307430565396012e-07, 'kl': 0.0003, 'entropy': -0.0225, 'ce_loss': 0.0006, 'epoch': 0.92}

 46%|████▌     | 648/1410 [2:06:00<2:28:38, 11.70s/it]
 46%|████▌     | 649/1410 [2:06:11<2:24:39, 11.41s/it]
                                                      
{'loss': 0.0233, 'grad_norm': 2.484868049621582, 'learning_rate': 5.297254532982352e-07, 'kl': 0.0022, 'entropy': -0.031, 'ce_loss': 0.0094, 'epoch': 0.92}

 46%|████▌     | 649/1410 [2:06:11<2:24:39, 11.41s/it]
 46%|████▌     | 650/1410 [2:06:23<2:26:10, 11.54s/it]
                                                      
{'loss': 0.0343, 'grad_norm': 2.0193803310394287, 'learning_rate': 5.287074289818062e-07, 'kl': 0.0009, 'entropy': -0.0649, 'ce_loss': 0.0222, 'epoch': 0.92}

 46%|████▌     | 650/1410 [2:06:23<2:26:10, 11.54s/it]
 46%|████▌     | 651/1410 [2:06:35<2:27:01, 11.62s/it]
                                                      
{'loss': 0.0443, 'grad_norm': 2.1540963649749756, 'learning_rate': 5.276889889670746e-07, 'kl': -0.0066, 'entropy': -0.0359, 'ce_loss': 0.0241, 'epoch': 0.92}

 46%|████▌     | 651/1410 [2:06:35<2:27:01, 11.62s/it]
 46%|████▌     | 652/1410 [2:06:46<2:25:37, 11.53s/it]
                                                      
{'loss': 0.0285, 'grad_norm': 2.3175439834594727, 'learning_rate': 5.266701386329962e-07, 'kl': 0.0139, 'entropy': -0.043, 'ce_loss': 0.0021, 'epoch': 0.92}

 46%|████▌     | 652/1410 [2:06:46<2:25:37, 11.53s/it]
 46%|████▋     | 653/1410 [2:06:58<2:26:47, 11.63s/it]
                                                      
{'loss': 0.028, 'grad_norm': 1.4466232061386108, 'learning_rate': 5.256508833606937e-07, 'kl': -0.0038, 'entropy': -0.0381, 'ce_loss': 0.0089, 'epoch': 0.93}

 46%|████▋     | 653/1410 [2:06:58<2:26:47, 11.63s/it]
 46%|████▋     | 654/1410 [2:07:10<2:28:05, 11.75s/it]
                                                      
{'loss': 0.0421, 'grad_norm': 1.7365047931671143, 'learning_rate': 5.246312285334291e-07, 'kl': -0.0003, 'entropy': 0.0903, 'ce_loss': 0.1016, 'epoch': 0.93}

 46%|████▋     | 654/1410 [2:07:10<2:28:05, 11.75s/it]
 46%|████▋     | 655/1410 [2:07:21<2:26:43, 11.66s/it]
                                                      
{'loss': 0.021, 'grad_norm': 1.8009169101715088, 'learning_rate': 5.236111795365743e-07, 'kl': 0.0166, 'entropy': -0.051, 'ce_loss': 0.0337, 'epoch': 0.93}

 46%|████▋     | 655/1410 [2:07:21<2:26:43, 11.66s/it]
 47%|████▋     | 656/1410 [2:07:33<2:26:20, 11.65s/it]
                                                      
{'loss': 0.0254, 'grad_norm': 2.302374839782715, 'learning_rate': 5.225907417575829e-07, 'kl': 0.0007, 'entropy': -0.0581, 'ce_loss': 0.0152, 'epoch': 0.93}

 47%|████▋     | 656/1410 [2:07:33<2:26:20, 11.65s/it]
 47%|████▋     | 657/1410 [2:07:45<2:26:01, 11.64s/it]
                                                      
{'loss': 0.0263, 'grad_norm': 2.3225619792938232, 'learning_rate': 5.215699205859625e-07, 'kl': 0.0179, 'entropy': -0.0302, 'ce_loss': 0.0273, 'epoch': 0.93}

 47%|████▋     | 657/1410 [2:07:45<2:26:01, 11.64s/it]
 47%|████▋     | 658/1410 [2:07:56<2:24:14, 11.51s/it]
                                                      
{'loss': 0.0323, 'grad_norm': 2.399925947189331, 'learning_rate': 5.205487214132447e-07, 'kl': -0.0049, 'entropy': -0.0962, 'ce_loss': 0.0289, 'epoch': 0.93}

 47%|████▋     | 658/1410 [2:07:56<2:24:14, 11.51s/it]
 47%|████▋     | 659/1410 [2:08:07<2:24:40, 11.56s/it]
                                                      
{'loss': 0.0041, 'grad_norm': 0.6786096096038818, 'learning_rate': 5.195271496329584e-07, 'kl': 0.0039, 'entropy': 0.016, 'ce_loss': 0.1263, 'epoch': 0.93}

 47%|████▋     | 659/1410 [2:08:08<2:24:40, 11.56s/it]
 47%|████▋     | 660/1410 [2:08:19<2:23:59, 11.52s/it]
                                                      
{'loss': 0.0324, 'grad_norm': 2.568338632583618, 'learning_rate': 5.185052106405997e-07, 'kl': 0.006, 'entropy': -0.0498, 'ce_loss': 0.0054, 'epoch': 0.94}

 47%|████▋     | 660/1410 [2:08:19<2:23:59, 11.52s/it]
 47%|████▋     | 661/1410 [2:08:31<2:24:39, 11.59s/it]
                                                      
{'loss': 0.0566, 'grad_norm': 2.182640790939331, 'learning_rate': 5.174829098336049e-07, 'kl': -0.0039, 'entropy': 0.0515, 'ce_loss': 0.1031, 'epoch': 0.94}

 47%|████▋     | 661/1410 [2:08:31<2:24:39, 11.59s/it]
 47%|████▋     | 662/1410 [2:08:43<2:25:37, 11.68s/it]
                                                      
{'loss': 0.0492, 'grad_norm': 2.414548397064209, 'learning_rate': 5.164602526113205e-07, 'kl': 0.0006, 'entropy': -0.0209, 'ce_loss': 0.0044, 'epoch': 0.94}

 47%|████▋     | 662/1410 [2:08:43<2:25:37, 11.68s/it]
 47%|████▋     | 663/1410 [2:08:54<2:26:01, 11.73s/it]
                                                      
{'loss': 0.0334, 'grad_norm': 2.184286117553711, 'learning_rate': 5.154372443749761e-07, 'kl': 0.0024, 'entropy': -0.0189, 'ce_loss': 0.0007, 'epoch': 0.94}

 47%|████▋     | 663/1410 [2:08:54<2:26:01, 11.73s/it]
 47%|████▋     | 664/1410 [2:09:06<2:25:52, 11.73s/it]
                                                      
{'loss': 0.0449, 'grad_norm': 2.4942989349365234, 'learning_rate': 5.144138905276548e-07, 'kl': 0.0033, 'entropy': -0.0405, 'ce_loss': 0.0274, 'epoch': 0.94}

 47%|████▋     | 664/1410 [2:09:06<2:25:52, 11.73s/it]
 47%|████▋     | 665/1410 [2:09:18<2:26:05, 11.77s/it]
                                                      
{'loss': 0.0266, 'grad_norm': 1.8714088201522827, 'learning_rate': 5.133901964742652e-07, 'kl': -0.0025, 'entropy': 0.0247, 'ce_loss': 0.0934, 'epoch': 0.94}

 47%|████▋     | 665/1410 [2:09:18<2:26:05, 11.77s/it]
 47%|████▋     | 666/1410 [2:09:30<2:26:25, 11.81s/it]
                                                      
{'loss': 0.03, 'grad_norm': 1.772731065750122, 'learning_rate': 5.123661676215125e-07, 'kl': -0.0087, 'entropy': 0.0403, 'ce_loss': 0.0906, 'epoch': 0.94}

 47%|████▋     | 666/1410 [2:09:30<2:26:25, 11.81s/it]
 47%|████▋     | 667/1410 [2:09:41<2:23:37, 11.60s/it]
                                                      
{'loss': 0.0222, 'grad_norm': 1.9459611177444458, 'learning_rate': 5.113418093778706e-07, 'kl': -0.0001, 'entropy': -0.0376, 'ce_loss': 0.0101, 'epoch': 0.95}

 47%|████▋     | 667/1410 [2:09:41<2:23:37, 11.60s/it]
 47%|████▋     | 668/1410 [2:09:53<2:24:42, 11.70s/it]
                                                      
{'loss': 0.0307, 'grad_norm': 1.421966314315796, 'learning_rate': 5.103171271535529e-07, 'kl': -0.0019, 'entropy': -0.0505, 'ce_loss': 0.0321, 'epoch': 0.95}

 47%|████▋     | 668/1410 [2:09:53<2:24:42, 11.70s/it]
 47%|████▋     | 669/1410 [2:10:05<2:25:11, 11.76s/it]
                                                      
{'loss': 0.0278, 'grad_norm': 1.163221836090088, 'learning_rate': 5.092921263604835e-07, 'kl': -0.006, 'entropy': 0.0835, 'ce_loss': 0.1633, 'epoch': 0.95}

 47%|████▋     | 669/1410 [2:10:05<2:25:11, 11.76s/it]
 48%|████▊     | 670/1410 [2:10:17<2:25:09, 11.77s/it]
                                                      
{'loss': 0.0247, 'grad_norm': 1.5163726806640625, 'learning_rate': 5.082668124122697e-07, 'kl': -0.0015, 'entropy': -0.0413, 'ce_loss': 0.0175, 'epoch': 0.95}

 48%|████▊     | 670/1410 [2:10:17<2:25:09, 11.77s/it]
 48%|████▊     | 671/1410 [2:10:28<2:23:57, 11.69s/it]
                                                      
{'loss': 0.0226, 'grad_norm': 1.6655759811401367, 'learning_rate': 5.072411907241725e-07, 'kl': -0.0004, 'entropy': -0.021, 'ce_loss': 0.0011, 'epoch': 0.95}

 48%|████▊     | 671/1410 [2:10:28<2:23:57, 11.69s/it]
 48%|████▊     | 672/1410 [2:10:39<2:21:46, 11.53s/it]
                                                      
{'loss': 0.0309, 'grad_norm': 2.0967299938201904, 'learning_rate': 5.062152667130783e-07, 'kl': -0.0022, 'entropy': -0.0549, 'ce_loss': 0.0223, 'epoch': 0.95}

 48%|████▊     | 672/1410 [2:10:39<2:21:46, 11.53s/it]
 48%|████▊     | 673/1410 [2:10:51<2:22:11, 11.58s/it]
                                                      
{'loss': 0.0461, 'grad_norm': 2.720214605331421, 'learning_rate': 5.051890457974697e-07, 'kl': -0.0045, 'entropy': -0.0302, 'ce_loss': 0.1019, 'epoch': 0.95}

 48%|████▊     | 673/1410 [2:10:51<2:22:11, 11.58s/it]
 48%|████▊     | 674/1410 [2:11:03<2:22:29, 11.62s/it]
                                                      
{'loss': 0.0181, 'grad_norm': 1.494168996810913, 'learning_rate': 5.041625333973982e-07, 'kl': 0.0031, 'entropy': -0.0508, 'ce_loss': 0.0376, 'epoch': 0.96}

 48%|████▊     | 674/1410 [2:11:03<2:22:29, 11.62s/it]
 48%|████▊     | 675/1410 [2:11:14<2:21:27, 11.55s/it]
                                                      
{'loss': 0.021, 'grad_norm': 2.3798956871032715, 'learning_rate': 5.031357349344546e-07, 'kl': 0.0013, 'entropy': -0.0679, 'ce_loss': 0.0224, 'epoch': 0.96}

 48%|████▊     | 675/1410 [2:11:14<2:21:27, 11.55s/it]
 48%|████▊     | 676/1410 [2:11:26<2:21:50, 11.59s/it]
                                                      
{'loss': 0.0439, 'grad_norm': 2.502988576889038, 'learning_rate': 5.021086558317402e-07, 'kl': -0.0031, 'entropy': -0.0374, 'ce_loss': 0.0188, 'epoch': 0.96}

 48%|████▊     | 676/1410 [2:11:26<2:21:50, 11.59s/it]
 48%|████▊     | 677/1410 [2:11:38<2:23:07, 11.72s/it]
                                                      
{'loss': 0.0472, 'grad_norm': 2.057990550994873, 'learning_rate': 5.010813015138387e-07, 'kl': 0.0041, 'entropy': -0.0272, 'ce_loss': 0.0181, 'epoch': 0.96}

 48%|████▊     | 677/1410 [2:11:38<2:23:07, 11.72s/it]
 48%|████▊     | 678/1410 [2:11:49<2:21:53, 11.63s/it]
                                                      
{'loss': 0.0229, 'grad_norm': 1.6258925199508667, 'learning_rate': 5.000536774067877e-07, 'kl': -0.0007, 'entropy': -0.0121, 'ce_loss': 0.0002, 'epoch': 0.96}

 48%|████▊     | 678/1410 [2:11:49<2:21:53, 11.63s/it]
 48%|████▊     | 679/1410 [2:12:01<2:21:57, 11.65s/it]
                                                      
{'loss': 0.0153, 'grad_norm': 1.537358283996582, 'learning_rate': 4.990257889380491e-07, 'kl': -0.0055, 'entropy': -0.0942, 'ce_loss': 0.0281, 'epoch': 0.96}

 48%|████▊     | 679/1410 [2:12:01<2:21:57, 11.65s/it]
 48%|████▊     | 680/1410 [2:12:12<2:18:42, 11.40s/it]
                                                      
{'loss': 0.0354, 'grad_norm': 2.7207555770874023, 'learning_rate': 4.979976415364814e-07, 'kl': -0.0, 'entropy': -0.1196, 'ce_loss': 0.0272, 'epoch': 0.96}

 48%|████▊     | 680/1410 [2:12:12<2:18:42, 11.40s/it]
 48%|████▊     | 681/1410 [2:12:23<2:16:19, 11.22s/it]
                                                      
{'loss': 0.0388, 'grad_norm': 2.9153034687042236, 'learning_rate': 4.969692406323108e-07, 'kl': 0.0093, 'entropy': -0.0173, 'ce_loss': 0.0054, 'epoch': 0.97}

 48%|████▊     | 681/1410 [2:12:23<2:16:19, 11.22s/it]
 48%|████▊     | 682/1410 [2:12:33<2:13:52, 11.03s/it]
                                                      
{'loss': 0.0386, 'grad_norm': 2.5496437549591064, 'learning_rate': 4.959405916571022e-07, 'kl': -0.0101, 'entropy': -0.0474, 'ce_loss': 0.0434, 'epoch': 0.97}

 48%|████▊     | 682/1410 [2:12:33<2:13:52, 11.03s/it]
 48%|████▊     | 683/1410 [2:12:45<2:16:47, 11.29s/it]
                                                      
{'loss': 0.0312, 'grad_norm': 2.078629970550537, 'learning_rate': 4.949117000437306e-07, 'kl': -0.0098, 'entropy': -0.0036, 'ce_loss': 0.0813, 'epoch': 0.97}

 48%|████▊     | 683/1410 [2:12:45<2:16:47, 11.29s/it]
 49%|████▊     | 684/1410 [2:12:57<2:18:42, 11.46s/it]
                                                      
{'loss': 0.0328, 'grad_norm': 2.1510767936706543, 'learning_rate': 4.938825712263528e-07, 'kl': -0.0003, 'entropy': -0.0403, 'ce_loss': 0.0175, 'epoch': 0.97}

 49%|████▊     | 684/1410 [2:12:57<2:18:42, 11.46s/it]
 49%|████▊     | 685/1410 [2:13:08<2:16:38, 11.31s/it]
                                                      
{'loss': 0.0135, 'grad_norm': 1.5732635259628296, 'learning_rate': 4.92853210640378e-07, 'kl': 0.0001, 'entropy': -0.031, 'ce_loss': 0.0016, 'epoch': 0.97}

 49%|████▊     | 685/1410 [2:13:08<2:16:38, 11.31s/it]
 49%|████▊     | 686/1410 [2:13:19<2:17:33, 11.40s/it]
                                                      
{'loss': 0.0333, 'grad_norm': 2.2301993370056152, 'learning_rate': 4.9182362372244e-07, 'kl': -0.0002, 'entropy': -0.0408, 'ce_loss': 0.0253, 'epoch': 0.97}

 49%|████▊     | 686/1410 [2:13:19<2:17:33, 11.40s/it]
 49%|████▊     | 687/1410 [2:13:30<2:15:38, 11.26s/it]
                                                      
{'loss': 0.0178, 'grad_norm': 1.9127609729766846, 'learning_rate': 4.907938159103678e-07, 'kl': -0.0016, 'entropy': -0.0359, 'ce_loss': 0.0009, 'epoch': 0.97}

 49%|████▊     | 687/1410 [2:13:30<2:15:38, 11.26s/it]
 49%|████▉     | 688/1410 [2:13:42<2:17:40, 11.44s/it]
                                                      
{'loss': 0.0392, 'grad_norm': 2.2847461700439453, 'learning_rate': 4.897637926431566e-07, 'kl': -0.0063, 'entropy': -0.0049, 'ce_loss': 0.0782, 'epoch': 0.98}

 49%|████▉     | 688/1410 [2:13:42<2:17:40, 11.44s/it]
 49%|████▉     | 689/1410 [2:13:54<2:19:22, 11.60s/it]
                                                      
{'loss': 0.0481, 'grad_norm': 2.3221898078918457, 'learning_rate': 4.887335593609401e-07, 'kl': -0.0092, 'entropy': -0.0422, 'ce_loss': 0.037, 'epoch': 0.98}

 49%|████▉     | 689/1410 [2:13:54<2:19:22, 11.60s/it]
 49%|████▉     | 690/1410 [2:14:06<2:18:23, 11.53s/it]
                                                      
{'loss': 0.0389, 'grad_norm': 2.828967332839966, 'learning_rate': 4.87703121504961e-07, 'kl': 0.0044, 'entropy': -0.0923, 'ce_loss': 0.0271, 'epoch': 0.98}

 49%|████▉     | 690/1410 [2:14:06<2:18:23, 11.53s/it]
 49%|████▉     | 691/1410 [2:14:17<2:16:48, 11.42s/it]
                                                      
{'loss': 0.0113, 'grad_norm': 1.4886236190795898, 'learning_rate': 4.866724845175429e-07, 'kl': -0.002, 'entropy': -0.041, 'ce_loss': 0.0053, 'epoch': 0.98}

 49%|████▉     | 691/1410 [2:14:17<2:16:48, 11.42s/it]
 49%|████▉     | 692/1410 [2:14:28<2:17:46, 11.51s/it]
                                                      
{'loss': 0.0163, 'grad_norm': 1.5280983448028564, 'learning_rate': 4.856416538420604e-07, 'kl': -0.0045, 'entropy': 0.0564, 'ce_loss': 0.0938, 'epoch': 0.98}

 49%|████▉     | 692/1410 [2:14:28<2:17:46, 11.51s/it]
 49%|████▉     | 693/1410 [2:14:40<2:18:35, 11.60s/it]
                                                      
{'loss': 0.0186, 'grad_norm': 1.8969346284866333, 'learning_rate': 4.846106349229112e-07, 'kl': -0.0079, 'entropy': 0.064, 'ce_loss': 0.1802, 'epoch': 0.98}

 49%|████▉     | 693/1410 [2:14:40<2:18:35, 11.60s/it]
 49%|████▉     | 694/1410 [2:14:51<2:15:15, 11.33s/it]
                                                      
{'loss': 0.0313, 'grad_norm': 2.742969036102295, 'learning_rate': 4.835794332054879e-07, 'kl': -0.0038, 'entropy': -0.0554, 'ce_loss': 0.0181, 'epoch': 0.98}

 49%|████▉     | 694/1410 [2:14:51<2:15:15, 11.33s/it]
 49%|████▉     | 695/1410 [2:15:03<2:15:47, 11.39s/it]
                                                      
{'loss': 0.0272, 'grad_norm': 2.224729299545288, 'learning_rate': 4.825480541361479e-07, 'kl': 0.006, 'entropy': -0.0659, 'ce_loss': 0.0187, 'epoch': 0.99}

 49%|████▉     | 695/1410 [2:15:03<2:15:47, 11.39s/it]
 49%|████▉     | 696/1410 [2:15:14<2:15:01, 11.35s/it]
                                                      
{'loss': 0.0212, 'grad_norm': 1.9107328653335571, 'learning_rate': 4.815165031621854e-07, 'kl': 0.0017, 'entropy': -0.0378, 'ce_loss': 0.0094, 'epoch': 0.99}

 49%|████▉     | 696/1410 [2:15:14<2:15:01, 11.35s/it]
 49%|████▉     | 697/1410 [2:15:25<2:14:01, 11.28s/it]
                                                      
{'loss': 0.0219, 'grad_norm': 2.1321916580200195, 'learning_rate': 4.804847857318028e-07, 'kl': -0.0078, 'entropy': -0.0076, 'ce_loss': 0.0901, 'epoch': 0.99}

 49%|████▉     | 697/1410 [2:15:25<2:14:01, 11.28s/it]
 50%|████▉     | 698/1410 [2:15:37<2:15:20, 11.40s/it]
                                                      
{'loss': 0.0364, 'grad_norm': 1.8958439826965332, 'learning_rate': 4.794529072940813e-07, 'kl': -0.0002, 'entropy': -0.0576, 'ce_loss': 0.0205, 'epoch': 0.99}

 50%|████▉     | 698/1410 [2:15:37<2:15:20, 11.40s/it]
 50%|████▉     | 699/1410 [2:15:48<2:16:33, 11.52s/it]
                                                      
{'loss': 0.017, 'grad_norm': 1.2603397369384766, 'learning_rate': 4.784208732989527e-07, 'kl': -0.0088, 'entropy': -0.0537, 'ce_loss': 0.0559, 'epoch': 0.99}

 50%|████▉     | 699/1410 [2:15:48<2:16:33, 11.52s/it]
 50%|████▉     | 700/1410 [2:16:00<2:16:49, 11.56s/it]
                                                      
{'loss': 0.0172, 'grad_norm': 1.307206153869629, 'learning_rate': 4.773886891971702e-07, 'kl': -0.0005, 'entropy': -0.0164, 'ce_loss': 0.0109, 'epoch': 0.99}

 50%|████▉     | 700/1410 [2:16:00<2:16:49, 11.56s/it]
 50%|████▉     | 701/1410 [2:16:11<2:14:59, 11.42s/it]
                                                      
{'loss': 0.0373, 'grad_norm': 2.906741142272949, 'learning_rate': 4.763563604402803e-07, 'kl': -0.0042, 'entropy': -0.0396, 'ce_loss': 0.0128, 'epoch': 0.99}

 50%|████▉     | 701/1410 [2:16:11<2:14:59, 11.42s/it]
 50%|████▉     | 702/1410 [2:16:23<2:16:23, 11.56s/it]
                                                      
{'loss': 0.0357, 'grad_norm': 1.8590211868286133, 'learning_rate': 4.753238924805928e-07, 'kl': -0.0022, 'entropy': -0.014, 'ce_loss': 0.0717, 'epoch': 1.0}

 50%|████▉     | 702/1410 [2:16:23<2:16:23, 11.56s/it]
 50%|████▉     | 703/1410 [2:16:34<2:14:11, 11.39s/it]
                                                      
{'loss': 0.0265, 'grad_norm': 2.401613712310791, 'learning_rate': 4.74291290771153e-07, 'kl': -0.0004, 'entropy': -0.0491, 'ce_loss': 0.0222, 'epoch': 1.0}

 50%|████▉     | 703/1410 [2:16:34<2:14:11, 11.39s/it]
 50%|████▉     | 704/1410 [2:16:45<2:14:25, 11.42s/it]
                                                      
{'loss': 0.0215, 'grad_norm': 2.043485164642334, 'learning_rate': 4.732585607657127e-07, 'kl': -0.0049, 'entropy': -0.0374, 'ce_loss': 0.0304, 'epoch': 1.0}

 50%|████▉     | 704/1410 [2:16:45<2:14:25, 11.42s/it]
 50%|█████     | 705/1410 [2:16:57<2:15:09, 11.50s/it]
                                                      
{'loss': 0.0251, 'grad_norm': 2.3539295196533203, 'learning_rate': 4.7222570791870155e-07, 'kl': -0.0054, 'entropy': -0.064, 'ce_loss': 0.0239, 'epoch': 1.0}

 50%|█████     | 705/1410 [2:16:57<2:15:09, 11.50s/it]
 50%|█████     | 706/1410 [2:17:09<2:16:09, 11.60s/it]
                                                      
{'loss': 0.009, 'grad_norm': 0.6529402732849121, 'learning_rate': 4.7119273768519735e-07, 'kl': -0.0019, 'entropy': 0.1494, 'ce_loss': 0.1456, 'epoch': 1.0}

 50%|█████     | 706/1410 [2:17:09<2:16:09, 11.60s/it]
 50%|█████     | 707/1410 [2:17:20<2:15:01, 11.52s/it]
                                                      
{'loss': 0.0101, 'grad_norm': 1.0686907768249512, 'learning_rate': 4.701596555208982e-07, 'kl': 0.0369, 'entropy': -0.0432, 'ce_loss': 0.0016, 'epoch': 1.0}

 50%|█████     | 707/1410 [2:17:20<2:15:01, 11.52s/it]
 50%|█████     | 708/1410 [2:17:32<2:15:37, 11.59s/it]
                                                      
{'loss': 0.0202, 'grad_norm': 1.178264856338501, 'learning_rate': 4.6912646688209356e-07, 'kl': 0.0186, 'entropy': -0.0488, 'ce_loss': 0.0055, 'epoch': 1.0}

 50%|█████     | 708/1410 [2:17:32<2:15:37, 11.59s/it]
 50%|█████     | 709/1410 [2:17:44<2:17:03, 11.73s/it]
                                                      
{'loss': 0.022, 'grad_norm': 0.9412334561347961, 'learning_rate': 4.6809317722563506e-07, 'kl': -0.0018, 'entropy': 0.0204, 'ce_loss': 0.1324, 'epoch': 1.01}

 50%|█████     | 709/1410 [2:17:44<2:17:03, 11.73s/it]
 50%|█████     | 710/1410 [2:17:56<2:17:17, 11.77s/it]
                                                      
{'loss': 0.013, 'grad_norm': 1.126090168952942, 'learning_rate': 4.6705979200890776e-07, 'kl': -0.0091, 'entropy': 0.0498, 'ce_loss': 0.0945, 'epoch': 1.01}

 50%|█████     | 710/1410 [2:17:56<2:17:17, 11.77s/it]
 50%|█████     | 711/1410 [2:18:08<2:17:35, 11.81s/it]
                                                      
{'loss': 0.0243, 'grad_norm': 1.351561427116394, 'learning_rate': 4.6602631668980174e-07, 'kl': 0.0286, 'entropy': -0.0928, 'ce_loss': 0.0104, 'epoch': 1.01}

 50%|█████     | 711/1410 [2:18:08<2:17:35, 11.81s/it]
 50%|█████     | 712/1410 [2:18:20<2:16:51, 11.76s/it]
                                                      
{'loss': 0.0116, 'grad_norm': 0.9409106969833374, 'learning_rate': 4.6499275672668256e-07, 'kl': 0.0232, 'entropy': -0.0674, 'ce_loss': 0.0097, 'epoch': 1.01}

 50%|█████     | 712/1410 [2:18:20<2:16:51, 11.76s/it]
 51%|█████     | 713/1410 [2:18:32<2:17:25, 11.83s/it]
                                                      
{'loss': 0.0272, 'grad_norm': 1.3420252799987793, 'learning_rate': 4.6395911757836337e-07, 'kl': 0.0354, 'entropy': -0.061, 'ce_loss': 0.0109, 'epoch': 1.01}

 51%|█████     | 713/1410 [2:18:32<2:17:25, 11.83s/it]
 51%|█████     | 714/1410 [2:18:43<2:16:51, 11.80s/it]
                                                      
{'loss': 0.0121, 'grad_norm': 0.9994145035743713, 'learning_rate': 4.629254047040747e-07, 'kl': 0.0176, 'entropy': -0.083, 'ce_loss': 0.0124, 'epoch': 1.01}

 51%|█████     | 714/1410 [2:18:43<2:16:51, 11.80s/it]
 51%|█████     | 715/1410 [2:18:55<2:16:46, 11.81s/it]
                                                      
{'loss': 0.0125, 'grad_norm': 0.7985936403274536, 'learning_rate': 4.6189162356343745e-07, 'kl': -0.0027, 'entropy': 0.0327, 'ce_loss': 0.1315, 'epoch': 1.01}

 51%|█████     | 715/1410 [2:18:55<2:16:46, 11.81s/it]
 51%|█████     | 716/1410 [2:19:07<2:15:55, 11.75s/it]
                                                      
{'loss': 0.0112, 'grad_norm': 1.3262311220169067, 'learning_rate': 4.6085777961643225e-07, 'kl': 0.0237, 'entropy': -0.0253, 'ce_loss': 0.0037, 'epoch': 1.02}

 51%|█████     | 716/1410 [2:19:07<2:15:55, 11.75s/it]
 51%|█████     | 717/1410 [2:19:18<2:15:20, 11.72s/it]
                                                      
{'loss': 0.0043, 'grad_norm': 0.78331458568573, 'learning_rate': 4.59823878323372e-07, 'kl': 0.0229, 'entropy': -0.0537, 'ce_loss': 0.0071, 'epoch': 1.02}

 51%|█████     | 717/1410 [2:19:18<2:15:20, 11.72s/it]
 51%|█████     | 718/1410 [2:19:30<2:15:15, 11.73s/it]
                                                      
{'loss': 0.0191, 'grad_norm': 1.0914496183395386, 'learning_rate': 4.587899251448724e-07, 'kl': -0.0141, 'entropy': -0.0613, 'ce_loss': 0.0113, 'epoch': 1.02}

 51%|█████     | 718/1410 [2:19:30<2:15:15, 11.73s/it]
 51%|█████     | 719/1410 [2:19:42<2:15:10, 11.74s/it]
                                                      
{'loss': 0.0139, 'grad_norm': 0.9123445749282837, 'learning_rate': 4.577559255418228e-07, 'kl': 0.0178, 'entropy': -0.051, 'ce_loss': 0.0084, 'epoch': 1.02}

 51%|█████     | 719/1410 [2:19:42<2:15:10, 11.74s/it]
 51%|█████     | 720/1410 [2:19:54<2:14:46, 11.72s/it]
                                                      
{'loss': 0.0085, 'grad_norm': 0.8257191777229309, 'learning_rate': 4.5672188497535816e-07, 'kl': 0.0294, 'entropy': -0.0942, 'ce_loss': 0.0142, 'epoch': 1.02}

 51%|█████     | 720/1410 [2:19:54<2:14:46, 11.72s/it]
 51%|█████     | 721/1410 [2:20:05<2:15:02, 11.76s/it]
                                                      
{'loss': 0.0163, 'grad_norm': 1.4540057182312012, 'learning_rate': 4.5568780890682957e-07, 'kl': 0.0304, 'entropy': -0.0981, 'ce_loss': 0.0168, 'epoch': 1.02}

 51%|█████     | 721/1410 [2:20:05<2:15:02, 11.76s/it]
 51%|█████     | 722/1410 [2:20:16<2:11:36, 11.48s/it]
                                                      
{'loss': 0.0249, 'grad_norm': 1.6859208345413208, 'learning_rate': 4.5465370279777575e-07, 'kl': 0.0322, 'entropy': -0.0544, 'ce_loss': 0.0476, 'epoch': 1.02}

 51%|█████     | 722/1410 [2:20:16<2:11:36, 11.48s/it]
 51%|█████▏    | 723/1410 [2:20:28<2:12:10, 11.54s/it]
                                                      
{'loss': 0.0068, 'grad_norm': 0.7096707820892334, 'learning_rate': 4.53619572109894e-07, 'kl': -0.0047, 'entropy': 0.0079, 'ce_loss': 0.0662, 'epoch': 1.03}

 51%|█████▏    | 723/1410 [2:20:28<2:12:10, 11.54s/it]
 51%|█████▏    | 724/1410 [2:20:39<2:10:06, 11.38s/it]
                                                      
{'loss': 0.0168, 'grad_norm': 0.8784163594245911, 'learning_rate': 4.525854223050113e-07, 'kl': -0.0045, 'entropy': 0.0273, 'ce_loss': 0.1048, 'epoch': 1.03}

 51%|█████▏    | 724/1410 [2:20:39<2:10:06, 11.38s/it]
 51%|█████▏    | 725/1410 [2:20:50<2:07:12, 11.14s/it]
                                                      
{'loss': 0.0135, 'grad_norm': 1.827286958694458, 'learning_rate': 4.5155125884505613e-07, 'kl': -0.0103, 'entropy': -0.0141, 'ce_loss': 0.0755, 'epoch': 1.03}

 51%|█████▏    | 725/1410 [2:20:50<2:07:12, 11.14s/it]
 51%|█████▏    | 726/1410 [2:21:01<2:06:57, 11.14s/it]
                                                      
{'loss': 0.0123, 'grad_norm': 4.677656650543213, 'learning_rate': 4.5051708719202847e-07, 'kl': 0.0033, 'entropy': -0.0239, 'ce_loss': 0.0023, 'epoch': 1.03}

 51%|█████▏    | 726/1410 [2:21:01<2:06:57, 11.14s/it]
 52%|█████▏    | 727/1410 [2:21:13<2:09:13, 11.35s/it]
                                                      
{'loss': 0.0134, 'grad_norm': 1.0600411891937256, 'learning_rate': 4.494829128079717e-07, 'kl': -0.0054, 'entropy': -0.0156, 'ce_loss': 0.1055, 'epoch': 1.03}

 52%|█████▏    | 727/1410 [2:21:13<2:09:13, 11.35s/it]
 52%|█████▏    | 728/1410 [2:21:24<2:10:10, 11.45s/it]
                                                      
{'loss': 0.0172, 'grad_norm': 1.5121225118637085, 'learning_rate': 4.4844874115494393e-07, 'kl': 0.0238, 'entropy': -0.083, 'ce_loss': 0.0083, 'epoch': 1.03}

 52%|█████▏    | 728/1410 [2:21:24<2:10:10, 11.45s/it]
 52%|█████▏    | 729/1410 [2:21:36<2:11:23, 11.58s/it]
                                                      
{'loss': 0.0183, 'grad_norm': 0.9503545165061951, 'learning_rate': 4.4741457769498856e-07, 'kl': 0.0182, 'entropy': -0.0391, 'ce_loss': 0.0025, 'epoch': 1.03}

 52%|█████▏    | 729/1410 [2:21:36<2:11:23, 11.58s/it]
 52%|█████▏    | 730/1410 [2:21:48<2:11:47, 11.63s/it]
                                                      
{'loss': 0.0157, 'grad_norm': 0.9524443745613098, 'learning_rate': 4.46380427890106e-07, 'kl': 0.0334, 'entropy': -0.0923, 'ce_loss': 0.013, 'epoch': 1.04}

 52%|█████▏    | 730/1410 [2:21:48<2:11:47, 11.63s/it]
 52%|█████▏    | 731/1410 [2:21:59<2:11:32, 11.62s/it]
                                                      
{'loss': 0.0105, 'grad_norm': 1.2480357885360718, 'learning_rate': 4.4534629720222426e-07, 'kl': 0.0039, 'entropy': -0.0349, 'ce_loss': 0.0016, 'epoch': 1.04}

 52%|█████▏    | 731/1410 [2:21:59<2:11:32, 11.62s/it]
 52%|█████▏    | 732/1410 [2:22:11<2:11:23, 11.63s/it]
                                                      
{'loss': 0.0097, 'grad_norm': 0.9662496447563171, 'learning_rate': 4.4431219109317055e-07, 'kl': 0.0332, 'entropy': -0.1021, 'ce_loss': 0.0076, 'epoch': 1.04}

 52%|█████▏    | 732/1410 [2:22:11<2:11:23, 11.63s/it]
 52%|█████▏    | 733/1410 [2:22:23<2:11:45, 11.68s/it]
                                                      
{'loss': 0.0221, 'grad_norm': 1.0010939836502075, 'learning_rate': 4.432781150246419e-07, 'kl': 0.0152, 'entropy': 0.027, 'ce_loss': 0.0743, 'epoch': 1.04}

 52%|█████▏    | 733/1410 [2:22:23<2:11:45, 11.68s/it]
 52%|█████▏    | 734/1410 [2:22:34<2:11:14, 11.65s/it]
                                                      
{'loss': 0.0158, 'grad_norm': 1.1486389636993408, 'learning_rate': 4.422440744581771e-07, 'kl': 0.0288, 'entropy': -0.0776, 'ce_loss': 0.0116, 'epoch': 1.04}

 52%|█████▏    | 734/1410 [2:22:34<2:11:14, 11.65s/it]
 52%|█████▏    | 735/1410 [2:22:46<2:11:33, 11.69s/it]
                                                      
{'loss': 0.0121, 'grad_norm': 0.7109599113464355, 'learning_rate': 4.412100748551277e-07, 'kl': 0.0061, 'entropy': -0.0306, 'ce_loss': 0.0053, 'epoch': 1.04}

 52%|█████▏    | 735/1410 [2:22:46<2:11:33, 11.69s/it]
 52%|█████▏    | 736/1410 [2:22:58<2:11:21, 11.69s/it]
                                                      
{'loss': 0.0201, 'grad_norm': 1.1861162185668945, 'learning_rate': 4.4017612167662794e-07, 'kl': -0.0023, 'entropy': 0.0232, 'ce_loss': 0.1229, 'epoch': 1.04}

 52%|█████▏    | 736/1410 [2:22:58<2:11:21, 11.69s/it]
 52%|█████▏    | 737/1410 [2:23:10<2:11:54, 11.76s/it]
                                                      
{'loss': 0.0124, 'grad_norm': 1.2005234956741333, 'learning_rate': 4.3914222038356776e-07, 'kl': 0.0007, 'entropy': -0.0486, 'ce_loss': 0.0014, 'epoch': 1.05}

 52%|█████▏    | 737/1410 [2:23:10<2:11:54, 11.76s/it]
 52%|█████▏    | 738/1410 [2:23:22<2:11:52, 11.77s/it]
                                                      
{'loss': 0.0201, 'grad_norm': 0.7940223217010498, 'learning_rate': 4.3810837643656267e-07, 'kl': 0.0083, 'entropy': -0.0425, 'ce_loss': 0.0056, 'epoch': 1.05}

 52%|█████▏    | 738/1410 [2:23:22<2:11:52, 11.77s/it]
 52%|█████▏    | 739/1410 [2:23:33<2:11:30, 11.76s/it]
                                                      
{'loss': 0.0116, 'grad_norm': 1.2548593282699585, 'learning_rate': 4.3707459529592533e-07, 'kl': 0.0262, 'entropy': -0.0796, 'ce_loss': 0.0192, 'epoch': 1.05}

 52%|█████▏    | 739/1410 [2:23:33<2:11:30, 11.76s/it]
 52%|█████▏    | 740/1410 [2:23:45<2:11:09, 11.75s/it]
                                                      
{'loss': 0.0095, 'grad_norm': 1.0255472660064697, 'learning_rate': 4.360408824216367e-07, 'kl': -0.006, 'entropy': 0.0297, 'ce_loss': 0.086, 'epoch': 1.05}

 52%|█████▏    | 740/1410 [2:23:45<2:11:09, 11.75s/it]
 53%|█████▎    | 741/1410 [2:23:56<2:09:16, 11.59s/it]
                                                      
{'loss': 0.0111, 'grad_norm': 1.1899267435073853, 'learning_rate': 4.350072432733174e-07, 'kl': 0.0104, 'entropy': -0.0408, 'ce_loss': 0.0064, 'epoch': 1.05}

 53%|█████▎    | 741/1410 [2:23:56<2:09:16, 11.59s/it]
 53%|█████▎    | 742/1410 [2:24:08<2:09:42, 11.65s/it]
                                                      
{'loss': 0.0291, 'grad_norm': 1.6474051475524902, 'learning_rate': 4.339736833101982e-07, 'kl': 0.0479, 'entropy': -0.0588, 'ce_loss': 0.0049, 'epoch': 1.05}

 53%|█████▎    | 742/1410 [2:24:08<2:09:42, 11.65s/it]
 53%|█████▎    | 743/1410 [2:24:20<2:10:09, 11.71s/it]
                                                      
{'loss': 0.0336, 'grad_norm': 1.2348018884658813, 'learning_rate': 4.329402079910922e-07, 'kl': -0.0067, 'entropy': -0.0718, 'ce_loss': 0.024, 'epoch': 1.05}

 53%|█████▎    | 743/1410 [2:24:20<2:10:09, 11.71s/it]
 53%|█████▎    | 744/1410 [2:24:31<2:08:59, 11.62s/it]
                                                      
{'loss': 0.0129, 'grad_norm': 1.1841216087341309, 'learning_rate': 4.31906822774365e-07, 'kl': -0.0114, 'entropy': 0.015, 'ce_loss': 0.1059, 'epoch': 1.06}

 53%|█████▎    | 744/1410 [2:24:31<2:08:59, 11.62s/it]
 53%|█████▎    | 745/1410 [2:24:43<2:09:25, 11.68s/it]
                                                      
{'loss': 0.0107, 'grad_norm': 1.3341995477676392, 'learning_rate': 4.308735331179064e-07, 'kl': 0.0168, 'entropy': -0.104, 'ce_loss': 0.021, 'epoch': 1.06}

 53%|█████▎    | 745/1410 [2:24:43<2:09:25, 11.68s/it]
 53%|█████▎    | 746/1410 [2:24:55<2:09:57, 11.74s/it]
                                                      
{'loss': 0.0189, 'grad_norm': 0.9815958738327026, 'learning_rate': 4.298403444791019e-07, 'kl': 0.0299, 'entropy': -0.0518, 'ce_loss': 0.0118, 'epoch': 1.06}

 53%|█████▎    | 746/1410 [2:24:55<2:09:57, 11.74s/it]
 53%|█████▎    | 747/1410 [2:25:07<2:09:43, 11.74s/it]
                                                      
{'loss': 0.0062, 'grad_norm': 1.6946475505828857, 'learning_rate': 4.2880726231480266e-07, 'kl': 0.0167, 'entropy': -0.0542, 'ce_loss': 0.0065, 'epoch': 1.06}

 53%|█████▎    | 747/1410 [2:25:07<2:09:43, 11.74s/it]
 53%|█████▎    | 748/1410 [2:25:19<2:09:27, 11.73s/it]
                                                      
{'loss': 0.0202, 'grad_norm': 1.589823603630066, 'learning_rate': 4.2777429208129836e-07, 'kl': 0.0332, 'entropy': -0.1191, 'ce_loss': 0.0083, 'epoch': 1.06}

 53%|█████▎    | 748/1410 [2:25:19<2:09:27, 11.73s/it]
 53%|█████▎    | 749/1410 [2:25:30<2:09:29, 11.75s/it]
                                                      
{'loss': 0.0309, 'grad_norm': 1.3681743144989014, 'learning_rate': 4.2674143923428725e-07, 'kl': 0.0078, 'entropy': -0.0344, 'ce_loss': 0.0051, 'epoch': 1.06}

 53%|█████▎    | 749/1410 [2:25:30<2:09:29, 11.75s/it]
 53%|█████▎    | 750/1410 [2:25:41<2:07:03, 11.55s/it]
                                                      
{'loss': 0.0229, 'grad_norm': 1.3071894645690918, 'learning_rate': 4.25708709228847e-07, 'kl': 0.0187, 'entropy': -0.0898, 'ce_loss': 0.0064, 'epoch': 1.06}

 53%|█████▎    | 750/1410 [2:25:41<2:07:03, 11.55s/it]
 53%|█████▎    | 751/1410 [2:25:53<2:07:36, 11.62s/it]
                                                      
{'loss': 0.0071, 'grad_norm': 0.8897430896759033, 'learning_rate': 4.246761075194073e-07, 'kl': -0.0112, 'entropy': 0.0515, 'ce_loss': 0.1512, 'epoch': 1.07}

 53%|█████▎    | 751/1410 [2:25:53<2:07:36, 11.62s/it]
 53%|█████▎    | 752/1410 [2:26:05<2:06:35, 11.54s/it]
                                                      
{'loss': 0.0154, 'grad_norm': 1.0301045179367065, 'learning_rate': 4.236436395597197e-07, 'kl': 0.0317, 'entropy': -0.0679, 'ce_loss': 0.0139, 'epoch': 1.07}

 53%|█████▎    | 752/1410 [2:26:05<2:06:35, 11.54s/it]
 53%|█████▎    | 753/1410 [2:26:16<2:06:08, 11.52s/it]
                                                      
{'loss': 0.0242, 'grad_norm': 1.4865223169326782, 'learning_rate': 4.226113108028296e-07, 'kl': 0.0203, 'entropy': -0.0874, 'ce_loss': 0.0063, 'epoch': 1.07}

 53%|█████▎    | 753/1410 [2:26:16<2:06:08, 11.52s/it]
 53%|█████▎    | 754/1410 [2:26:28<2:05:52, 11.51s/it]
                                                      
{'loss': 0.0084, 'grad_norm': 1.1639481782913208, 'learning_rate': 4.2157912670104735e-07, 'kl': 0.0225, 'entropy': -0.0354, 'ce_loss': 0.0024, 'epoch': 1.07}

 53%|█████▎    | 754/1410 [2:26:28<2:05:52, 11.51s/it]
 54%|█████▎    | 755/1410 [2:26:39<2:06:27, 11.58s/it]
                                                      
{'loss': 0.0206, 'grad_norm': 1.2828600406646729, 'learning_rate': 4.205470927059187e-07, 'kl': 0.0306, 'entropy': -0.061, 'ce_loss': 0.0071, 'epoch': 1.07}

 54%|█████▎    | 755/1410 [2:26:39<2:06:27, 11.58s/it]
 54%|█████▎    | 756/1410 [2:26:51<2:06:31, 11.61s/it]
                                                      
{'loss': 0.0125, 'grad_norm': 0.9867958426475525, 'learning_rate': 4.1951521426819714e-07, 'kl': -0.0123, 'entropy': -0.0679, 'ce_loss': 0.0337, 'epoch': 1.07}

 54%|█████▎    | 756/1410 [2:26:51<2:06:31, 11.61s/it]
 54%|█████▎    | 757/1410 [2:27:02<2:06:01, 11.58s/it]
                                                      
{'loss': 0.0175, 'grad_norm': 1.5309962034225464, 'learning_rate': 4.1848349683781453e-07, 'kl': 0.0128, 'entropy': -0.0381, 'ce_loss': 0.0035, 'epoch': 1.07}

 54%|█████▎    | 757/1410 [2:27:02<2:06:01, 11.58s/it]
 54%|█████▍    | 758/1410 [2:27:14<2:07:03, 11.69s/it]
                                                      
{'loss': 0.0125, 'grad_norm': 0.8898863792419434, 'learning_rate': 4.1745194586385204e-07, 'kl': -0.003, 'entropy': -0.0128, 'ce_loss': 0.0487, 'epoch': 1.08}

 54%|█████▍    | 758/1410 [2:27:14<2:07:03, 11.69s/it]
 54%|█████▍    | 759/1410 [2:27:25<2:03:53, 11.42s/it]
                                                      
{'loss': 0.0136, 'grad_norm': 1.2685781717300415, 'learning_rate': 4.1642056679451215e-07, 'kl': -0.0034, 'entropy': -0.0796, 'ce_loss': 0.0418, 'epoch': 1.08}

 54%|█████▍    | 759/1410 [2:27:25<2:03:53, 11.42s/it]
 54%|█████▍    | 760/1410 [2:27:37<2:04:42, 11.51s/it]
                                                      
{'loss': 0.0097, 'grad_norm': 1.0177468061447144, 'learning_rate': 4.153893650770888e-07, 'kl': 0.0295, 'entropy': -0.0275, 'ce_loss': 0.001, 'epoch': 1.08}

 54%|█████▍    | 760/1410 [2:27:37<2:04:42, 11.51s/it]
 54%|█████▍    | 761/1410 [2:27:49<2:05:40, 11.62s/it]
                                                      
{'loss': 0.015, 'grad_norm': 0.7872145771980286, 'learning_rate': 4.143583461579396e-07, 'kl': -0.0084, 'entropy': -0.019, 'ce_loss': 0.0421, 'epoch': 1.08}

 54%|█████▍    | 761/1410 [2:27:49<2:05:40, 11.62s/it]
 54%|█████▍    | 762/1410 [2:28:00<2:05:38, 11.63s/it]
                                                      
{'loss': 0.0085, 'grad_norm': 1.054828405380249, 'learning_rate': 4.1332751548245715e-07, 'kl': -0.0026, 'entropy': 0.0349, 'ce_loss': 0.0913, 'epoch': 1.08}

 54%|█████▍    | 762/1410 [2:28:00<2:05:38, 11.63s/it]
 54%|█████▍    | 763/1410 [2:28:12<2:04:57, 11.59s/it]
                                                      
{'loss': 0.016, 'grad_norm': 1.323117971420288, 'learning_rate': 4.122968784950389e-07, 'kl': 0.0669, 'entropy': -0.1128, 'ce_loss': 0.0111, 'epoch': 1.08}

 54%|█████▍    | 763/1410 [2:28:12<2:04:57, 11.59s/it]
 54%|█████▍    | 764/1410 [2:28:24<2:05:33, 11.66s/it]
                                                      
{'loss': 0.0085, 'grad_norm': 0.6898389458656311, 'learning_rate': 4.112664406390599e-07, 'kl': -0.0083, 'entropy': 0.0334, 'ce_loss': 0.0889, 'epoch': 1.08}

 54%|█████▍    | 764/1410 [2:28:24<2:05:33, 11.66s/it]
 54%|█████▍    | 765/1410 [2:28:35<2:05:06, 11.64s/it]
                                                      
{'loss': 0.0063, 'grad_norm': 1.7933416366577148, 'learning_rate': 4.1023620735684355e-07, 'kl': -0.0072, 'entropy': 0.0525, 'ce_loss': 0.1426, 'epoch': 1.09}

 54%|█████▍    | 765/1410 [2:28:35<2:05:06, 11.64s/it]
 54%|█████▍    | 766/1410 [2:28:47<2:05:33, 11.70s/it]
                                                      
{'loss': 0.0148, 'grad_norm': 1.2468079328536987, 'learning_rate': 4.0920618408963236e-07, 'kl': 0.0266, 'entropy': -0.0967, 'ce_loss': 0.0218, 'epoch': 1.09}

 54%|█████▍    | 766/1410 [2:28:47<2:05:33, 11.70s/it]
 54%|█████▍    | 767/1410 [2:28:58<2:03:07, 11.49s/it]
                                                      
{'loss': 0.0143, 'grad_norm': 1.2724266052246094, 'learning_rate': 4.0817637627755986e-07, 'kl': 0.0197, 'entropy': -0.0469, 'ce_loss': 0.0134, 'epoch': 1.09}

 54%|█████▍    | 767/1410 [2:28:58<2:03:07, 11.49s/it]
 54%|█████▍    | 768/1410 [2:29:10<2:03:58, 11.59s/it]
                                                      
{'loss': 0.0112, 'grad_norm': 1.0860044956207275, 'learning_rate': 4.071467893596219e-07, 'kl': 0.0087, 'entropy': -0.0381, 'ce_loss': 0.0195, 'epoch': 1.09}

 54%|█████▍    | 768/1410 [2:29:10<2:03:58, 11.59s/it]
 55%|█████▍    | 769/1410 [2:29:22<2:04:06, 11.62s/it]
                                                      
{'loss': 0.0143, 'grad_norm': 1.3456742763519287, 'learning_rate': 4.061174287736472e-07, 'kl': 0.0255, 'entropy': -0.0908, 'ce_loss': 0.0052, 'epoch': 1.09}

 55%|█████▍    | 769/1410 [2:29:22<2:04:06, 11.62s/it]
 55%|█████▍    | 770/1410 [2:29:33<2:04:11, 11.64s/it]
                                                      
{'loss': 0.0112, 'grad_norm': 1.5068720579147339, 'learning_rate': 4.050882999562694e-07, 'kl': -0.0098, 'entropy': 0.054, 'ce_loss': 0.1396, 'epoch': 1.09}

 55%|█████▍    | 770/1410 [2:29:33<2:04:11, 11.64s/it]
 55%|█████▍    | 771/1410 [2:29:45<2:04:30, 11.69s/it]
                                                      
{'loss': 0.008, 'grad_norm': 0.7369887828826904, 'learning_rate': 4.040594083428978e-07, 'kl': -0.0037, 'entropy': 0.1426, 'ce_loss': 0.1344, 'epoch': 1.09}

 55%|█████▍    | 771/1410 [2:29:45<2:04:30, 11.69s/it]
 55%|█████▍    | 772/1410 [2:29:57<2:04:43, 11.73s/it]
                                                      
{'loss': 0.0141, 'grad_norm': 0.8879409432411194, 'learning_rate': 4.030307593676891e-07, 'kl': 0.014, 'entropy': -0.0452, 'ce_loss': 0.0036, 'epoch': 1.1}

 55%|█████▍    | 772/1410 [2:29:57<2:04:43, 11.73s/it]
 55%|█████▍    | 773/1410 [2:30:08<2:02:57, 11.58s/it]
                                                      
{'loss': 0.0121, 'grad_norm': 1.0273240804672241, 'learning_rate': 4.020023584635187e-07, 'kl': 0.025, 'entropy': -0.085, 'ce_loss': 0.0054, 'epoch': 1.1}

 55%|█████▍    | 773/1410 [2:30:08<2:02:57, 11.58s/it]
 55%|█████▍    | 774/1410 [2:30:19<2:01:10, 11.43s/it]
                                                      
{'loss': 0.0111, 'grad_norm': 1.139452576637268, 'learning_rate': 4.00974211061951e-07, 'kl': 0.0063, 'entropy': -0.0231, 'ce_loss': 0.0018, 'epoch': 1.1}

 55%|█████▍    | 774/1410 [2:30:19<2:01:10, 11.43s/it]
 55%|█████▍    | 775/1410 [2:30:30<1:59:18, 11.27s/it]
                                                      
{'loss': 0.0083, 'grad_norm': 0.9428015947341919, 'learning_rate': 3.9994632259321235e-07, 'kl': -0.0048, 'entropy': -0.0481, 'ce_loss': 0.0308, 'epoch': 1.1}

 55%|█████▍    | 775/1410 [2:30:30<1:59:18, 11.27s/it]
 55%|█████▌    | 776/1410 [2:30:42<1:59:57, 11.35s/it]
                                                      
{'loss': 0.011, 'grad_norm': 1.2535674571990967, 'learning_rate': 3.9891869848616135e-07, 'kl': -0.0064, 'entropy': 0.0208, 'ce_loss': 0.089, 'epoch': 1.1}

 55%|█████▌    | 776/1410 [2:30:42<1:59:57, 11.35s/it]
 55%|█████▌    | 777/1410 [2:30:53<1:59:38, 11.34s/it]
                                                      
{'loss': 0.0109, 'grad_norm': 1.1749262809753418, 'learning_rate': 3.978913441682599e-07, 'kl': 0.0015, 'entropy': -0.0308, 'ce_loss': 0.0009, 'epoch': 1.1}

 55%|█████▌    | 777/1410 [2:30:53<1:59:38, 11.34s/it]
 55%|█████▌    | 778/1410 [2:31:04<1:58:52, 11.29s/it]
                                                      
{'loss': 0.0112, 'grad_norm': 0.9933481812477112, 'learning_rate': 3.968642650655455e-07, 'kl': 0.0107, 'entropy': -0.0398, 'ce_loss': 0.0103, 'epoch': 1.1}

 55%|█████▌    | 778/1410 [2:31:04<1:58:52, 11.29s/it]
 55%|█████▌    | 779/1410 [2:31:16<1:59:56, 11.41s/it]
                                                      
{'loss': 0.0124, 'grad_norm': 1.5197809934616089, 'learning_rate': 3.958374666026018e-07, 'kl': 0.0206, 'entropy': -0.1128, 'ce_loss': 0.0053, 'epoch': 1.1}

 55%|█████▌    | 779/1410 [2:31:16<1:59:56, 11.41s/it]
 55%|█████▌    | 780/1410 [2:31:28<2:01:51, 11.61s/it]
                                                      
{'loss': 0.0289, 'grad_norm': 1.204685091972351, 'learning_rate': 3.9481095420253033e-07, 'kl': 0.0249, 'entropy': 0.0645, 'ce_loss': 0.0826, 'epoch': 1.11}

 55%|█████▌    | 780/1410 [2:31:28<2:01:51, 11.61s/it]
 55%|█████▌    | 781/1410 [2:31:39<2:01:03, 11.55s/it]
                                                      
{'loss': 0.0147, 'grad_norm': 1.149584412574768, 'learning_rate': 3.9378473328692186e-07, 'kl': 0.0481, 'entropy': -0.0713, 'ce_loss': 0.0127, 'epoch': 1.11}

 55%|█████▌    | 781/1410 [2:31:39<2:01:03, 11.55s/it]
 55%|█████▌    | 782/1410 [2:31:51<2:01:51, 11.64s/it]
                                                      
{'loss': 0.0128, 'grad_norm': 0.786512553691864, 'learning_rate': 3.9275880927582744e-07, 'kl': -0.004, 'entropy': 0.063, 'ce_loss': 0.1218, 'epoch': 1.11}

 55%|█████▌    | 782/1410 [2:31:51<2:01:51, 11.64s/it]
 56%|█████▌    | 783/1410 [2:32:03<2:01:58, 11.67s/it]
                                                      
{'loss': 0.0077, 'grad_norm': 0.7887884378433228, 'learning_rate': 3.917331875877302e-07, 'kl': 0.0115, 'entropy': -0.0315, 'ce_loss': 0.0036, 'epoch': 1.11}

 56%|█████▌    | 783/1410 [2:32:03<2:01:58, 11.67s/it]
 56%|█████▌    | 784/1410 [2:32:14<2:00:25, 11.54s/it]
                                                      
{'loss': 0.0231, 'grad_norm': 1.4512536525726318, 'learning_rate': 3.907078736395165e-07, 'kl': 0.0339, 'entropy': -0.1001, 'ce_loss': 0.006, 'epoch': 1.11}

 56%|█████▌    | 784/1410 [2:32:14<2:00:25, 11.54s/it]
 56%|█████▌    | 785/1410 [2:32:26<2:01:27, 11.66s/it]
                                                      
{'loss': 0.0154, 'grad_norm': 1.3572810888290405, 'learning_rate': 3.896828728464472e-07, 'kl': 0.0168, 'entropy': -0.0237, 'ce_loss': 0.0451, 'epoch': 1.11}

 56%|█████▌    | 785/1410 [2:32:26<2:01:27, 11.66s/it]
 56%|█████▌    | 786/1410 [2:32:38<2:02:07, 11.74s/it]
                                                      
{'loss': 0.0131, 'grad_norm': 0.9295387268066406, 'learning_rate': 3.886581906221293e-07, 'kl': -0.0069, 'entropy': -0.0273, 'ce_loss': 0.0924, 'epoch': 1.11}

 56%|█████▌    | 786/1410 [2:32:38<2:02:07, 11.74s/it]
 56%|█████▌    | 787/1410 [2:32:50<2:02:33, 11.80s/it]
                                                      
{'loss': 0.0176, 'grad_norm': 0.9339917302131653, 'learning_rate': 3.876338323784876e-07, 'kl': -0.0026, 'entropy': 0.0347, 'ce_loss': 0.1458, 'epoch': 1.12}

 56%|█████▌    | 787/1410 [2:32:50<2:02:33, 11.80s/it]
 56%|█████▌    | 788/1410 [2:33:02<2:02:43, 11.84s/it]
                                                      
{'loss': 0.0135, 'grad_norm': 1.1693412065505981, 'learning_rate': 3.866098035257348e-07, 'kl': 0.0066, 'entropy': -0.0244, 'ce_loss': 0.0006, 'epoch': 1.12}

 56%|█████▌    | 788/1410 [2:33:02<2:02:43, 11.84s/it]
 56%|█████▌    | 789/1410 [2:33:14<2:02:51, 11.87s/it]
                                                      
{'loss': 0.023, 'grad_norm': 1.54367196559906, 'learning_rate': 3.8558610947234535e-07, 'kl': -0.0042, 'entropy': 0.0703, 'ce_loss': 0.1084, 'epoch': 1.12}

 56%|█████▌    | 789/1410 [2:33:14<2:02:51, 11.87s/it]
 56%|█████▌    | 790/1410 [2:33:25<2:00:04, 11.62s/it]
                                                      
{'loss': 0.0128, 'grad_norm': 1.3437317609786987, 'learning_rate': 3.8456275562502396e-07, 'kl': 0.0239, 'entropy': -0.0659, 'ce_loss': 0.0037, 'epoch': 1.12}

 56%|█████▌    | 790/1410 [2:33:25<2:00:04, 11.62s/it]
 56%|█████▌    | 791/1410 [2:33:37<2:01:14, 11.75s/it]
                                                      
{'loss': 0.0292, 'grad_norm': 1.0256023406982422, 'learning_rate': 3.835397473886795e-07, 'kl': 0.0227, 'entropy': -0.0127, 'ce_loss': 0.0414, 'epoch': 1.12}

 56%|█████▌    | 791/1410 [2:33:37<2:01:14, 11.75s/it]
 56%|█████▌    | 792/1410 [2:33:48<1:58:24, 11.50s/it]
                                                      
{'loss': 0.0129, 'grad_norm': 1.3668477535247803, 'learning_rate': 3.825170901663953e-07, 'kl': 0.0312, 'entropy': -0.0942, 'ce_loss': 0.0164, 'epoch': 1.12}

 56%|█████▌    | 792/1410 [2:33:48<1:58:24, 11.50s/it]
 56%|█████▌    | 793/1410 [2:34:00<1:59:12, 11.59s/it]
                                                      
{'loss': 0.0156, 'grad_norm': 1.2315199375152588, 'learning_rate': 3.8149478935940033e-07, 'kl': 0.0046, 'entropy': -0.0396, 'ce_loss': 0.0052, 'epoch': 1.12}

 56%|█████▌    | 793/1410 [2:34:00<1:59:12, 11.59s/it]
 56%|█████▋    | 794/1410 [2:34:11<1:59:21, 11.63s/it]
                                                      
{'loss': 0.0092, 'grad_norm': 1.1706622838974, 'learning_rate': 3.804728503670417e-07, 'kl': -0.008, 'entropy': 0.0679, 'ce_loss': 0.1002, 'epoch': 1.13}

 56%|█████▋    | 794/1410 [2:34:11<1:59:21, 11.63s/it]
 56%|█████▋    | 795/1410 [2:34:23<1:59:42, 11.68s/it]
                                                      
{'loss': 0.011, 'grad_norm': 1.3762112855911255, 'learning_rate': 3.7945127858675534e-07, 'kl': 0.0282, 'entropy': -0.0723, 'ce_loss': 0.0179, 'epoch': 1.13}

 56%|█████▋    | 795/1410 [2:34:23<1:59:42, 11.68s/it]
 56%|█████▋    | 796/1410 [2:34:35<1:59:42, 11.70s/it]
                                                      
{'loss': 0.0108, 'grad_norm': 1.3401551246643066, 'learning_rate': 3.784300794140376e-07, 'kl': -0.0042, 'entropy': -0.0566, 'ce_loss': 0.0474, 'epoch': 1.13}

 56%|█████▋    | 796/1410 [2:34:35<1:59:42, 11.70s/it]
 57%|█████▋    | 797/1410 [2:34:47<1:59:33, 11.70s/it]
                                                      
{'loss': 0.0082, 'grad_norm': 1.5349045991897583, 'learning_rate': 3.7740925824241707e-07, 'kl': -0.0067, 'entropy': 0.0192, 'ce_loss': 0.1067, 'epoch': 1.13}

 57%|█████▋    | 797/1410 [2:34:47<1:59:33, 11.70s/it]
 57%|█████▋    | 798/1410 [2:34:59<2:00:09, 11.78s/it]
                                                      
{'loss': 0.0256, 'grad_norm': 1.0335819721221924, 'learning_rate': 3.7638882046342576e-07, 'kl': 0.0275, 'entropy': -0.0723, 'ce_loss': 0.0146, 'epoch': 1.13}

 57%|█████▋    | 798/1410 [2:34:59<2:00:09, 11.78s/it]
 57%|█████▋    | 799/1410 [2:35:11<2:00:13, 11.81s/it]
                                                      
{'loss': 0.0182, 'grad_norm': 1.002083420753479, 'learning_rate': 3.753687714665709e-07, 'kl': 0.0374, 'entropy': -0.1187, 'ce_loss': 0.0083, 'epoch': 1.13}

 57%|█████▋    | 799/1410 [2:35:11<2:00:13, 11.81s/it]
 57%|█████▋    | 800/1410 [2:35:22<1:58:34, 11.66s/it]
                                                      
{'loss': 0.0176, 'grad_norm': 1.2295546531677246, 'learning_rate': 3.7434911663930646e-07, 'kl': 0.0474, 'entropy': -0.0703, 'ce_loss': 0.0106, 'epoch': 1.13}

 57%|█████▋    | 800/1410 [2:35:22<1:58:34, 11.66s/it]
 57%|█████▋    | 801/1410 [2:35:34<1:58:39, 11.69s/it]
                                                      
{'loss': 0.0143, 'grad_norm': 1.1975197792053223, 'learning_rate': 3.733298613670039e-07, 'kl': -0.0022, 'entropy': 0.0176, 'ce_loss': 0.0574, 'epoch': 1.14}

 57%|█████▋    | 801/1410 [2:35:34<1:58:39, 11.69s/it]
 57%|█████▋    | 802/1410 [2:35:45<1:59:01, 11.75s/it]
                                                      
{'loss': 0.0174, 'grad_norm': 1.0287220478057861, 'learning_rate': 3.7231101103292533e-07, 'kl': -0.0058, 'entropy': -0.0093, 'ce_loss': 0.0532, 'epoch': 1.14}

 57%|█████▋    | 802/1410 [2:35:45<1:59:01, 11.75s/it]
 57%|█████▋    | 803/1410 [2:35:57<1:58:58, 11.76s/it]
                                                      
{'loss': 0.0109, 'grad_norm': 0.9040339589118958, 'learning_rate': 3.7129257101819384e-07, 'kl': -0.0036, 'entropy': 0.0977, 'ce_loss': 0.1705, 'epoch': 1.14}

 57%|█████▋    | 803/1410 [2:35:57<1:58:58, 11.76s/it]
 57%|█████▋    | 804/1410 [2:36:09<1:59:19, 11.81s/it]
                                                      
{'loss': 0.0199, 'grad_norm': 0.8717811107635498, 'learning_rate': 3.7027454670176475e-07, 'kl': 0.0339, 'entropy': -0.0649, 'ce_loss': 0.0298, 'epoch': 1.14}

 57%|█████▋    | 804/1410 [2:36:09<1:59:19, 11.81s/it]
 57%|█████▋    | 805/1410 [2:36:20<1:56:54, 11.59s/it]
                                                      
{'loss': 0.0127, 'grad_norm': 1.0493534803390503, 'learning_rate': 3.692569434603987e-07, 'kl': 0.0258, 'entropy': -0.0649, 'ce_loss': 0.0111, 'epoch': 1.14}

 57%|█████▋    | 805/1410 [2:36:20<1:56:54, 11.59s/it]
 57%|█████▋    | 806/1410 [2:36:32<1:57:23, 11.66s/it]
                                                      
{'loss': 0.0188, 'grad_norm': 1.2794784307479858, 'learning_rate': 3.682397666686323e-07, 'kl': 0.0299, 'entropy': -0.0474, 'ce_loss': 0.0074, 'epoch': 1.14}

 57%|█████▋    | 806/1410 [2:36:32<1:57:23, 11.66s/it]
 57%|█████▋    | 807/1410 [2:36:44<1:57:16, 11.67s/it]
                                                      
{'loss': 0.0084, 'grad_norm': 1.1351956129074097, 'learning_rate': 3.672230216987491e-07, 'kl': 0.0093, 'entropy': -0.0674, 'ce_loss': 0.0082, 'epoch': 1.14}

 57%|█████▋    | 807/1410 [2:36:44<1:57:16, 11.67s/it]
 57%|█████▋    | 808/1410 [2:36:55<1:57:08, 11.67s/it]
                                                      
{'loss': 0.0202, 'grad_norm': 1.1193737983703613, 'learning_rate': 3.6620671392075285e-07, 'kl': 0.0176, 'entropy': -0.0518, 'ce_loss': 0.0075, 'epoch': 1.15}

 57%|█████▋    | 808/1410 [2:36:55<1:57:08, 11.67s/it]
 57%|█████▋    | 809/1410 [2:37:07<1:57:02, 11.69s/it]
                                                      
{'loss': 0.0108, 'grad_norm': 1.0017995834350586, 'learning_rate': 3.6519084870233765e-07, 'kl': -0.006, 'entropy': 0.0303, 'ce_loss': 0.1282, 'epoch': 1.15}

 57%|█████▋    | 809/1410 [2:37:07<1:57:02, 11.69s/it]
 57%|█████▋    | 810/1410 [2:37:18<1:55:07, 11.51s/it]
                                                      
{'loss': 0.0154, 'grad_norm': 1.0720542669296265, 'learning_rate': 3.641754314088604e-07, 'kl': 0.0525, 'entropy': -0.0786, 'ce_loss': 0.0048, 'epoch': 1.15}

 57%|█████▋    | 810/1410 [2:37:18<1:55:07, 11.51s/it]
 58%|█████▊    | 811/1410 [2:37:29<1:53:34, 11.38s/it]
                                                      
{'loss': 0.0236, 'grad_norm': 1.3832440376281738, 'learning_rate': 3.631604674033125e-07, 'kl': 0.0045, 'entropy': -0.0369, 'ce_loss': 0.0089, 'epoch': 1.15}

 58%|█████▊    | 811/1410 [2:37:29<1:53:34, 11.38s/it]
 58%|█████▊    | 812/1410 [2:37:41<1:54:48, 11.52s/it]
                                                      
{'loss': 0.0209, 'grad_norm': 1.3913688659667969, 'learning_rate': 3.621459620462907e-07, 'kl': 0.0352, 'entropy': -0.1147, 'ce_loss': 0.0034, 'epoch': 1.15}

 58%|█████▊    | 812/1410 [2:37:41<1:54:48, 11.52s/it]
 58%|█████▊    | 813/1410 [2:37:53<1:55:15, 11.58s/it]
                                                      
{'loss': 0.0079, 'grad_norm': 2.2638306617736816, 'learning_rate': 3.611319206959699e-07, 'kl': 0.0258, 'entropy': -0.1216, 'ce_loss': 0.0096, 'epoch': 1.15}

 58%|█████▊    | 813/1410 [2:37:53<1:55:15, 11.58s/it]
 58%|█████▊    | 814/1410 [2:38:05<1:56:05, 11.69s/it]
                                                      
{'loss': 0.0165, 'grad_norm': 1.0962402820587158, 'learning_rate': 3.601183487080742e-07, 'kl': 0.0515, 'entropy': -0.0811, 'ce_loss': 0.0143, 'epoch': 1.15}

 58%|█████▊    | 814/1410 [2:38:05<1:56:05, 11.69s/it]
 58%|█████▊    | 815/1410 [2:38:16<1:54:11, 11.51s/it]
                                                      
{'loss': 0.0139, 'grad_norm': 1.2085816860198975, 'learning_rate': 3.591052514358483e-07, 'kl': 0.0408, 'entropy': -0.0688, 'ce_loss': 0.0101, 'epoch': 1.16}

 58%|█████▊    | 815/1410 [2:38:16<1:54:11, 11.51s/it]
 58%|█████▊    | 816/1410 [2:38:27<1:53:09, 11.43s/it]
                                                      
{'loss': 0.0117, 'grad_norm': 1.7052818536758423, 'learning_rate': 3.5809263423003056e-07, 'kl': 0.0232, 'entropy': -0.0664, 'ce_loss': 0.016, 'epoch': 1.16}

 58%|█████▊    | 816/1410 [2:38:27<1:53:09, 11.43s/it]
 58%|█████▊    | 817/1410 [2:38:39<1:53:42, 11.50s/it]
                                                      
{'loss': 0.0081, 'grad_norm': 1.0882209539413452, 'learning_rate': 3.5708050243882277e-07, 'kl': 0.0032, 'entropy': -0.0294, 'ce_loss': 0.0019, 'epoch': 1.16}

 58%|█████▊    | 817/1410 [2:38:39<1:53:42, 11.50s/it]
 58%|█████▊    | 818/1410 [2:38:51<1:53:55, 11.55s/it]
                                                      
{'loss': 0.0153, 'grad_norm': 1.9796266555786133, 'learning_rate': 3.560688614078637e-07, 'kl': 0.0352, 'entropy': -0.0928, 'ce_loss': 0.0205, 'epoch': 1.16}

 58%|█████▊    | 818/1410 [2:38:51<1:53:55, 11.55s/it]
 58%|█████▊    | 819/1410 [2:39:02<1:54:28, 11.62s/it]
                                                      
{'loss': 0.0149, 'grad_norm': 1.2482472658157349, 'learning_rate': 3.5505771648020005e-07, 'kl': 0.041, 'entropy': -0.1147, 'ce_loss': 0.0094, 'epoch': 1.16}

 58%|█████▊    | 819/1410 [2:39:02<1:54:28, 11.62s/it]
 58%|█████▊    | 820/1410 [2:39:14<1:54:42, 11.67s/it]
                                                      
{'loss': 0.009, 'grad_norm': 1.3047866821289062, 'learning_rate': 3.54047072996258e-07, 'kl': -0.0004, 'entropy': 0.0396, 'ce_loss': 0.1051, 'epoch': 1.16}

 58%|█████▊    | 820/1410 [2:39:14<1:54:42, 11.67s/it]
 58%|█████▊    | 821/1410 [2:39:26<1:54:34, 11.67s/it]
                                                      
{'loss': 0.0063, 'grad_norm': 0.9843763113021851, 'learning_rate': 3.5303693629381566e-07, 'kl': -0.0116, 'entropy': -0.0137, 'ce_loss': 0.0041, 'epoch': 1.16}

 58%|█████▊    | 821/1410 [2:39:26<1:54:34, 11.67s/it]
 58%|█████▊    | 822/1410 [2:39:37<1:53:51, 11.62s/it]
                                                      
{'loss': 0.0116, 'grad_norm': 1.4385168552398682, 'learning_rate': 3.520273117079745e-07, 'kl': 0.0461, 'entropy': -0.0693, 'ce_loss': 0.0116, 'epoch': 1.17}

 58%|█████▊    | 822/1410 [2:39:37<1:53:51, 11.62s/it]
 58%|█████▊    | 823/1410 [2:39:49<1:54:00, 11.65s/it]
                                                      
{'loss': 0.0069, 'grad_norm': 1.0297678709030151, 'learning_rate': 3.510182045711308e-07, 'kl': -0.0079, 'entropy': 0.0086, 'ce_loss': 0.1063, 'epoch': 1.17}

 58%|█████▊    | 823/1410 [2:39:49<1:54:00, 11.65s/it]
 58%|█████▊    | 824/1410 [2:40:01<1:54:29, 11.72s/it]
                                                      
{'loss': 0.0246, 'grad_norm': 1.1471927165985107, 'learning_rate': 3.500096202129485e-07, 'kl': -0.0055, 'entropy': 0.0767, 'ce_loss': 0.0896, 'epoch': 1.17}

 58%|█████▊    | 824/1410 [2:40:01<1:54:29, 11.72s/it]
 59%|█████▊    | 825/1410 [2:40:13<1:54:27, 11.74s/it]
                                                      
{'loss': 0.0244, 'grad_norm': 1.5267257690429688, 'learning_rate': 3.4900156396033003e-07, 'kl': -0.0089, 'entropy': -0.0272, 'ce_loss': 0.0067, 'epoch': 1.17}

 59%|█████▊    | 825/1410 [2:40:13<1:54:27, 11.74s/it]
 59%|█████▊    | 826/1410 [2:40:24<1:54:01, 11.72s/it]
                                                      
{'loss': 0.0077, 'grad_norm': 0.9732896685600281, 'learning_rate': 3.4799404113738877e-07, 'kl': -0.0089, 'entropy': 0.0186, 'ce_loss': 0.0738, 'epoch': 1.17}

 59%|█████▊    | 826/1410 [2:40:24<1:54:01, 11.72s/it]
 59%|█████▊    | 827/1410 [2:40:36<1:54:13, 11.76s/it]
                                                      
{'loss': 0.0249, 'grad_norm': 1.0090383291244507, 'learning_rate': 3.46987057065421e-07, 'kl': 0.0003, 'entropy': -0.0845, 'ce_loss': 0.0404, 'epoch': 1.17}

 59%|█████▊    | 827/1410 [2:40:36<1:54:13, 11.76s/it]
 59%|█████▊    | 828/1410 [2:40:48<1:54:14, 11.78s/it]
                                                      
{'loss': 0.0413, 'grad_norm': 1.2539360523223877, 'learning_rate': 3.459806170628767e-07, 'kl': -0.0048, 'entropy': -0.0635, 'ce_loss': 0.0238, 'epoch': 1.17}

 59%|█████▊    | 828/1410 [2:40:48<1:54:14, 11.78s/it]
 59%|█████▉    | 829/1410 [2:40:59<1:52:15, 11.59s/it]
                                                      
{'loss': 0.0143, 'grad_norm': 1.3351556062698364, 'learning_rate': 3.449747264453333e-07, 'kl': 0.009, 'entropy': -0.0684, 'ce_loss': 0.0416, 'epoch': 1.18}

 59%|█████▉    | 829/1410 [2:40:59<1:52:15, 11.59s/it]
 59%|█████▉    | 830/1410 [2:41:11<1:52:36, 11.65s/it]
                                                      
{'loss': 0.0215, 'grad_norm': 1.2781834602355957, 'learning_rate': 3.4396939052546625e-07, 'kl': 0.03, 'entropy': 0.0078, 'ce_loss': 0.0579, 'epoch': 1.18}

 59%|█████▉    | 830/1410 [2:41:11<1:52:36, 11.65s/it]
 59%|█████▉    | 831/1410 [2:41:23<1:52:41, 11.68s/it]
                                                      
{'loss': 0.0257, 'grad_norm': 1.1524333953857422, 'learning_rate': 3.4296461461302104e-07, 'kl': 0.0186, 'entropy': -0.0349, 'ce_loss': 0.0071, 'epoch': 1.18}

 59%|█████▉    | 831/1410 [2:41:23<1:52:41, 11.68s/it]
 59%|█████▉    | 832/1410 [2:41:34<1:52:10, 11.64s/it]
                                                      
{'loss': 0.0083, 'grad_norm': 1.2194125652313232, 'learning_rate': 3.419604040147857e-07, 'kl': 0.0021, 'entropy': -0.0664, 'ce_loss': 0.0309, 'epoch': 1.18}

 59%|█████▉    | 832/1410 [2:41:34<1:52:10, 11.64s/it]
 59%|█████▉    | 833/1410 [2:41:46<1:52:17, 11.68s/it]
                                                      
{'loss': 0.0036, 'grad_norm': 0.6578533053398132, 'learning_rate': 3.4095676403456293e-07, 'kl': -0.0132, 'entropy': -0.0684, 'ce_loss': 0.033, 'epoch': 1.18}

 59%|█████▉    | 833/1410 [2:41:46<1:52:17, 11.68s/it]
 59%|█████▉    | 834/1410 [2:41:58<1:52:33, 11.72s/it]
                                                      
{'loss': 0.0052, 'grad_norm': 1.0107464790344238, 'learning_rate': 3.3995369997314074e-07, 'kl': -0.0121, 'entropy': 0.0815, 'ce_loss': 0.1096, 'epoch': 1.18}

 59%|█████▉    | 834/1410 [2:41:58<1:52:33, 11.72s/it]
 59%|█████▉    | 835/1410 [2:42:09<1:52:01, 11.69s/it]
                                                      
{'loss': 0.0162, 'grad_norm': 1.5745078325271606, 'learning_rate': 3.3895121712826617e-07, 'kl': 0.0072, 'entropy': -0.0437, 'ce_loss': 0.0181, 'epoch': 1.18}

 59%|█████▉    | 835/1410 [2:42:09<1:52:01, 11.69s/it]
 59%|█████▉    | 836/1410 [2:42:21<1:51:40, 11.67s/it]
                                                      
{'loss': 0.0143, 'grad_norm': 1.4004923105239868, 'learning_rate': 3.379493207946161e-07, 'kl': 0.0284, 'entropy': 0.0195, 'ce_loss': 0.0534, 'epoch': 1.19}

 59%|█████▉    | 836/1410 [2:42:21<1:51:40, 11.67s/it]
 59%|█████▉    | 837/1410 [2:42:33<1:51:34, 11.68s/it]
                                                      
{'loss': 0.0125, 'grad_norm': 1.1781889200210571, 'learning_rate': 3.3694801626377e-07, 'kl': -0.0002, 'entropy': -0.0124, 'ce_loss': 0.0063, 'epoch': 1.19}

 59%|█████▉    | 837/1410 [2:42:33<1:51:34, 11.68s/it]
 59%|█████▉    | 838/1410 [2:42:45<1:51:40, 11.71s/it]
                                                      
{'loss': 0.0091, 'grad_norm': 1.0937992334365845, 'learning_rate': 3.3594730882418166e-07, 'kl': -0.0056, 'entropy': -0.0099, 'ce_loss': 0.0871, 'epoch': 1.19}

 59%|█████▉    | 838/1410 [2:42:45<1:51:40, 11.71s/it]
 60%|█████▉    | 839/1410 [2:42:56<1:51:46, 11.75s/it]
                                                      
{'loss': 0.0181, 'grad_norm': 1.726078748703003, 'learning_rate': 3.349472037611509e-07, 'kl': 0.0265, 'entropy': -0.0393, 'ce_loss': 0.019, 'epoch': 1.19}

 60%|█████▉    | 839/1410 [2:42:56<1:51:46, 11.75s/it]
 60%|█████▉    | 840/1410 [2:43:08<1:51:24, 11.73s/it]
                                                      
{'loss': 0.0154, 'grad_norm': 1.2255961894989014, 'learning_rate': 3.3394770635679663e-07, 'kl': 0.0208, 'entropy': -0.0201, 'ce_loss': 0.0359, 'epoch': 1.19}

 60%|█████▉    | 840/1410 [2:43:08<1:51:24, 11.73s/it]
 60%|█████▉    | 841/1410 [2:43:20<1:51:38, 11.77s/it]
                                                      
{'loss': 0.0227, 'grad_norm': 1.4276047945022583, 'learning_rate': 3.3294882189002806e-07, 'kl': 0.0281, 'entropy': -0.0503, 'ce_loss': 0.0044, 'epoch': 1.19}

 60%|█████▉    | 841/1410 [2:43:20<1:51:38, 11.77s/it]
 60%|█████▉    | 842/1410 [2:43:32<1:51:17, 11.76s/it]
                                                      
{'loss': 0.0129, 'grad_norm': 1.1858631372451782, 'learning_rate': 3.31950555636517e-07, 'kl': 0.0183, 'entropy': -0.0591, 'ce_loss': 0.0328, 'epoch': 1.19}

 60%|█████▉    | 842/1410 [2:43:32<1:51:17, 11.76s/it]
 60%|█████▉    | 843/1410 [2:43:43<1:50:52, 11.73s/it]
                                                      
{'loss': 0.0036, 'grad_norm': 0.8876781463623047, 'learning_rate': 3.309529128686706e-07, 'kl': -0.0044, 'entropy': 0.0776, 'ce_loss': 0.1114, 'epoch': 1.2}

 60%|█████▉    | 843/1410 [2:43:43<1:50:52, 11.73s/it]
 60%|█████▉    | 844/1410 [2:43:55<1:50:05, 11.67s/it]
                                                      
{'loss': 0.0112, 'grad_norm': 1.4021522998809814, 'learning_rate': 3.2995589885560235e-07, 'kl': -0.0058, 'entropy': -0.0427, 'ce_loss': 0.0383, 'epoch': 1.2}

 60%|█████▉    | 844/1410 [2:43:55<1:50:05, 11.67s/it]
 60%|█████▉    | 845/1410 [2:44:07<1:50:41, 11.76s/it]
                                                      
{'loss': 0.025, 'grad_norm': 0.7982925772666931, 'learning_rate': 3.2895951886310545e-07, 'kl': -0.0067, 'entropy': -0.0121, 'ce_loss': 0.0729, 'epoch': 1.2}

 60%|█████▉    | 845/1410 [2:44:07<1:50:41, 11.76s/it]
 60%|██████    | 846/1410 [2:44:18<1:50:01, 11.70s/it]
                                                      
{'loss': 0.0132, 'grad_norm': 1.1053167581558228, 'learning_rate': 3.279637781536245e-07, 'kl': 0.0337, 'entropy': -0.1045, 'ce_loss': 0.0097, 'epoch': 1.2}

 60%|██████    | 846/1410 [2:44:18<1:50:01, 11.70s/it]
 60%|██████    | 847/1410 [2:44:30<1:49:30, 11.67s/it]
                                                      
{'loss': 0.0134, 'grad_norm': 1.466830849647522, 'learning_rate': 3.269686819862273e-07, 'kl': 0.0181, 'entropy': -0.0344, 'ce_loss': 0.0051, 'epoch': 1.2}

 60%|██████    | 847/1410 [2:44:30<1:49:30, 11.67s/it]
 60%|██████    | 848/1410 [2:44:41<1:48:36, 11.59s/it]
                                                      
{'loss': 0.0175, 'grad_norm': 1.7581050395965576, 'learning_rate': 3.2597423561657774e-07, 'kl': 0.0347, 'entropy': -0.1069, 'ce_loss': 0.0144, 'epoch': 1.2}

 60%|██████    | 848/1410 [2:44:41<1:48:36, 11.59s/it]
 60%|██████    | 849/1410 [2:44:53<1:48:42, 11.63s/it]
                                                      
{'loss': 0.0129, 'grad_norm': 1.0938283205032349, 'learning_rate': 3.249804442969079e-07, 'kl': 0.0337, 'entropy': -0.0369, 'ce_loss': 0.0057, 'epoch': 1.2}

 60%|██████    | 849/1410 [2:44:53<1:48:42, 11.63s/it]
 60%|██████    | 850/1410 [2:45:04<1:47:26, 11.51s/it]
                                                      
{'loss': 0.0169, 'grad_norm': 1.3414294719696045, 'learning_rate': 3.239873132759896e-07, 'kl': 0.032, 'entropy': -0.0742, 'ce_loss': 0.0307, 'epoch': 1.21}

 60%|██████    | 850/1410 [2:45:04<1:47:26, 11.51s/it]
 60%|██████    | 851/1410 [2:45:16<1:47:55, 11.58s/it]
                                                      
{'loss': 0.0125, 'grad_norm': 1.170225977897644, 'learning_rate': 3.22994847799108e-07, 'kl': -0.0068, 'entropy': 0.0292, 'ce_loss': 0.0816, 'epoch': 1.21}

 60%|██████    | 851/1410 [2:45:16<1:47:55, 11.58s/it]
 60%|██████    | 852/1410 [2:45:28<1:49:12, 11.74s/it]
                                                      
{'loss': 0.0133, 'grad_norm': 1.2378159761428833, 'learning_rate': 3.220030531080326e-07, 'kl': 0.0187, 'entropy': -0.0623, 'ce_loss': 0.0106, 'epoch': 1.21}

 60%|██████    | 852/1410 [2:45:28<1:49:12, 11.74s/it]
 60%|██████    | 853/1410 [2:45:40<1:48:48, 11.72s/it]
                                                      
{'loss': 0.0096, 'grad_norm': 1.2144099473953247, 'learning_rate': 3.210119344409902e-07, 'kl': 0.0317, 'entropy': -0.0193, 'ce_loss': 0.0039, 'epoch': 1.21}

 60%|██████    | 853/1410 [2:45:40<1:48:48, 11.72s/it]
 61%|██████    | 854/1410 [2:45:52<1:48:39, 11.73s/it]
                                                      
{'loss': 0.008, 'grad_norm': 0.873991072177887, 'learning_rate': 3.200214970326375e-07, 'kl': 0.0452, 'entropy': -0.1221, 'ce_loss': 0.0064, 'epoch': 1.21}

 61%|██████    | 854/1410 [2:45:52<1:48:39, 11.73s/it]
 61%|██████    | 855/1410 [2:46:04<1:50:55, 11.99s/it]
                                                      
{'loss': 0.0296, 'grad_norm': 1.3488318920135498, 'learning_rate': 3.1903174611403234e-07, 'kl': -0.0101, 'entropy': -0.0386, 'ce_loss': 0.0989, 'epoch': 1.21}

 61%|██████    | 855/1410 [2:46:04<1:50:55, 11.99s/it]
 61%|██████    | 856/1410 [2:46:16<1:50:55, 12.01s/it]
                                                      
{'loss': 0.0212, 'grad_norm': 1.6486129760742188, 'learning_rate': 3.180426869126076e-07, 'kl': -0.0021, 'entropy': -0.0179, 'ce_loss': 0.064, 'epoch': 1.21}

 61%|██████    | 856/1410 [2:46:16<1:50:55, 12.01s/it]
 61%|██████    | 857/1410 [2:46:28<1:50:00, 11.94s/it]
                                                      
{'loss': 0.0113, 'grad_norm': 1.1131823062896729, 'learning_rate': 3.170543246521426e-07, 'kl': 0.0262, 'entropy': -0.0859, 'ce_loss': 0.0317, 'epoch': 1.22}

 61%|██████    | 857/1410 [2:46:28<1:50:00, 11.94s/it]
 61%|██████    | 858/1410 [2:46:40<1:49:41, 11.92s/it]
                                                      
{'loss': 0.0216, 'grad_norm': 0.998152494430542, 'learning_rate': 3.160666645527353e-07, 'kl': -0.0107, 'entropy': 0.0118, 'ce_loss': 0.1167, 'epoch': 1.22}

 61%|██████    | 858/1410 [2:46:40<1:49:41, 11.92s/it]
 61%|██████    | 859/1410 [2:46:51<1:46:39, 11.61s/it]
                                                      
{'loss': 0.0217, 'grad_norm': 1.1960190534591675, 'learning_rate': 3.150797118307754e-07, 'kl': 0.009, 'entropy': -0.0197, 'ce_loss': 0.0002, 'epoch': 1.22}

 61%|██████    | 859/1410 [2:46:51<1:46:39, 11.61s/it]
 61%|██████    | 860/1410 [2:47:03<1:46:46, 11.65s/it]
                                                      
{'loss': 0.0115, 'grad_norm': 0.8304599523544312, 'learning_rate': 3.1409347169891724e-07, 'kl': 0.007, 'entropy': -0.0432, 'ce_loss': 0.0058, 'epoch': 1.22}

 61%|██████    | 860/1410 [2:47:03<1:46:46, 11.65s/it]
 61%|██████    | 861/1410 [2:47:14<1:46:36, 11.65s/it]
                                                      
{'loss': 0.0051, 'grad_norm': 0.658154308795929, 'learning_rate': 3.131079493660502e-07, 'kl': 0.053, 'entropy': -0.1025, 'ce_loss': 0.0064, 'epoch': 1.22}

 61%|██████    | 861/1410 [2:47:14<1:46:36, 11.65s/it]
 61%|██████    | 862/1410 [2:47:26<1:45:21, 11.54s/it]
                                                      
{'loss': 0.0105, 'grad_norm': 1.0383495092391968, 'learning_rate': 3.1212315003727375e-07, 'kl': 0.0276, 'entropy': -0.0576, 'ce_loss': 0.0039, 'epoch': 1.22}

 61%|██████    | 862/1410 [2:47:26<1:45:21, 11.54s/it]
 61%|██████    | 863/1410 [2:47:38<1:46:26, 11.68s/it]
                                                      
{'loss': 0.0179, 'grad_norm': 0.9063313603401184, 'learning_rate': 3.111390789138681e-07, 'kl': 0.0281, 'entropy': -0.082, 'ce_loss': 0.0337, 'epoch': 1.22}

 61%|██████    | 863/1410 [2:47:38<1:46:26, 11.68s/it]
 61%|██████▏   | 864/1410 [2:47:49<1:45:41, 11.61s/it]
                                                      
{'loss': 0.019, 'grad_norm': 0.9098448157310486, 'learning_rate': 3.1015574119326764e-07, 'kl': -0.0031, 'entropy': 0.0066, 'ce_loss': 0.0763, 'epoch': 1.23}

 61%|██████▏   | 864/1410 [2:47:49<1:45:41, 11.61s/it]
 61%|██████▏   | 865/1410 [2:48:00<1:43:58, 11.45s/it]
                                                      
{'loss': 0.0353, 'grad_norm': 1.1258533000946045, 'learning_rate': 3.091731420690333e-07, 'kl': 0.0206, 'entropy': -0.0576, 'ce_loss': 0.0049, 'epoch': 1.23}

 61%|██████▏   | 865/1410 [2:48:00<1:43:58, 11.45s/it]
 61%|██████▏   | 866/1410 [2:48:11<1:42:53, 11.35s/it]
                                                      
{'loss': 0.0103, 'grad_norm': 1.3621550798416138, 'learning_rate': 3.081912867308249e-07, 'kl': 0.0255, 'entropy': -0.043, 'ce_loss': 0.0057, 'epoch': 1.23}

 61%|██████▏   | 866/1410 [2:48:11<1:42:53, 11.35s/it]
 61%|██████▏   | 867/1410 [2:48:23<1:44:26, 11.54s/it]
                                                      
{'loss': 0.0259, 'grad_norm': 2.151097297668457, 'learning_rate': 3.07210180364374e-07, 'kl': 0.0432, 'entropy': 0.0752, 'ce_loss': 0.1002, 'epoch': 1.23}

 61%|██████▏   | 867/1410 [2:48:23<1:44:26, 11.54s/it]
 62%|██████▏   | 868/1410 [2:48:35<1:45:03, 11.63s/it]
                                                      
{'loss': 0.0263, 'grad_norm': 1.2480928897857666, 'learning_rate': 3.0622982815145655e-07, 'kl': -0.0072, 'entropy': 0.0454, 'ce_loss': 0.1324, 'epoch': 1.23}

 62%|██████▏   | 868/1410 [2:48:35<1:45:03, 11.63s/it]
 62%|██████▏   | 869/1410 [2:48:47<1:45:10, 11.66s/it]
                                                      
{'loss': 0.0079, 'grad_norm': 0.915510356426239, 'learning_rate': 3.0525023526986503e-07, 'kl': 0.0012, 'entropy': -0.0127, 'ce_loss': 0.0001, 'epoch': 1.23}

 62%|██████▏   | 869/1410 [2:48:47<1:45:10, 11.66s/it]
 62%|██████▏   | 870/1410 [2:48:59<1:45:32, 11.73s/it]
                                                      
{'loss': 0.0209, 'grad_norm': 1.1374857425689697, 'learning_rate': 3.0427140689338193e-07, 'kl': 0.0194, 'entropy': -0.0752, 'ce_loss': 0.015, 'epoch': 1.23}

 62%|██████▏   | 870/1410 [2:48:59<1:45:32, 11.73s/it]
 62%|██████▏   | 871/1410 [2:49:11<1:47:43, 11.99s/it]
                                                      
{'loss': 0.0185, 'grad_norm': 1.2436333894729614, 'learning_rate': 3.0329334819175135e-07, 'kl': 0.0175, 'entropy': -0.082, 'ce_loss': 0.013, 'epoch': 1.24}

 62%|██████▏   | 871/1410 [2:49:11<1:47:43, 11.99s/it]
 62%|██████▏   | 872/1410 [2:49:23<1:46:46, 11.91s/it]
                                                      
{'loss': 0.0111, 'grad_norm': 1.163177251815796, 'learning_rate': 3.023160643306529e-07, 'kl': 0.0161, 'entropy': -0.0297, 'ce_loss': 0.0046, 'epoch': 1.24}

 62%|██████▏   | 872/1410 [2:49:23<1:46:46, 11.91s/it]
 62%|██████▏   | 873/1410 [2:49:35<1:46:34, 11.91s/it]
                                                      
{'loss': 0.0157, 'grad_norm': 1.0770584344863892, 'learning_rate': 3.013395604716736e-07, 'kl': 0.0344, 'entropy': -0.0806, 'ce_loss': 0.0072, 'epoch': 1.24}

 62%|██████▏   | 873/1410 [2:49:35<1:46:34, 11.91s/it]
 62%|██████▏   | 874/1410 [2:49:46<1:44:28, 11.69s/it]
                                                      
{'loss': 0.0157, 'grad_norm': 1.390581727027893, 'learning_rate': 3.003638417722807e-07, 'kl': 0.0134, 'entropy': -0.041, 'ce_loss': 0.0018, 'epoch': 1.24}

 62%|██████▏   | 874/1410 [2:49:46<1:44:28, 11.69s/it]
 62%|██████▏   | 875/1410 [2:49:58<1:44:19, 11.70s/it]
                                                      
{'loss': 0.0077, 'grad_norm': 1.0328079462051392, 'learning_rate': 2.993889133857947e-07, 'kl': 0.0003, 'entropy': -0.0601, 'ce_loss': 0.0429, 'epoch': 1.24}

 62%|██████▏   | 875/1410 [2:49:58<1:44:19, 11.70s/it]
 62%|██████▏   | 876/1410 [2:50:10<1:46:01, 11.91s/it]
                                                      
{'loss': 0.0248, 'grad_norm': 1.1377187967300415, 'learning_rate': 2.9841478046136223e-07, 'kl': 0.0408, 'entropy': -0.106, 'ce_loss': 0.0063, 'epoch': 1.24}

 62%|██████▏   | 876/1410 [2:50:10<1:46:01, 11.91s/it]
 62%|██████▏   | 877/1410 [2:50:21<1:42:59, 11.59s/it]
                                                      
{'loss': 0.025, 'grad_norm': 1.1459407806396484, 'learning_rate': 2.974414481439282e-07, 'kl': 0.0508, 'entropy': -0.0361, 'ce_loss': 0.0788, 'epoch': 1.24}

 62%|██████▏   | 877/1410 [2:50:21<1:42:59, 11.59s/it]
 62%|██████▏   | 878/1410 [2:50:33<1:43:55, 11.72s/it]
                                                      
{'loss': 0.0133, 'grad_norm': 1.200063943862915, 'learning_rate': 2.964689215742093e-07, 'kl': -0.008, 'entropy': 0.0242, 'ce_loss': 0.0678, 'epoch': 1.25}

 62%|██████▏   | 878/1410 [2:50:33<1:43:55, 11.72s/it]
 62%|██████▏   | 879/1410 [2:50:45<1:43:52, 11.74s/it]
                                                      
{'loss': 0.0079, 'grad_norm': 0.8302594423294067, 'learning_rate': 2.954972058886667e-07, 'kl': 0.0267, 'entropy': -0.0684, 'ce_loss': 0.0236, 'epoch': 1.25}

 62%|██████▏   | 879/1410 [2:50:45<1:43:52, 11.74s/it]
 62%|██████▏   | 880/1410 [2:50:57<1:44:12, 11.80s/it]
                                                      
{'loss': 0.0073, 'grad_norm': 0.9988616108894348, 'learning_rate': 2.945263062194786e-07, 'kl': 0.0198, 'entropy': -0.0613, 'ce_loss': 0.0125, 'epoch': 1.25}

 62%|██████▏   | 880/1410 [2:50:57<1:44:12, 11.80s/it]
 62%|██████▏   | 881/1410 [2:51:09<1:44:32, 11.86s/it]
                                                      
{'loss': 0.0149, 'grad_norm': 1.4006109237670898, 'learning_rate': 2.935562276945138e-07, 'kl': 0.0332, 'entropy': -0.0713, 'ce_loss': 0.0078, 'epoch': 1.25}

 62%|██████▏   | 881/1410 [2:51:09<1:44:32, 11.86s/it]
 63%|██████▎   | 882/1410 [2:51:21<1:44:11, 11.84s/it]
                                                      
{'loss': 0.0295, 'grad_norm': 1.5024937391281128, 'learning_rate': 2.9258697543730325e-07, 'kl': 0.0014, 'entropy': -0.012, 'ce_loss': 0.0008, 'epoch': 1.25}

 63%|██████▎   | 882/1410 [2:51:21<1:44:11, 11.84s/it]
 63%|██████▎   | 883/1410 [2:51:32<1:43:53, 11.83s/it]
                                                      
{'loss': 0.0154, 'grad_norm': 1.4077341556549072, 'learning_rate': 2.91618554567015e-07, 'kl': 0.0244, 'entropy': -0.0898, 'ce_loss': 0.0098, 'epoch': 1.25}

 63%|██████▎   | 883/1410 [2:51:32<1:43:53, 11.83s/it]
 63%|██████▎   | 884/1410 [2:51:44<1:43:31, 11.81s/it]
                                                      
{'loss': 0.0115, 'grad_norm': 1.085497498512268, 'learning_rate': 2.906509701984256e-07, 'kl': 0.0334, 'entropy': -0.106, 'ce_loss': 0.0143, 'epoch': 1.25}

 63%|██████▎   | 884/1410 [2:51:44<1:43:31, 11.81s/it]
 63%|██████▎   | 885/1410 [2:51:56<1:43:35, 11.84s/it]
                                                      
{'loss': 0.0229, 'grad_norm': 1.2335909605026245, 'learning_rate': 2.896842274418934e-07, 'kl': 0.0292, 'entropy': -0.0693, 'ce_loss': 0.0103, 'epoch': 1.26}

 63%|██████▎   | 885/1410 [2:51:56<1:43:35, 11.84s/it]
 63%|██████▎   | 886/1410 [2:52:08<1:43:41, 11.87s/it]
                                                      
{'loss': 0.0097, 'grad_norm': 1.075945496559143, 'learning_rate': 2.887183314033317e-07, 'kl': -0.0053, 'entropy': -0.0154, 'ce_loss': 0.0401, 'epoch': 1.26}

 63%|██████▎   | 886/1410 [2:52:08<1:43:41, 11.87s/it]
 63%|██████▎   | 887/1410 [2:52:20<1:43:27, 11.87s/it]
                                                      
{'loss': 0.0098, 'grad_norm': 1.0378262996673584, 'learning_rate': 2.877532871841826e-07, 'kl': 0.0162, 'entropy': -0.062, 'ce_loss': 0.0036, 'epoch': 1.26}

 63%|██████▎   | 887/1410 [2:52:20<1:43:27, 11.87s/it]
 63%|██████▎   | 888/1410 [2:52:32<1:43:27, 11.89s/it]
                                                      
{'loss': 0.0342, 'grad_norm': 1.1730644702911377, 'learning_rate': 2.867890998813881e-07, 'kl': 0.0272, 'entropy': -0.0664, 'ce_loss': 0.0298, 'epoch': 1.26}

 63%|██████▎   | 888/1410 [2:52:32<1:43:27, 11.89s/it]
 63%|██████▎   | 889/1410 [2:52:44<1:43:33, 11.93s/it]
                                                      
{'loss': 0.0124, 'grad_norm': 0.8807491660118103, 'learning_rate': 2.8582577458736503e-07, 'kl': -0.0075, 'entropy': 0.1543, 'ce_loss': 0.1617, 'epoch': 1.26}

 63%|██████▎   | 889/1410 [2:52:44<1:43:33, 11.93s/it]
 63%|██████▎   | 890/1410 [2:52:56<1:42:54, 11.87s/it]
                                                      
{'loss': 0.0172, 'grad_norm': 1.215038776397705, 'learning_rate': 2.848633163899774e-07, 'kl': 0.0105, 'entropy': -0.0459, 'ce_loss': 0.0313, 'epoch': 1.26}

 63%|██████▎   | 890/1410 [2:52:56<1:42:54, 11.87s/it]
 63%|██████▎   | 891/1410 [2:53:07<1:40:24, 11.61s/it]
                                                      
{'loss': 0.0159, 'grad_norm': 1.0797910690307617, 'learning_rate': 2.839017303725094e-07, 'kl': 0.0294, 'entropy': -0.013, 'ce_loss': 0.0843, 'epoch': 1.26}

 63%|██████▎   | 891/1410 [2:53:07<1:40:24, 11.61s/it]
 63%|██████▎   | 892/1410 [2:53:18<1:41:01, 11.70s/it]
                                                      
{'loss': 0.0099, 'grad_norm': 0.9783384799957275, 'learning_rate': 2.8294102161363907e-07, 'kl': -0.0068, 'entropy': 0.0391, 'ce_loss': 0.0686, 'epoch': 1.27}

 63%|██████▎   | 892/1410 [2:53:18<1:41:01, 11.70s/it]
 63%|██████▎   | 893/1410 [2:53:30<1:40:32, 11.67s/it]
                                                      
{'loss': 0.0116, 'grad_norm': 0.9350097179412842, 'learning_rate': 2.819811951874108e-07, 'kl': -0.0032, 'entropy': 0.0101, 'ce_loss': 0.0644, 'epoch': 1.27}

 63%|██████▎   | 893/1410 [2:53:30<1:40:32, 11.67s/it]
 63%|██████▎   | 894/1410 [2:53:41<1:38:23, 11.44s/it]
                                                      
{'loss': 0.0156, 'grad_norm': 1.591884732246399, 'learning_rate': 2.8102225616320905e-07, 'kl': 0.0437, 'entropy': -0.0928, 'ce_loss': 0.0069, 'epoch': 1.27}

 63%|██████▎   | 894/1410 [2:53:41<1:38:23, 11.44s/it]
 63%|██████▎   | 895/1410 [2:53:53<1:39:26, 11.59s/it]
                                                      
{'loss': 0.0186, 'grad_norm': 0.9218676686286926, 'learning_rate': 2.800642096057315e-07, 'kl': 0.0369, 'entropy': -0.1387, 'ce_loss': 0.0104, 'epoch': 1.27}

 63%|██████▎   | 895/1410 [2:53:53<1:39:26, 11.59s/it]
 64%|██████▎   | 896/1410 [2:54:05<1:39:39, 11.63s/it]
                                                      
{'loss': 0.0058, 'grad_norm': 0.8866950273513794, 'learning_rate': 2.791070605749619e-07, 'kl': -0.0004, 'entropy': 0.168, 'ce_loss': 0.1387, 'epoch': 1.27}

 64%|██████▎   | 896/1410 [2:54:05<1:39:39, 11.63s/it]
 64%|██████▎   | 897/1410 [2:54:16<1:39:13, 11.61s/it]
                                                      
{'loss': 0.0126, 'grad_norm': 1.3010021448135376, 'learning_rate': 2.7815081412614404e-07, 'kl': 0.0227, 'entropy': -0.0564, 'ce_loss': 0.0104, 'epoch': 1.27}

 64%|██████▎   | 897/1410 [2:54:16<1:39:13, 11.61s/it]
 64%|██████▎   | 898/1410 [2:54:28<1:40:01, 11.72s/it]
                                                      
{'loss': 0.0132, 'grad_norm': 1.9689220190048218, 'learning_rate': 2.771954753097546e-07, 'kl': 0.028, 'entropy': -0.0835, 'ce_loss': 0.0225, 'epoch': 1.27}

 64%|██████▎   | 898/1410 [2:54:28<1:40:01, 11.72s/it]
 64%|██████▍   | 899/1410 [2:54:40<1:40:07, 11.76s/it]
                                                      
{'loss': 0.0308, 'grad_norm': 1.1530218124389648, 'learning_rate': 2.7624104917147603e-07, 'kl': 0.0374, 'entropy': 0.0238, 'ce_loss': 0.0829, 'epoch': 1.28}

 64%|██████▍   | 899/1410 [2:54:40<1:40:07, 11.76s/it]
 64%|██████▍   | 900/1410 [2:54:52<1:39:55, 11.76s/it]
                                                      
{'loss': 0.0102, 'grad_norm': 1.0441948175430298, 'learning_rate': 2.752875407521713e-07, 'kl': 0.0001, 'entropy': -0.0142, 'ce_loss': 0.0002, 'epoch': 1.28}

 64%|██████▍   | 900/1410 [2:54:52<1:39:55, 11.76s/it]
 64%|██████▍   | 901/1410 [2:55:04<1:39:45, 11.76s/it]
                                                      
{'loss': 0.0129, 'grad_norm': 1.188940167427063, 'learning_rate': 2.7433495508785545e-07, 'kl': -0.0128, 'entropy': 0.0179, 'ce_loss': 0.0723, 'epoch': 1.28}

 64%|██████▍   | 901/1410 [2:55:04<1:39:45, 11.76s/it]
 64%|██████▍   | 902/1410 [2:55:15<1:39:31, 11.76s/it]
                                                      
{'loss': 0.0101, 'grad_norm': 0.9837133288383484, 'learning_rate': 2.733832972096706e-07, 'kl': -0.0071, 'entropy': 0.0069, 'ce_loss': 0.0621, 'epoch': 1.28}

 64%|██████▍   | 902/1410 [2:55:15<1:39:31, 11.76s/it]
 64%|██████▍   | 903/1410 [2:55:27<1:38:03, 11.61s/it]
                                                      
{'loss': 0.0157, 'grad_norm': 1.491588830947876, 'learning_rate': 2.7243257214385846e-07, 'kl': 0.0282, 'entropy': -0.0762, 'ce_loss': 0.0186, 'epoch': 1.28}

 64%|██████▍   | 903/1410 [2:55:27<1:38:03, 11.61s/it]
 64%|██████▍   | 904/1410 [2:55:38<1:38:43, 11.71s/it]
                                                      
{'loss': 0.0072, 'grad_norm': 0.8255116939544678, 'learning_rate': 2.714827849117341e-07, 'kl': 0.0026, 'entropy': 0.167, 'ce_loss': 0.2194, 'epoch': 1.28}

 64%|██████▍   | 904/1410 [2:55:38<1:38:43, 11.71s/it]
 64%|██████▍   | 905/1410 [2:55:50<1:39:07, 11.78s/it]
                                                      
{'loss': 0.025, 'grad_norm': 1.2790336608886719, 'learning_rate': 2.705339405296592e-07, 'kl': 0.0488, 'entropy': -0.1299, 'ce_loss': 0.0088, 'epoch': 1.28}

 64%|██████▍   | 905/1410 [2:55:50<1:39:07, 11.78s/it]
 64%|██████▍   | 906/1410 [2:56:02<1:39:22, 11.83s/it]
                                                      
{'loss': 0.0173, 'grad_norm': 0.9083985090255737, 'learning_rate': 2.695860440090161e-07, 'kl': 0.021, 'entropy': -0.0435, 'ce_loss': 0.0385, 'epoch': 1.29}

 64%|██████▍   | 906/1410 [2:56:02<1:39:22, 11.83s/it]
 64%|██████▍   | 907/1410 [2:56:14<1:39:26, 11.86s/it]
                                                      
{'loss': 0.0161, 'grad_norm': 1.029339075088501, 'learning_rate': 2.686391003561804e-07, 'kl': 0.0208, 'entropy': 0.0564, 'ce_loss': 0.0901, 'epoch': 1.29}

 64%|██████▍   | 907/1410 [2:56:14<1:39:26, 11.86s/it]
 64%|██████▍   | 908/1410 [2:56:26<1:38:17, 11.75s/it]
                                                      
{'loss': 0.0129, 'grad_norm': 1.391869306564331, 'learning_rate': 2.6769311457249565e-07, 'kl': 0.0356, 'entropy': -0.103, 'ce_loss': 0.0061, 'epoch': 1.29}

 64%|██████▍   | 908/1410 [2:56:26<1:38:17, 11.75s/it]
 64%|██████▍   | 909/1410 [2:56:38<1:38:52, 11.84s/it]
                                                      
{'loss': 0.0252, 'grad_norm': 1.6157104969024658, 'learning_rate': 2.667480916542456e-07, 'kl': 0.0136, 'entropy': -0.0243, 'ce_loss': 0.0006, 'epoch': 1.29}

 64%|██████▍   | 909/1410 [2:56:38<1:38:52, 11.84s/it]
 65%|██████▍   | 910/1410 [2:56:49<1:37:21, 11.68s/it]
                                                      
{'loss': 0.0145, 'grad_norm': 1.292135238647461, 'learning_rate': 2.6580403659262945e-07, 'kl': 0.0012, 'entropy': -0.0299, 'ce_loss': 0.0303, 'epoch': 1.29}

 65%|██████▍   | 910/1410 [2:56:49<1:37:21, 11.68s/it]
 65%|██████▍   | 911/1410 [2:57:00<1:35:35, 11.49s/it]
                                                      
{'loss': 0.0139, 'grad_norm': 27.64780044555664, 'learning_rate': 2.6486095437373415e-07, 'kl': 0.0415, 'entropy': -0.0986, 'ce_loss': 0.0112, 'epoch': 1.29}

 65%|██████▍   | 911/1410 [2:57:00<1:35:35, 11.49s/it]
 65%|██████▍   | 912/1410 [2:57:12<1:36:45, 11.66s/it]
                                                      
{'loss': 0.0189, 'grad_norm': 0.7208298444747925, 'learning_rate': 2.6391884997850835e-07, 'kl': 0.0156, 'entropy': -0.0583, 'ce_loss': 0.0056, 'epoch': 1.29}

 65%|██████▍   | 912/1410 [2:57:12<1:36:45, 11.66s/it]
 65%|██████▍   | 913/1410 [2:57:25<1:38:33, 11.90s/it]
                                                      
{'loss': 0.0263, 'grad_norm': 0.887376070022583, 'learning_rate': 2.6297772838273634e-07, 'kl': 0.0337, 'entropy': -0.1084, 'ce_loss': 0.0157, 'epoch': 1.3}

 65%|██████▍   | 913/1410 [2:57:25<1:38:33, 11.90s/it]
 65%|██████▍   | 914/1410 [2:57:36<1:37:37, 11.81s/it]
                                                      
{'loss': 0.0121, 'grad_norm': 1.105659008026123, 'learning_rate': 2.6203759455701224e-07, 'kl': 0.015, 'entropy': -0.061, 'ce_loss': 0.0236, 'epoch': 1.3}

 65%|██████▍   | 914/1410 [2:57:36<1:37:37, 11.81s/it]
 65%|██████▍   | 915/1410 [2:57:48<1:37:27, 11.81s/it]
                                                      
{'loss': 0.0151, 'grad_norm': 1.2228854894638062, 'learning_rate': 2.610984534667126e-07, 'kl': 0.0064, 'entropy': -0.0211, 'ce_loss': 0.0018, 'epoch': 1.3}

 65%|██████▍   | 915/1410 [2:57:48<1:37:27, 11.81s/it]
 65%|██████▍   | 916/1410 [2:58:00<1:37:05, 11.79s/it]
                                                      
{'loss': 0.0133, 'grad_norm': 1.117708444595337, 'learning_rate': 2.601603100719709e-07, 'kl': 0.0309, 'entropy': -0.0869, 'ce_loss': 0.0098, 'epoch': 1.3}

 65%|██████▍   | 916/1410 [2:58:00<1:37:05, 11.79s/it]
 65%|██████▌   | 917/1410 [2:58:12<1:38:01, 11.93s/it]
                                                      
{'loss': 0.0217, 'grad_norm': 1.2083262205123901, 'learning_rate': 2.5922316932765116e-07, 'kl': -0.0051, 'entropy': -0.0537, 'ce_loss': 0.0325, 'epoch': 1.3}

 65%|██████▌   | 917/1410 [2:58:12<1:38:01, 11.93s/it]
 65%|██████▌   | 918/1410 [2:58:24<1:37:46, 11.92s/it]
                                                      
{'loss': 0.0081, 'grad_norm': 1.0879802703857422, 'learning_rate': 2.58287036183322e-07, 'kl': 0.0344, 'entropy': -0.0786, 'ce_loss': 0.0129, 'epoch': 1.3}

 65%|██████▌   | 918/1410 [2:58:24<1:37:46, 11.92s/it]
 65%|██████▌   | 919/1410 [2:58:35<1:36:21, 11.78s/it]
                                                      
{'loss': 0.0146, 'grad_norm': 1.118453860282898, 'learning_rate': 2.5735191558323054e-07, 'kl': 0.0129, 'entropy': -0.0442, 'ce_loss': 0.0075, 'epoch': 1.3}

 65%|██████▌   | 919/1410 [2:58:35<1:36:21, 11.78s/it]
 65%|██████▌   | 920/1410 [2:58:48<1:37:39, 11.96s/it]
                                                      
{'loss': 0.0144, 'grad_norm': 1.1383519172668457, 'learning_rate': 2.564178124662755e-07, 'kl': 0.0002, 'entropy': 0.1406, 'ce_loss': 0.2314, 'epoch': 1.3}

 65%|██████▌   | 920/1410 [2:58:48<1:37:39, 11.96s/it]
 65%|██████▌   | 921/1410 [2:59:00<1:37:10, 11.92s/it]
                                                      
{'loss': 0.006, 'grad_norm': 0.6886287331581116, 'learning_rate': 2.5548473176598213e-07, 'kl': -0.0092, 'entropy': -0.0115, 'ce_loss': 0.0895, 'epoch': 1.31}

 65%|██████▌   | 921/1410 [2:59:00<1:37:10, 11.92s/it]
 65%|██████▌   | 922/1410 [2:59:12<1:36:51, 11.91s/it]
                                                      
{'loss': 0.0147, 'grad_norm': 0.868649959564209, 'learning_rate': 2.5455267841047575e-07, 'kl': 0.0532, 'entropy': -0.0835, 'ce_loss': 0.0133, 'epoch': 1.31}

 65%|██████▌   | 922/1410 [2:59:12<1:36:51, 11.91s/it]
 65%|██████▌   | 923/1410 [2:59:23<1:35:14, 11.73s/it]
                                                      
{'loss': 0.0143, 'grad_norm': 1.327020525932312, 'learning_rate': 2.536216573224555e-07, 'kl': -0.0024, 'entropy': -0.0164, 'ce_loss': 0.0018, 'epoch': 1.31}

 65%|██████▌   | 923/1410 [2:59:23<1:35:14, 11.73s/it]
 66%|██████▌   | 924/1410 [2:59:35<1:35:01, 11.73s/it]
                                                      
{'loss': 0.011, 'grad_norm': 0.9757272601127625, 'learning_rate': 2.5269167341916844e-07, 'kl': 0.0352, 'entropy': -0.0962, 'ce_loss': 0.0222, 'epoch': 1.31}

 66%|██████▌   | 924/1410 [2:59:35<1:35:01, 11.73s/it]
 66%|██████▌   | 925/1410 [2:59:46<1:34:55, 11.74s/it]
                                                      
{'loss': 0.0204, 'grad_norm': 1.0678282976150513, 'learning_rate': 2.517627316123841e-07, 'kl': 0.0354, 'entropy': -0.1089, 'ce_loss': 0.0084, 'epoch': 1.31}

 66%|██████▌   | 925/1410 [2:59:46<1:34:55, 11.74s/it]
 66%|██████▌   | 926/1410 [2:59:58<1:35:03, 11.78s/it]
                                                      
{'loss': 0.0128, 'grad_norm': 1.1760159730911255, 'learning_rate': 2.508348368083677e-07, 'kl': 0.026, 'entropy': -0.042, 'ce_loss': 0.0189, 'epoch': 1.31}

 66%|██████▌   | 926/1410 [2:59:58<1:35:03, 11.78s/it]
 66%|██████▌   | 927/1410 [3:00:10<1:34:58, 11.80s/it]
                                                      
{'loss': 0.0144, 'grad_norm': 1.2382686138153076, 'learning_rate': 2.499079939078547e-07, 'kl': -0.0079, 'entropy': 0.0562, 'ce_loss': 0.1091, 'epoch': 1.31}

 66%|██████▌   | 927/1410 [3:00:10<1:34:58, 11.80s/it]
 66%|██████▌   | 928/1410 [3:00:22<1:35:03, 11.83s/it]
                                                      
{'loss': 0.0336, 'grad_norm': 1.37477707862854, 'learning_rate': 2.4898220780602497e-07, 'kl': 0.0508, 'entropy': -0.0864, 'ce_loss': 0.0103, 'epoch': 1.32}

 66%|██████▌   | 928/1410 [3:00:22<1:35:03, 11.83s/it]
 66%|██████▌   | 929/1410 [3:00:34<1:34:42, 11.81s/it]
                                                      
{'loss': 0.011, 'grad_norm': 1.0920331478118896, 'learning_rate': 2.48057483392477e-07, 'kl': 0.04, 'entropy': -0.0991, 'ce_loss': 0.0074, 'epoch': 1.32}

 66%|██████▌   | 929/1410 [3:00:34<1:34:42, 11.81s/it]
 66%|██████▌   | 930/1410 [3:00:45<1:34:07, 11.77s/it]
                                                      
{'loss': 0.0078, 'grad_norm': 0.9941506385803223, 'learning_rate': 2.471338255512016e-07, 'kl': 0.0034, 'entropy': -0.0209, 'ce_loss': 0.0011, 'epoch': 1.32}

 66%|██████▌   | 930/1410 [3:00:45<1:34:07, 11.77s/it]
 66%|██████▌   | 931/1410 [3:00:57<1:33:15, 11.68s/it]
                                                      
{'loss': 0.011, 'grad_norm': 1.0144903659820557, 'learning_rate': 2.462112391605564e-07, 'kl': 0.0334, 'entropy': -0.0947, 'ce_loss': 0.0112, 'epoch': 1.32}

 66%|██████▌   | 931/1410 [3:00:57<1:33:15, 11.68s/it]
 66%|██████▌   | 932/1410 [3:01:09<1:33:05, 11.68s/it]
                                                      
{'loss': 0.0112, 'grad_norm': 1.2440285682678223, 'learning_rate': 2.452897290932405e-07, 'kl': -0.0058, 'entropy': 0.0192, 'ce_loss': 0.0909, 'epoch': 1.32}

 66%|██████▌   | 932/1410 [3:01:09<1:33:05, 11.68s/it]
 66%|██████▌   | 933/1410 [3:01:21<1:34:45, 11.92s/it]
                                                      
{'loss': 0.0156, 'grad_norm': 0.9358482360839844, 'learning_rate': 2.443693002162679e-07, 'kl': 0.0199, 'entropy': -0.0674, 'ce_loss': 0.0249, 'epoch': 1.32}

 66%|██████▌   | 933/1410 [3:01:21<1:34:45, 11.92s/it]
 66%|██████▌   | 934/1410 [3:01:33<1:35:05, 11.99s/it]
                                                      
{'loss': 0.0317, 'grad_norm': 1.759408950805664, 'learning_rate': 2.434499573909424e-07, 'kl': 0.0417, 'entropy': 0.0181, 'ce_loss': 0.0766, 'epoch': 1.32}

 66%|██████▌   | 934/1410 [3:01:33<1:35:05, 11.99s/it]
 66%|██████▋   | 935/1410 [3:01:45<1:34:22, 11.92s/it]
                                                      
{'loss': 0.0112, 'grad_norm': 1.2301894426345825, 'learning_rate': 2.425317054728317e-07, 'kl': -0.0004, 'entropy': -0.0096, 'ce_loss': 0.0011, 'epoch': 1.33}

 66%|██████▋   | 935/1410 [3:01:45<1:34:22, 11.92s/it]
 66%|██████▋   | 936/1410 [3:01:56<1:31:49, 11.62s/it]
                                                      
{'loss': 0.0057, 'grad_norm': 0.7348238229751587, 'learning_rate': 2.4161454931174173e-07, 'kl': -0.0031, 'entropy': -0.0659, 'ce_loss': 0.04, 'epoch': 1.33}

 66%|██████▋   | 936/1410 [3:01:56<1:31:49, 11.62s/it]
 66%|██████▋   | 937/1410 [3:02:08<1:32:21, 11.71s/it]
                                                      
{'loss': 0.0269, 'grad_norm': 1.2964808940887451, 'learning_rate': 2.4069849375169147e-07, 'kl': 0.012, 'entropy': -0.0229, 'ce_loss': 0.0021, 'epoch': 1.33}

 66%|██████▋   | 937/1410 [3:02:08<1:32:21, 11.71s/it]
 67%|██████▋   | 938/1410 [3:02:19<1:30:44, 11.54s/it]
                                                      
{'loss': 0.0144, 'grad_norm': 1.4072613716125488, 'learning_rate': 2.3978354363088653e-07, 'kl': 0.0327, 'entropy': -0.083, 'ce_loss': 0.0103, 'epoch': 1.33}

 67%|██████▋   | 938/1410 [3:02:19<1:30:44, 11.54s/it]
 67%|██████▋   | 939/1410 [3:02:31<1:31:10, 11.61s/it]
                                                      
{'loss': 0.0277, 'grad_norm': 1.32077157497406, 'learning_rate': 2.3886970378169426e-07, 'kl': 0.0227, 'entropy': -0.0669, 'ce_loss': 0.0162, 'epoch': 1.33}

 67%|██████▋   | 939/1410 [3:02:31<1:31:10, 11.61s/it]
 67%|██████▋   | 940/1410 [3:02:42<1:29:39, 11.45s/it]
                                                      
{'loss': 0.0111, 'grad_norm': 0.973384439945221, 'learning_rate': 2.379569790306183e-07, 'kl': -0.0041, 'entropy': -0.0245, 'ce_loss': 0.0475, 'epoch': 1.33}

 67%|██████▋   | 940/1410 [3:02:42<1:29:39, 11.45s/it]
 67%|██████▋   | 941/1410 [3:02:53<1:29:48, 11.49s/it]
                                                      
{'loss': 0.0121, 'grad_norm': 1.126012921333313, 'learning_rate': 2.3704537419827254e-07, 'kl': 0.0283, 'entropy': -0.0757, 'ce_loss': 0.0096, 'epoch': 1.33}

 67%|██████▋   | 941/1410 [3:02:53<1:29:48, 11.49s/it]
 67%|██████▋   | 942/1410 [3:03:05<1:30:44, 11.63s/it]
                                                      
{'loss': 0.0285, 'grad_norm': 1.836732268333435, 'learning_rate': 2.361348940993557e-07, 'kl': 0.0215, 'entropy': 0.0145, 'ce_loss': 0.0688, 'epoch': 1.34}

 67%|██████▋   | 942/1410 [3:03:05<1:30:44, 11.63s/it]
 67%|██████▋   | 943/1410 [3:03:17<1:29:45, 11.53s/it]
                                                      
{'loss': 0.0094, 'grad_norm': 1.1929543018341064, 'learning_rate': 2.3522554354262688e-07, 'kl': 0.026, 'entropy': -0.0762, 'ce_loss': 0.0049, 'epoch': 1.34}

 67%|██████▋   | 943/1410 [3:03:17<1:29:45, 11.53s/it]
 67%|██████▋   | 944/1410 [3:03:29<1:30:19, 11.63s/it]
                                                      
{'loss': 0.0245, 'grad_norm': 1.1541603803634644, 'learning_rate': 2.3431732733087877e-07, 'kl': 0.0444, 'entropy': -0.1187, 'ce_loss': 0.0109, 'epoch': 1.34}

 67%|██████▋   | 944/1410 [3:03:29<1:30:19, 11.63s/it]
 67%|██████▋   | 945/1410 [3:03:40<1:30:03, 11.62s/it]
                                                      
{'loss': 0.0149, 'grad_norm': 1.007230281829834, 'learning_rate': 2.3341025026091312e-07, 'kl': 0.0176, 'entropy': -0.0479, 'ce_loss': 0.0079, 'epoch': 1.34}

 67%|██████▋   | 945/1410 [3:03:40<1:30:03, 11.62s/it]
 67%|██████▋   | 946/1410 [3:03:51<1:27:58, 11.38s/it]
                                                      
{'loss': 0.0147, 'grad_norm': 1.36088228225708, 'learning_rate': 2.3250431712351568e-07, 'kl': 0.032, 'entropy': -0.0981, 'ce_loss': 0.0234, 'epoch': 1.34}

 67%|██████▋   | 946/1410 [3:03:51<1:27:58, 11.38s/it]
 67%|██████▋   | 947/1410 [3:04:03<1:29:07, 11.55s/it]
                                                      
{'loss': 0.0137, 'grad_norm': 0.8425134420394897, 'learning_rate': 2.3159953270342942e-07, 'kl': 0.0294, 'entropy': -0.04, 'ce_loss': 0.0055, 'epoch': 1.34}

 67%|██████▋   | 947/1410 [3:04:03<1:29:07, 11.55s/it]
 67%|██████▋   | 948/1410 [3:04:15<1:29:42, 11.65s/it]
                                                      
{'loss': 0.0204, 'grad_norm': 1.3209272623062134, 'learning_rate': 2.3069590177933143e-07, 'kl': 0.0069, 'entropy': -0.0442, 'ce_loss': 0.0082, 'epoch': 1.34}

 67%|██████▋   | 948/1410 [3:04:15<1:29:42, 11.65s/it]
 67%|██████▋   | 949/1410 [3:04:27<1:30:29, 11.78s/it]
                                                      
{'loss': 0.0312, 'grad_norm': 1.3994122743606567, 'learning_rate': 2.297934291238059e-07, 'kl': 0.0254, 'entropy': -0.1001, 'ce_loss': 0.0123, 'epoch': 1.35}

 67%|██████▋   | 949/1410 [3:04:27<1:30:29, 11.78s/it]
 67%|██████▋   | 950/1410 [3:04:38<1:28:43, 11.57s/it]
                                                      
{'loss': 0.0243, 'grad_norm': 1.5569826364517212, 'learning_rate': 2.2889211950331944e-07, 'kl': 0.042, 'entropy': -0.1055, 'ce_loss': 0.0097, 'epoch': 1.35}

 67%|██████▋   | 950/1410 [3:04:38<1:28:43, 11.57s/it]
 67%|██████▋   | 951/1410 [3:04:50<1:28:53, 11.62s/it]
                                                      
{'loss': 0.012, 'grad_norm': 1.0956541299819946, 'learning_rate': 2.279919776781967e-07, 'kl': -0.007, 'entropy': -0.0593, 'ce_loss': 0.0411, 'epoch': 1.35}

 67%|██████▋   | 951/1410 [3:04:50<1:28:53, 11.62s/it]
 68%|██████▊   | 952/1410 [3:05:01<1:29:06, 11.67s/it]
                                                      
{'loss': 0.0039, 'grad_norm': 0.7861023545265198, 'learning_rate': 2.2709300840259384e-07, 'kl': -0.0042, 'entropy': 0.0294, 'ce_loss': 0.0969, 'epoch': 1.35}

 68%|██████▊   | 952/1410 [3:05:01<1:29:06, 11.67s/it]
 68%|██████▊   | 953/1410 [3:05:14<1:29:43, 11.78s/it]
                                                      
{'loss': 0.0288, 'grad_norm': 0.9053357839584351, 'learning_rate': 2.2619521642447416e-07, 'kl': -0.0078, 'entropy': 0.0442, 'ce_loss': 0.1325, 'epoch': 1.35}

 68%|██████▊   | 953/1410 [3:05:14<1:29:43, 11.78s/it]
 68%|██████▊   | 954/1410 [3:05:25<1:29:49, 11.82s/it]
                                                      
{'loss': 0.0065, 'grad_norm': 0.802997350692749, 'learning_rate': 2.252986064855837e-07, 'kl': 0.0134, 'entropy': -0.0503, 'ce_loss': 0.0061, 'epoch': 1.35}

 68%|██████▊   | 954/1410 [3:05:25<1:29:49, 11.82s/it]
 68%|██████▊   | 955/1410 [3:05:37<1:29:48, 11.84s/it]
                                                      
{'loss': 0.0536, 'grad_norm': 1.417814016342163, 'learning_rate': 2.2440318332142476e-07, 'kl': 0.0347, 'entropy': 0.0305, 'ce_loss': 0.1126, 'epoch': 1.35}

 68%|██████▊   | 955/1410 [3:05:37<1:29:48, 11.84s/it]
 68%|██████▊   | 956/1410 [3:05:48<1:27:34, 11.57s/it]
                                                      
{'loss': 0.0207, 'grad_norm': 1.4974465370178223, 'learning_rate': 2.2350895166123167e-07, 'kl': 0.0253, 'entropy': -0.0179, 'ce_loss': 0.004, 'epoch': 1.36}

 68%|██████▊   | 956/1410 [3:05:48<1:27:34, 11.57s/it]
 68%|██████▊   | 957/1410 [3:06:00<1:27:45, 11.62s/it]
                                                      
{'loss': 0.025, 'grad_norm': 1.6546672582626343, 'learning_rate': 2.2261591622794652e-07, 'kl': 0.0408, 'entropy': -0.1079, 'ce_loss': 0.0157, 'epoch': 1.36}

 68%|██████▊   | 957/1410 [3:06:00<1:27:45, 11.62s/it]
 68%|██████▊   | 958/1410 [3:06:12<1:28:02, 11.69s/it]
                                                      
{'loss': 0.0224, 'grad_norm': 1.3997825384140015, 'learning_rate': 2.2172408173819208e-07, 'kl': 0.0145, 'entropy': -0.0143, 'ce_loss': 0.0433, 'epoch': 1.36}

 68%|██████▊   | 958/1410 [3:06:12<1:28:02, 11.69s/it]
 68%|██████▊   | 959/1410 [3:06:23<1:26:02, 11.45s/it]
                                                      
{'loss': 0.0124, 'grad_norm': 1.207595944404602, 'learning_rate': 2.2083345290224946e-07, 'kl': 0.014, 'entropy': -0.0845, 'ce_loss': 0.0122, 'epoch': 1.36}

 68%|██████▊   | 959/1410 [3:06:23<1:26:02, 11.45s/it]
 68%|██████▊   | 960/1410 [3:06:34<1:25:27, 11.39s/it]
                                                      
{'loss': 0.0106, 'grad_norm': 1.078833818435669, 'learning_rate': 2.1994403442403196e-07, 'kl': 0.032, 'entropy': -0.0737, 'ce_loss': 0.0129, 'epoch': 1.36}

 68%|██████▊   | 960/1410 [3:06:34<1:25:27, 11.39s/it]
 68%|██████▊   | 961/1410 [3:06:46<1:26:30, 11.56s/it]
                                                      
{'loss': 0.0219, 'grad_norm': 1.2020466327667236, 'learning_rate': 2.190558310010593e-07, 'kl': 0.0128, 'entropy': -0.02, 'ce_loss': 0.0026, 'epoch': 1.36}

 68%|██████▊   | 961/1410 [3:06:46<1:26:30, 11.56s/it]
 68%|██████▊   | 962/1410 [3:06:58<1:26:52, 11.64s/it]
                                                      
{'loss': 0.0135, 'grad_norm': 1.2089991569519043, 'learning_rate': 2.1816884732443472e-07, 'kl': 0.0356, 'entropy': -0.0898, 'ce_loss': 0.0134, 'epoch': 1.36}

 68%|██████▊   | 962/1410 [3:06:58<1:26:52, 11.64s/it]
 68%|██████▊   | 963/1410 [3:07:09<1:25:13, 11.44s/it]
                                                      
{'loss': 0.0109, 'grad_norm': 1.022615671157837, 'learning_rate': 2.1728308807881944e-07, 'kl': 0.0068, 'entropy': -0.0265, 'ce_loss': 0.0007, 'epoch': 1.37}

 68%|██████▊   | 963/1410 [3:07:09<1:25:13, 11.44s/it]
 68%|██████▊   | 964/1410 [3:07:20<1:24:22, 11.35s/it]
                                                      
{'loss': 0.0081, 'grad_norm': 0.8859575390815735, 'learning_rate': 2.1639855794240667e-07, 'kl': 0.0284, 'entropy': -0.0535, 'ce_loss': 0.0054, 'epoch': 1.37}

 68%|██████▊   | 964/1410 [3:07:20<1:24:22, 11.35s/it]
 68%|██████▊   | 965/1410 [3:07:32<1:25:38, 11.55s/it]
                                                      
{'loss': 0.0523, 'grad_norm': 1.2981644868850708, 'learning_rate': 2.1551526158689928e-07, 'kl': -0.0027, 'entropy': -0.0562, 'ce_loss': 0.0314, 'epoch': 1.37}

 68%|██████▊   | 965/1410 [3:07:32<1:25:38, 11.55s/it]
 69%|██████▊   | 966/1410 [3:07:44<1:26:37, 11.71s/it]
                                                      
{'loss': 0.0094, 'grad_norm': 1.0979936122894287, 'learning_rate': 2.1463320367748306e-07, 'kl': 0.0317, 'entropy': -0.0894, 'ce_loss': 0.0163, 'epoch': 1.37}

 69%|██████▊   | 966/1410 [3:07:44<1:26:37, 11.71s/it]
 69%|██████▊   | 967/1410 [3:07:56<1:27:10, 11.81s/it]
                                                      
{'loss': 0.0104, 'grad_norm': 1.118309736251831, 'learning_rate': 2.137523888728029e-07, 'kl': 0.0244, 'entropy': -0.103, 'ce_loss': 0.0036, 'epoch': 1.37}

 69%|██████▊   | 967/1410 [3:07:56<1:27:10, 11.81s/it]
 69%|██████▊   | 968/1410 [3:08:08<1:27:01, 11.81s/it]
                                                      
{'loss': 0.0128, 'grad_norm': 1.4292348623275757, 'learning_rate': 2.1287282182493864e-07, 'kl': 0.0151, 'entropy': -0.0164, 'ce_loss': 0.068, 'epoch': 1.37}

 69%|██████▊   | 968/1410 [3:08:08<1:27:01, 11.81s/it]
 69%|██████▊   | 969/1410 [3:08:19<1:26:25, 11.76s/it]
                                                      
{'loss': 0.0194, 'grad_norm': 1.0908300876617432, 'learning_rate': 2.1199450717937955e-07, 'kl': 0.0099, 'entropy': -0.0679, 'ce_loss': 0.011, 'epoch': 1.37}

 69%|██████▊   | 969/1410 [3:08:19<1:26:25, 11.76s/it]
 69%|██████▉   | 970/1410 [3:08:32<1:27:28, 11.93s/it]
                                                      
{'loss': 0.0078, 'grad_norm': 0.9628505706787109, 'learning_rate': 2.111174495750003e-07, 'kl': -0.0068, 'entropy': -0.0183, 'ce_loss': 0.0609, 'epoch': 1.38}

 69%|██████▉   | 970/1410 [3:08:32<1:27:28, 11.93s/it]
 69%|██████▉   | 971/1410 [3:08:43<1:26:20, 11.80s/it]
                                                      
{'loss': 0.0132, 'grad_norm': 1.2851994037628174, 'learning_rate': 2.102416536440368e-07, 'kl': 0.0006, 'entropy': -0.0195, 'ce_loss': 0.0533, 'epoch': 1.38}

 69%|██████▉   | 971/1410 [3:08:43<1:26:20, 11.80s/it]
 69%|██████▉   | 972/1410 [3:08:55<1:25:46, 11.75s/it]
                                                      
{'loss': 0.0105, 'grad_norm': 1.0602126121520996, 'learning_rate': 2.0936712401206095e-07, 'kl': 0.0265, 'entropy': -0.0854, 'ce_loss': 0.0248, 'epoch': 1.38}

 69%|██████▉   | 972/1410 [3:08:55<1:25:46, 11.75s/it]
 69%|██████▉   | 973/1410 [3:09:07<1:25:29, 11.74s/it]
                                                      
{'loss': 0.0107, 'grad_norm': 1.2371233701705933, 'learning_rate': 2.0849386529795676e-07, 'kl': 0.0154, 'entropy': -0.0171, 'ce_loss': 0.0014, 'epoch': 1.38}

 69%|██████▉   | 973/1410 [3:09:07<1:25:29, 11.74s/it]
 69%|██████▉   | 974/1410 [3:09:19<1:25:50, 11.81s/it]
                                                      
{'loss': 0.0118, 'grad_norm': 1.2195014953613281, 'learning_rate': 2.0762188211389568e-07, 'kl': 0.0265, 'entropy': -0.1025, 'ce_loss': 0.0064, 'epoch': 1.38}

 69%|██████▉   | 974/1410 [3:09:19<1:25:50, 11.81s/it]
 69%|██████▉   | 975/1410 [3:09:31<1:25:57, 11.86s/it]
                                                      
{'loss': 0.0184, 'grad_norm': 0.8081316351890564, 'learning_rate': 2.0675117906531242e-07, 'kl': 0.0056, 'entropy': -0.0337, 'ce_loss': 0.0016, 'epoch': 1.38}

 69%|██████▉   | 975/1410 [3:09:31<1:25:57, 11.86s/it]
 69%|██████▉   | 976/1410 [3:09:43<1:26:00, 11.89s/it]
                                                      
{'loss': 0.0163, 'grad_norm': 0.7765870690345764, 'learning_rate': 2.0588176075088094e-07, 'kl': 0.0231, 'entropy': -0.0591, 'ce_loss': 0.0174, 'epoch': 1.38}

 69%|██████▉   | 976/1410 [3:09:43<1:26:00, 11.89s/it]
 69%|██████▉   | 977/1410 [3:09:54<1:25:30, 11.85s/it]
                                                      
{'loss': 0.0093, 'grad_norm': 1.0297391414642334, 'learning_rate': 2.050136317624892e-07, 'kl': 0.0142, 'entropy': -0.0459, 'ce_loss': 0.003, 'epoch': 1.39}

 69%|██████▉   | 977/1410 [3:09:54<1:25:30, 11.85s/it]
 69%|██████▉   | 978/1410 [3:10:06<1:24:57, 11.80s/it]
                                                      
{'loss': 0.0091, 'grad_norm': 1.0682178735733032, 'learning_rate': 2.041467966852158e-07, 'kl': -0.0081, 'entropy': 0.0002, 'ce_loss': 0.0982, 'epoch': 1.39}

 69%|██████▉   | 978/1410 [3:10:06<1:24:57, 11.80s/it]
 69%|██████▉   | 979/1410 [3:10:18<1:25:37, 11.92s/it]
                                                      
{'loss': 0.0052, 'grad_norm': 0.8357592821121216, 'learning_rate': 2.0328126009730568e-07, 'kl': 0.0038, 'entropy': -0.0221, 'ce_loss': 0.0026, 'epoch': 1.39}

 69%|██████▉   | 979/1410 [3:10:18<1:25:37, 11.92s/it]
 70%|██████▉   | 980/1410 [3:10:30<1:25:24, 11.92s/it]
                                                      
{'loss': 0.0242, 'grad_norm': 1.393385887145996, 'learning_rate': 2.024170265701453e-07, 'kl': 0.0396, 'entropy': -0.0933, 'ce_loss': 0.0345, 'epoch': 1.39}

 70%|██████▉   | 980/1410 [3:10:30<1:25:24, 11.92s/it]
 70%|██████▉   | 981/1410 [3:10:42<1:24:43, 11.85s/it]
                                                      
{'loss': 0.0209, 'grad_norm': 1.206201195716858, 'learning_rate': 2.0155410066823922e-07, 'kl': 0.0011, 'entropy': -0.0767, 'ce_loss': 0.0209, 'epoch': 1.39}

 70%|██████▉   | 981/1410 [3:10:42<1:24:43, 11.85s/it]
 70%|██████▉   | 982/1410 [3:10:54<1:25:14, 11.95s/it]
                                                      
{'loss': 0.0178, 'grad_norm': 1.1242456436157227, 'learning_rate': 2.0069248694918533e-07, 'kl': 0.0306, 'entropy': -0.0557, 'ce_loss': 0.0051, 'epoch': 1.39}

 70%|██████▉   | 982/1410 [3:10:54<1:25:14, 11.95s/it]
 70%|██████▉   | 983/1410 [3:11:06<1:24:17, 11.84s/it]
                                                      
{'loss': 0.0094, 'grad_norm': 1.0357255935668945, 'learning_rate': 1.998321899636518e-07, 'kl': 0.028, 'entropy': -0.0195, 'ce_loss': 0.0019, 'epoch': 1.39}

 70%|██████▉   | 983/1410 [3:11:06<1:24:17, 11.84s/it]
 70%|██████▉   | 984/1410 [3:11:17<1:23:44, 11.79s/it]
                                                      
{'loss': 0.015, 'grad_norm': 1.195918083190918, 'learning_rate': 1.9897321425535177e-07, 'kl': 0.0266, 'entropy': -0.0322, 'ce_loss': 0.0009, 'epoch': 1.4}

 70%|██████▉   | 984/1410 [3:11:17<1:23:44, 11.79s/it]
 70%|██████▉   | 985/1410 [3:11:29<1:23:22, 11.77s/it]
                                                      
{'loss': 0.0061, 'grad_norm': 0.9812929034233093, 'learning_rate': 1.9811556436102003e-07, 'kl': -0.0061, 'entropy': -0.0579, 'ce_loss': 0.0363, 'epoch': 1.4}

 70%|██████▉   | 985/1410 [3:11:29<1:23:22, 11.77s/it]
 70%|██████▉   | 986/1410 [3:11:41<1:23:01, 11.75s/it]
                                                      
{'loss': 0.0094, 'grad_norm': 1.0034912824630737, 'learning_rate': 1.9725924481038953e-07, 'kl': 0.0112, 'entropy': -0.0322, 'ce_loss': 0.0028, 'epoch': 1.4}

 70%|██████▉   | 986/1410 [3:11:41<1:23:01, 11.75s/it]
 70%|███████   | 987/1410 [3:11:53<1:23:07, 11.79s/it]
                                                      
{'loss': 0.0184, 'grad_norm': 1.4132965803146362, 'learning_rate': 1.9640426012616636e-07, 'kl': 0.0171, 'entropy': -0.0272, 'ce_loss': 0.0549, 'epoch': 1.4}

 70%|███████   | 987/1410 [3:11:53<1:23:07, 11.79s/it]
 70%|███████   | 988/1410 [3:12:04<1:22:41, 11.76s/it]
                                                      
{'loss': 0.0068, 'grad_norm': 0.9414806962013245, 'learning_rate': 1.9555061482400665e-07, 'kl': -0.0024, 'entropy': 0.0364, 'ce_loss': 0.0826, 'epoch': 1.4}

 70%|███████   | 988/1410 [3:12:04<1:22:41, 11.76s/it]
 70%|███████   | 989/1410 [3:12:16<1:22:55, 11.82s/it]
                                                      
{'loss': 0.0199, 'grad_norm': 1.333947777748108, 'learning_rate': 1.9469831341249227e-07, 'kl': 0.0295, 'entropy': -0.1118, 'ce_loss': 0.0074, 'epoch': 1.4}

 70%|███████   | 989/1410 [3:12:16<1:22:55, 11.82s/it]
 70%|███████   | 990/1410 [3:12:28<1:22:52, 11.84s/it]
                                                      
{'loss': 0.0103, 'grad_norm': 0.9294953942298889, 'learning_rate': 1.9384736039310784e-07, 'kl': -0.0057, 'entropy': -0.0195, 'ce_loss': 0.0658, 'epoch': 1.4}

 70%|███████   | 990/1410 [3:12:28<1:22:52, 11.84s/it]
 70%|███████   | 991/1410 [3:12:40<1:22:20, 11.79s/it]
                                                      
{'loss': 0.0094, 'grad_norm': 1.1321011781692505, 'learning_rate': 1.9299776026021574e-07, 'kl': 0.0024, 'entropy': -0.0106, 'ce_loss': 0.0007, 'epoch': 1.41}

 70%|███████   | 991/1410 [3:12:40<1:22:20, 11.79s/it]
 70%|███████   | 992/1410 [3:12:51<1:21:45, 11.74s/it]
                                                      
{'loss': 0.009, 'grad_norm': 1.3370262384414673, 'learning_rate': 1.9214951750103328e-07, 'kl': 0.0172, 'entropy': -0.0245, 'ce_loss': 0.0025, 'epoch': 1.41}

 70%|███████   | 992/1410 [3:12:51<1:21:45, 11.74s/it]
 70%|███████   | 993/1410 [3:13:03<1:21:50, 11.77s/it]
                                                      
{'loss': 0.0225, 'grad_norm': 1.294894814491272, 'learning_rate': 1.9130263659560843e-07, 'kl': -0.0021, 'entropy': 0.1289, 'ce_loss': 0.1958, 'epoch': 1.41}

 70%|███████   | 993/1410 [3:13:03<1:21:50, 11.77s/it]
 70%|███████   | 994/1410 [3:13:15<1:22:11, 11.85s/it]
                                                      
{'loss': 0.0217, 'grad_norm': 1.4957234859466553, 'learning_rate': 1.9045712201679697e-07, 'kl': 0.0116, 'entropy': -0.0054, 'ce_loss': 0.0004, 'epoch': 1.41}

 70%|███████   | 994/1410 [3:13:15<1:22:11, 11.85s/it]
 71%|███████   | 995/1410 [3:13:27<1:21:02, 11.72s/it]
                                                      
{'loss': 0.0203, 'grad_norm': 1.535280466079712, 'learning_rate': 1.8961297823023803e-07, 'kl': 0.0083, 'entropy': -0.0452, 'ce_loss': 0.0088, 'epoch': 1.41}

 71%|███████   | 995/1410 [3:13:27<1:21:02, 11.72s/it]
 71%|███████   | 996/1410 [3:13:38<1:20:13, 11.63s/it]
                                                      
{'loss': 0.0095, 'grad_norm': 1.1941008567810059, 'learning_rate': 1.887702096943305e-07, 'kl': 0.0332, 'entropy': -0.0952, 'ce_loss': 0.0132, 'epoch': 1.41}

 71%|███████   | 996/1410 [3:13:38<1:20:13, 11.63s/it]
 71%|███████   | 997/1410 [3:13:50<1:20:28, 11.69s/it]
                                                      
{'loss': 0.0077, 'grad_norm': 1.0684723854064941, 'learning_rate': 1.8792882086021053e-07, 'kl': 0.0008, 'entropy': -0.0233, 'ce_loss': 0.0015, 'epoch': 1.41}

 71%|███████   | 997/1410 [3:13:50<1:20:28, 11.69s/it]
 71%|███████   | 998/1410 [3:14:02<1:21:31, 11.87s/it]
                                                      
{'loss': 0.0149, 'grad_norm': 1.2087382078170776, 'learning_rate': 1.8708881617172668e-07, 'kl': 0.04, 'entropy': -0.0854, 'ce_loss': 0.0129, 'epoch': 1.42}

 71%|███████   | 998/1410 [3:14:02<1:21:31, 11.87s/it]
 71%|███████   | 999/1410 [3:14:14<1:21:45, 11.94s/it]
                                                      
{'loss': 0.0111, 'grad_norm': 1.1363303661346436, 'learning_rate': 1.8625020006541707e-07, 'kl': -0.01, 'entropy': 0.1553, 'ce_loss': 0.145, 'epoch': 1.42}

 71%|███████   | 999/1410 [3:14:14<1:21:45, 11.94s/it]
 71%|███████   | 1000/1410 [3:14:27<1:22:16, 12.04s/it]
                                                       
{'loss': 0.0144, 'grad_norm': 1.1516889333724976, 'learning_rate': 1.8541297697048664e-07, 'kl': 0.0339, 'entropy': -0.0884, 'ce_loss': 0.0149, 'epoch': 1.42}

 71%|███████   | 1000/1410 [3:14:27<1:22:16, 12.04s/it]
 71%|███████   | 1001/1410 [3:14:38<1:21:17, 11.92s/it]
                                                       
{'loss': 0.0067, 'grad_norm': 0.9524634480476379, 'learning_rate': 1.8457715130878185e-07, 'kl': 0.0396, 'entropy': -0.0728, 'ce_loss': 0.0077, 'epoch': 1.42}

 71%|███████   | 1001/1410 [3:14:38<1:21:17, 11.92s/it]
 71%|███████   | 1002/1410 [3:14:50<1:20:57, 11.91s/it]
                                                       
{'loss': 0.0069, 'grad_norm': 1.0611844062805176, 'learning_rate': 1.8374272749476957e-07, 'kl': 0.0288, 'entropy': -0.1113, 'ce_loss': 0.0079, 'epoch': 1.42}

 71%|███████   | 1002/1410 [3:14:50<1:20:57, 11.91s/it]
 71%|███████   | 1003/1410 [3:15:02<1:20:43, 11.90s/it]
                                                       
{'loss': 0.0122, 'grad_norm': 0.8661516904830933, 'learning_rate': 1.829097099355126e-07, 'kl': 0.0244, 'entropy': -0.1035, 'ce_loss': 0.0081, 'epoch': 1.42}

 71%|███████   | 1003/1410 [3:15:02<1:20:43, 11.90s/it]
 71%|███████   | 1004/1410 [3:15:14<1:20:24, 11.88s/it]
                                                       
{'loss': 0.0091, 'grad_norm': 0.8509209156036377, 'learning_rate': 1.8207810303064573e-07, 'kl': 0.0247, 'entropy': -0.0957, 'ce_loss': 0.0101, 'epoch': 1.42}

 71%|███████   | 1004/1410 [3:15:14<1:20:24, 11.88s/it]
 71%|███████▏  | 1005/1410 [3:15:26<1:21:02, 12.01s/it]
                                                       
{'loss': 0.0124, 'grad_norm': 1.094718337059021, 'learning_rate': 1.8124791117235424e-07, 'kl': 0.0287, 'entropy': -0.0437, 'ce_loss': 0.0117, 'epoch': 1.43}

 71%|███████▏  | 1005/1410 [3:15:26<1:21:02, 12.01s/it]
 71%|███████▏  | 1006/1410 [3:15:38<1:20:33, 11.96s/it]
                                                       
{'loss': 0.0171, 'grad_norm': 1.1581073999404907, 'learning_rate': 1.8041913874534926e-07, 'kl': 0.0233, 'entropy': -0.0618, 'ce_loss': 0.0148, 'epoch': 1.43}

 71%|███████▏  | 1006/1410 [3:15:38<1:20:33, 11.96s/it]
 71%|███████▏  | 1007/1410 [3:15:50<1:20:03, 11.92s/it]
                                                       
{'loss': 0.0123, 'grad_norm': 1.1571803092956543, 'learning_rate': 1.7959179012684492e-07, 'kl': -0.0077, 'entropy': -0.0352, 'ce_loss': 0.1045, 'epoch': 1.43}

 71%|███████▏  | 1007/1410 [3:15:50<1:20:03, 11.92s/it]
 71%|███████▏  | 1008/1410 [3:16:03<1:23:14, 12.42s/it]
                                                       
{'loss': 0.0183, 'grad_norm': 1.0276461839675903, 'learning_rate': 1.787658696865361e-07, 'kl': -0.0071, 'entropy': 0.0596, 'ce_loss': 0.1216, 'epoch': 1.43}

 71%|███████▏  | 1008/1410 [3:16:03<1:23:14, 12.42s/it]
 72%|███████▏  | 1009/1410 [3:16:15<1:22:11, 12.30s/it]
                                                       
{'loss': 0.0335, 'grad_norm': 1.1488476991653442, 'learning_rate': 1.77941381786574e-07, 'kl': -0.0003, 'entropy': -0.0352, 'ce_loss': 0.0009, 'epoch': 1.43}

 72%|███████▏  | 1009/1410 [3:16:15<1:22:11, 12.30s/it]
 72%|███████▏  | 1010/1410 [3:16:27<1:21:01, 12.15s/it]
                                                       
{'loss': 0.0088, 'grad_norm': 0.9915327429771423, 'learning_rate': 1.7711833078154392e-07, 'kl': 0.0264, 'entropy': -0.0654, 'ce_loss': 0.0232, 'epoch': 1.43}

 72%|███████▏  | 1010/1410 [3:16:27<1:21:01, 12.15s/it]
 72%|███████▏  | 1011/1410 [3:16:39<1:20:33, 12.11s/it]
                                                       
{'loss': 0.0216, 'grad_norm': 1.2113556861877441, 'learning_rate': 1.762967210184426e-07, 'kl': 0.0308, 'entropy': -0.1025, 'ce_loss': 0.0102, 'epoch': 1.43}

 72%|███████▏  | 1011/1410 [3:16:39<1:20:33, 12.11s/it]
 72%|███████▏  | 1012/1410 [3:16:51<1:19:33, 11.99s/it]
                                                       
{'loss': 0.0239, 'grad_norm': 1.2905267477035522, 'learning_rate': 1.7547655683665384e-07, 'kl': 0.0221, 'entropy': -0.0403, 'ce_loss': 0.0081, 'epoch': 1.44}

 72%|███████▏  | 1012/1410 [3:16:51<1:19:33, 11.99s/it]
 72%|███████▏  | 1013/1410 [3:17:03<1:18:58, 11.94s/it]
                                                       
{'loss': 0.0281, 'grad_norm': 1.3852170705795288, 'learning_rate': 1.746578425679273e-07, 'kl': 0.0322, 'entropy': 0.0057, 'ce_loss': 0.0887, 'epoch': 1.44}

 72%|███████▏  | 1013/1410 [3:17:03<1:18:58, 11.94s/it]
 72%|███████▏  | 1014/1410 [3:17:14<1:17:44, 11.78s/it]
                                                       
{'loss': 0.0133, 'grad_norm': 1.1897271871566772, 'learning_rate': 1.7384058253635491e-07, 'kl': 0.0298, 'entropy': -0.0898, 'ce_loss': 0.0086, 'epoch': 1.44}

 72%|███████▏  | 1014/1410 [3:17:14<1:17:44, 11.78s/it]
 72%|███████▏  | 1015/1410 [3:17:26<1:18:15, 11.89s/it]
                                                       
{'loss': 0.0222, 'grad_norm': 1.1276379823684692, 'learning_rate': 1.7302478105834697e-07, 'kl': 0.0297, 'entropy': -0.062, 'ce_loss': 0.0027, 'epoch': 1.44}

 72%|███████▏  | 1015/1410 [3:17:26<1:18:15, 11.89s/it]
 72%|███████▏  | 1016/1410 [3:17:38<1:18:15, 11.92s/it]
                                                       
{'loss': 0.0149, 'grad_norm': 0.9429448246955872, 'learning_rate': 1.7221044244261126e-07, 'kl': -0.0103, 'entropy': -0.0117, 'ce_loss': 0.0763, 'epoch': 1.44}

 72%|███████▏  | 1016/1410 [3:17:38<1:18:15, 11.92s/it]
 72%|███████▏  | 1017/1410 [3:17:51<1:18:45, 12.02s/it]
                                                       
{'loss': 0.0094, 'grad_norm': 1.0324366092681885, 'learning_rate': 1.713975709901295e-07, 'kl': 0.0195, 'entropy': -0.0498, 'ce_loss': 0.0209, 'epoch': 1.44}

 72%|███████▏  | 1017/1410 [3:17:51<1:18:45, 12.02s/it]
 72%|███████▏  | 1018/1410 [3:18:03<1:18:39, 12.04s/it]
                                                       
{'loss': 0.028, 'grad_norm': 1.477913498878479, 'learning_rate': 1.7058617099413335e-07, 'kl': -0.0104, 'entropy': -0.0135, 'ce_loss': 0.0851, 'epoch': 1.44}

 72%|███████▏  | 1018/1410 [3:18:03<1:18:39, 12.04s/it]
 72%|███████▏  | 1019/1410 [3:18:15<1:18:05, 11.98s/it]
                                                       
{'loss': 0.0108, 'grad_norm': 1.2025960683822632, 'learning_rate': 1.697762467400841e-07, 'kl': 0.0298, 'entropy': -0.0708, 'ce_loss': 0.0191, 'epoch': 1.45}

 72%|███████▏  | 1019/1410 [3:18:15<1:18:05, 11.98s/it]
 72%|███████▏  | 1020/1410 [3:18:27<1:18:08, 12.02s/it]
                                                       
{'loss': 0.0263, 'grad_norm': 1.4007792472839355, 'learning_rate': 1.6896780250564815e-07, 'kl': 0.002, 'entropy': -0.0417, 'ce_loss': 0.031, 'epoch': 1.45}

 72%|███████▏  | 1020/1410 [3:18:27<1:18:08, 12.02s/it]
 72%|███████▏  | 1021/1410 [3:18:39<1:18:28, 12.10s/it]
                                                       
{'loss': 0.0377, 'grad_norm': 1.2210721969604492, 'learning_rate': 1.681608425606751e-07, 'kl': 0.0356, 'entropy': 0.0, 'ce_loss': 0.0504, 'epoch': 1.45}

 72%|███████▏  | 1021/1410 [3:18:39<1:18:28, 12.10s/it]
 72%|███████▏  | 1022/1410 [3:18:50<1:17:16, 11.95s/it]
                                                       
{'loss': 0.0194, 'grad_norm': 1.1881930828094482, 'learning_rate': 1.6735537116717574e-07, 'kl': 0.0203, 'entropy': -0.0583, 'ce_loss': 0.0067, 'epoch': 1.45}

 72%|███████▏  | 1022/1410 [3:18:50<1:17:16, 11.95s/it]
 73%|███████▎  | 1023/1410 [3:19:02<1:16:42, 11.89s/it]
                                                       
{'loss': 0.0102, 'grad_norm': 0.890213668346405, 'learning_rate': 1.6655139257929829e-07, 'kl': 0.0544, 'entropy': -0.0898, 'ce_loss': 0.0036, 'epoch': 1.45}

 73%|███████▎  | 1023/1410 [3:19:02<1:16:42, 11.89s/it]
 73%|███████▎  | 1024/1410 [3:19:13<1:15:11, 11.69s/it]
                                                       
{'loss': 0.0126, 'grad_norm': 1.2209450006484985, 'learning_rate': 1.6574891104330681e-07, 'kl': 0.0327, 'entropy': -0.104, 'ce_loss': 0.014, 'epoch': 1.45}

 73%|███████▎  | 1024/1410 [3:19:13<1:15:11, 11.69s/it]
 73%|███████▎  | 1025/1410 [3:19:25<1:15:26, 11.76s/it]
                                                       
{'loss': 0.0197, 'grad_norm': 1.5012445449829102, 'learning_rate': 1.6494793079755893e-07, 'kl': 0.0461, 'entropy': -0.127, 'ce_loss': 0.0126, 'epoch': 1.45}

 73%|███████▎  | 1025/1410 [3:19:25<1:15:26, 11.76s/it]
 73%|███████▎  | 1026/1410 [3:19:36<1:13:52, 11.54s/it]
                                                       
{'loss': 0.0172, 'grad_norm': 1.1041290760040283, 'learning_rate': 1.641484560724828e-07, 'kl': 0.0493, 'entropy': -0.1387, 'ce_loss': 0.006, 'epoch': 1.46}

 73%|███████▎  | 1026/1410 [3:19:36<1:13:52, 11.54s/it]
 73%|███████▎  | 1027/1410 [3:19:49<1:14:42, 11.70s/it]
                                                       
{'loss': 0.0074, 'grad_norm': 0.9559323787689209, 'learning_rate': 1.633504910905551e-07, 'kl': -0.001, 'entropy': 0.0593, 'ce_loss': 0.0695, 'epoch': 1.46}

 73%|███████▎  | 1027/1410 [3:19:49<1:14:42, 11.70s/it]
 73%|███████▎  | 1028/1410 [3:20:00<1:14:25, 11.69s/it]
                                                       
{'loss': 0.0113, 'grad_norm': 0.9931937456130981, 'learning_rate': 1.6255404006627871e-07, 'kl': 0.022, 'entropy': -0.0923, 'ce_loss': 0.0241, 'epoch': 1.46}

 73%|███████▎  | 1028/1410 [3:20:00<1:14:25, 11.69s/it]
 73%|███████▎  | 1029/1410 [3:20:12<1:14:39, 11.76s/it]
                                                       
{'loss': 0.0135, 'grad_norm': 0.9812417030334473, 'learning_rate': 1.617591072061604e-07, 'kl': 0.0291, 'entropy': -0.05, 'ce_loss': 0.0136, 'epoch': 1.46}

 73%|███████▎  | 1029/1410 [3:20:12<1:14:39, 11.76s/it]
 73%|███████▎  | 1030/1410 [3:20:24<1:14:12, 11.72s/it]
                                                       
{'loss': 0.0142, 'grad_norm': 1.314400553703308, 'learning_rate': 1.6096569670868898e-07, 'kl': 0.0277, 'entropy': -0.0549, 'ce_loss': 0.0292, 'epoch': 1.46}

 73%|███████▎  | 1030/1410 [3:20:24<1:14:12, 11.72s/it]
 73%|███████▎  | 1031/1410 [3:20:36<1:14:15, 11.76s/it]
                                                       
{'loss': 0.0057, 'grad_norm': 0.8446047902107239, 'learning_rate': 1.6017381276431242e-07, 'kl': -0.0031, 'entropy': 0.0129, 'ce_loss': 0.0882, 'epoch': 1.46}

 73%|███████▎  | 1031/1410 [3:20:36<1:14:15, 11.76s/it]
 73%|███████▎  | 1032/1410 [3:20:48<1:14:33, 11.84s/it]
                                                       
{'loss': 0.0276, 'grad_norm': 1.072068452835083, 'learning_rate': 1.593834595554161e-07, 'kl': 0.0015, 'entropy': 0.0674, 'ce_loss': 0.1188, 'epoch': 1.46}

 73%|███████▎  | 1032/1410 [3:20:48<1:14:33, 11.84s/it]
 73%|███████▎  | 1033/1410 [3:20:59<1:14:09, 11.80s/it]
                                                       
{'loss': 0.0165, 'grad_norm': 1.9774788618087769, 'learning_rate': 1.5859464125630118e-07, 'kl': -0.0027, 'entropy': -0.083, 'ce_loss': 0.0216, 'epoch': 1.47}

 73%|███████▎  | 1033/1410 [3:20:59<1:14:09, 11.80s/it]
 73%|███████▎  | 1034/1410 [3:21:11<1:14:07, 11.83s/it]
                                                       
{'loss': 0.0116, 'grad_norm': 1.0881437063217163, 'learning_rate': 1.5780736203316155e-07, 'kl': -0.0031, 'entropy': 0.0258, 'ce_loss': 0.0868, 'epoch': 1.47}

 73%|███████▎  | 1034/1410 [3:21:11<1:14:07, 11.83s/it]
 73%|███████▎  | 1035/1410 [3:21:23<1:13:48, 11.81s/it]
                                                       
{'loss': 0.0091, 'grad_norm': 0.988412082195282, 'learning_rate': 1.5702162604406258e-07, 'kl': -0.0033, 'entropy': -0.0376, 'ce_loss': 0.0691, 'epoch': 1.47}

 73%|███████▎  | 1035/1410 [3:21:23<1:13:48, 11.81s/it]
 73%|███████▎  | 1036/1410 [3:21:34<1:11:57, 11.55s/it]
                                                       
{'loss': 0.0091, 'grad_norm': 1.0178343057632446, 'learning_rate': 1.5623743743891923e-07, 'kl': -0.0056, 'entropy': -0.0294, 'ce_loss': 0.0316, 'epoch': 1.47}

 73%|███████▎  | 1036/1410 [3:21:34<1:11:57, 11.55s/it]
 74%|███████▎  | 1037/1410 [3:21:46<1:12:07, 11.60s/it]
                                                       
{'loss': 0.0101, 'grad_norm': 1.0055782794952393, 'learning_rate': 1.5545480035947342e-07, 'kl': 0.0361, 'entropy': -0.0933, 'ce_loss': 0.0081, 'epoch': 1.47}

 74%|███████▎  | 1037/1410 [3:21:46<1:12:07, 11.60s/it]
 74%|███████▎  | 1038/1410 [3:21:57<1:10:57, 11.45s/it]
                                                       
{'loss': 0.0103, 'grad_norm': 1.0465584993362427, 'learning_rate': 1.546737189392728e-07, 'kl': 0.0439, 'entropy': -0.0786, 'ce_loss': 0.0075, 'epoch': 1.47}

 74%|███████▎  | 1038/1410 [3:21:57<1:10:57, 11.45s/it]
 74%|███████▎  | 1039/1410 [3:22:09<1:11:31, 11.57s/it]
                                                       
{'loss': 0.0135, 'grad_norm': 0.9774352312088013, 'learning_rate': 1.5389419730364851e-07, 'kl': -0.0103, 'entropy': -0.0337, 'ce_loss': 0.0634, 'epoch': 1.47}

 74%|███████▎  | 1039/1410 [3:22:09<1:11:31, 11.57s/it]
 74%|███████▍  | 1040/1410 [3:22:19<1:09:28, 11.27s/it]
                                                       
{'loss': 0.0144, 'grad_norm': 1.6057168245315552, 'learning_rate': 1.5311623956969408e-07, 'kl': -0.0045, 'entropy': -0.0302, 'ce_loss': 0.0057, 'epoch': 1.48}

 74%|███████▍  | 1040/1410 [3:22:19<1:09:28, 11.27s/it]
 74%|███████▍  | 1041/1410 [3:22:31<1:10:33, 11.47s/it]
                                                       
{'loss': 0.0154, 'grad_norm': 1.0184088945388794, 'learning_rate': 1.5233984984624262e-07, 'kl': 0.0322, 'entropy': -0.0938, 'ce_loss': 0.0044, 'epoch': 1.48}

 74%|███████▍  | 1041/1410 [3:22:31<1:10:33, 11.47s/it]
 74%|███████▍  | 1042/1410 [3:22:43<1:11:42, 11.69s/it]
                                                       
{'loss': 0.0181, 'grad_norm': 0.9576972723007202, 'learning_rate': 1.5156503223384605e-07, 'kl': 0.0457, 'entropy': -0.053, 'ce_loss': 0.0664, 'epoch': 1.48}

 74%|███████▍  | 1042/1410 [3:22:43<1:11:42, 11.69s/it]
 74%|███████▍  | 1043/1410 [3:22:55<1:10:45, 11.57s/it]
                                                       
{'loss': 0.011, 'grad_norm': 0.9257550835609436, 'learning_rate': 1.5079179082475283e-07, 'kl': 0.0221, 'entropy': -0.0581, 'ce_loss': 0.0034, 'epoch': 1.48}

 74%|███████▍  | 1043/1410 [3:22:55<1:10:45, 11.57s/it]
 74%|███████▍  | 1044/1410 [3:23:07<1:12:24, 11.87s/it]
                                                       
{'loss': 0.0067, 'grad_norm': 0.8246296048164368, 'learning_rate': 1.5002012970288693e-07, 'kl': 0.0223, 'entropy': -0.0476, 'ce_loss': 0.0087, 'epoch': 1.48}

 74%|███████▍  | 1044/1410 [3:23:07<1:12:24, 11.87s/it]
 74%|███████▍  | 1045/1410 [3:23:19<1:12:12, 11.87s/it]
                                                       
{'loss': 0.0103, 'grad_norm': 0.8756478428840637, 'learning_rate': 1.4925005294382572e-07, 'kl': 0.0017, 'entropy': -0.0259, 'ce_loss': 0.0009, 'epoch': 1.48}

 74%|███████▍  | 1045/1410 [3:23:19<1:12:12, 11.87s/it]
 74%|███████▍  | 1046/1410 [3:23:31<1:11:37, 11.81s/it]
                                                       
{'loss': 0.0099, 'grad_norm': 1.0544843673706055, 'learning_rate': 1.4848156461477858e-07, 'kl': 0.0393, 'entropy': -0.1162, 'ce_loss': 0.0087, 'epoch': 1.48}

 74%|███████▍  | 1046/1410 [3:23:31<1:11:37, 11.81s/it]
 74%|███████▍  | 1047/1410 [3:23:42<1:10:04, 11.58s/it]
                                                       
{'loss': 0.0087, 'grad_norm': 1.1602835655212402, 'learning_rate': 1.4771466877456556e-07, 'kl': -0.0081, 'entropy': -0.0162, 'ce_loss': 0.0683, 'epoch': 1.49}

 74%|███████▍  | 1047/1410 [3:23:42<1:10:04, 11.58s/it]
 74%|███████▍  | 1048/1410 [3:23:54<1:10:31, 11.69s/it]
                                                       
{'loss': 0.0122, 'grad_norm': 0.7747344374656677, 'learning_rate': 1.4694936947359612e-07, 'kl': 0.0076, 'entropy': -0.0208, 'ce_loss': 0.0022, 'epoch': 1.49}

 74%|███████▍  | 1048/1410 [3:23:54<1:10:31, 11.69s/it]
 74%|███████▍  | 1049/1410 [3:24:06<1:11:08, 11.82s/it]
                                                       
{'loss': 0.0113, 'grad_norm': 0.8143588304519653, 'learning_rate': 1.461856707538472e-07, 'kl': 0.0304, 'entropy': -0.0732, 'ce_loss': 0.0166, 'epoch': 1.49}

 74%|███████▍  | 1049/1410 [3:24:06<1:11:08, 11.82s/it]
 74%|███████▍  | 1050/1410 [3:24:18<1:11:17, 11.88s/it]
                                                       
{'loss': 0.0062, 'grad_norm': 0.6158975958824158, 'learning_rate': 1.4542357664884213e-07, 'kl': 0.0019, 'entropy': -0.0184, 'ce_loss': 0.001, 'epoch': 1.49}

 74%|███████▍  | 1050/1410 [3:24:18<1:11:17, 11.88s/it]
 75%|███████▍  | 1051/1410 [3:24:30<1:11:07, 11.89s/it]
                                                       
{'loss': 0.0114, 'grad_norm': 1.1791850328445435, 'learning_rate': 1.4466309118362972e-07, 'kl': -0.0076, 'entropy': -0.0645, 'ce_loss': 0.0315, 'epoch': 1.49}

 75%|███████▍  | 1051/1410 [3:24:30<1:11:07, 11.89s/it]
 75%|███████▍  | 1052/1410 [3:24:42<1:11:01, 11.90s/it]
                                                       
{'loss': 0.0099, 'grad_norm': 0.9702492952346802, 'learning_rate': 1.4390421837476236e-07, 'kl': 0.0013, 'entropy': -0.0192, 'ce_loss': 0.0015, 'epoch': 1.49}

 75%|███████▍  | 1052/1410 [3:24:42<1:11:01, 11.90s/it]
 75%|███████▍  | 1053/1410 [3:24:54<1:11:45, 12.06s/it]
                                                       
{'loss': 0.0096, 'grad_norm': 0.9089755415916443, 'learning_rate': 1.4314696223027498e-07, 'kl': 0.0236, 'entropy': -0.0601, 'ce_loss': 0.0129, 'epoch': 1.49}

 75%|███████▍  | 1053/1410 [3:24:54<1:11:45, 12.06s/it]
 75%|███████▍  | 1054/1410 [3:25:06<1:10:34, 11.89s/it]
                                                       
{'loss': 0.0175, 'grad_norm': 1.2544430494308472, 'learning_rate': 1.4239132674966446e-07, 'kl': 0.0354, 'entropy': -0.0703, 'ce_loss': 0.0112, 'epoch': 1.5}

 75%|███████▍  | 1054/1410 [3:25:06<1:10:34, 11.89s/it]
 75%|███████▍  | 1055/1410 [3:25:17<1:10:11, 11.86s/it]
                                                       
{'loss': 0.0114, 'grad_norm': 0.8764076232910156, 'learning_rate': 1.416373159238676e-07, 'kl': 0.0022, 'entropy': 0.0688, 'ce_loss': 0.1724, 'epoch': 1.5}

 75%|███████▍  | 1055/1410 [3:25:17<1:10:11, 11.86s/it]
 75%|███████▍  | 1056/1410 [3:25:28<1:08:28, 11.61s/it]
                                                       
{'loss': 0.015, 'grad_norm': 1.3278411626815796, 'learning_rate': 1.4088493373524046e-07, 'kl': 0.0237, 'entropy': -0.0708, 'ce_loss': 0.0098, 'epoch': 1.5}

 75%|███████▍  | 1056/1410 [3:25:28<1:08:28, 11.61s/it]
 75%|███████▍  | 1057/1410 [3:25:40<1:08:14, 11.60s/it]
                                                       
{'loss': 0.0123, 'grad_norm': 1.0992567539215088, 'learning_rate': 1.4013418415753805e-07, 'kl': 0.0383, 'entropy': -0.0923, 'ce_loss': 0.0093, 'epoch': 1.5}

 75%|███████▍  | 1057/1410 [3:25:40<1:08:14, 11.60s/it]
 75%|███████▌  | 1058/1410 [3:25:53<1:10:17, 11.98s/it]
                                                       
{'loss': 0.0038, 'grad_norm': 0.6001553535461426, 'learning_rate': 1.3938507115589143e-07, 'kl': -0.0035, 'entropy': -0.053, 'ce_loss': 0.0419, 'epoch': 1.5}

 75%|███████▌  | 1058/1410 [3:25:53<1:10:17, 11.98s/it]
 75%|███████▌  | 1059/1410 [3:26:04<1:08:24, 11.69s/it]
                                                       
{'loss': 0.011, 'grad_norm': 1.1052794456481934, 'learning_rate': 1.3863759868678921e-07, 'kl': 0.0391, 'entropy': -0.1177, 'ce_loss': 0.0165, 'epoch': 1.5}

 75%|███████▌  | 1059/1410 [3:26:04<1:08:24, 11.69s/it]
 75%|███████▌  | 1060/1410 [3:26:16<1:08:39, 11.77s/it]
                                                       
{'loss': 0.0176, 'grad_norm': 0.8978265523910522, 'learning_rate': 1.3789177069805464e-07, 'kl': 0.0051, 'entropy': -0.0226, 'ce_loss': 0.0011, 'epoch': 1.5}

 75%|███████▌  | 1060/1410 [3:26:16<1:08:39, 11.77s/it]
 75%|███████▌  | 1061/1410 [3:26:28<1:09:36, 11.97s/it]
                                                       
{'loss': 0.0143, 'grad_norm': 1.0087743997573853, 'learning_rate': 1.3714759112882567e-07, 'kl': 0.0306, 'entropy': -0.0894, 'ce_loss': 0.0138, 'epoch': 1.5}

 75%|███████▌  | 1061/1410 [3:26:28<1:09:36, 11.97s/it]
 75%|███████▌  | 1062/1410 [3:26:40<1:09:33, 11.99s/it]
                                                       
{'loss': 0.012, 'grad_norm': 1.1952511072158813, 'learning_rate': 1.3640506390953445e-07, 'kl': 0.0195, 'entropy': -0.0737, 'ce_loss': 0.0043, 'epoch': 1.51}

 75%|███████▌  | 1062/1410 [3:26:40<1:09:33, 11.99s/it]
 75%|███████▌  | 1063/1410 [3:26:52<1:08:02, 11.77s/it]
                                                       
{'loss': 0.0147, 'grad_norm': 1.2371379137039185, 'learning_rate': 1.3566419296188558e-07, 'kl': 0.0055, 'entropy': -0.0332, 'ce_loss': 0.0082, 'epoch': 1.51}

 75%|███████▌  | 1063/1410 [3:26:52<1:08:02, 11.77s/it]
 75%|███████▌  | 1064/1410 [3:27:04<1:09:03, 11.98s/it]
                                                       
{'loss': 0.0118, 'grad_norm': 1.3569377660751343, 'learning_rate': 1.3492498219883613e-07, 'kl': 0.0374, 'entropy': -0.0894, 'ce_loss': 0.019, 'epoch': 1.51}

 75%|███████▌  | 1064/1410 [3:27:04<1:09:03, 11.98s/it]
 76%|███████▌  | 1065/1410 [3:27:16<1:08:23, 11.89s/it]
                                                       
{'loss': 0.0065, 'grad_norm': 0.8521744608879089, 'learning_rate': 1.3418743552457523e-07, 'kl': 0.0028, 'entropy': -0.0258, 'ce_loss': 0.0025, 'epoch': 1.51}

 76%|███████▌  | 1065/1410 [3:27:16<1:08:23, 11.89s/it]
 76%|███████▌  | 1066/1410 [3:27:28<1:08:07, 11.88s/it]
                                                       
{'loss': 0.0236, 'grad_norm': 1.1335800886154175, 'learning_rate': 1.3345155683450213e-07, 'kl': 0.0315, 'entropy': -0.0723, 'ce_loss': 0.032, 'epoch': 1.51}

 76%|███████▌  | 1066/1410 [3:27:28<1:08:07, 11.88s/it]
 76%|███████▌  | 1067/1410 [3:27:40<1:08:32, 11.99s/it]
                                                       
{'loss': 0.0114, 'grad_norm': 1.0472794771194458, 'learning_rate': 1.3271735001520727e-07, 'kl': -0.0027, 'entropy': -0.0461, 'ce_loss': 0.0283, 'epoch': 1.51}

 76%|███████▌  | 1067/1410 [3:27:40<1:08:32, 11.99s/it]
 76%|███████▌  | 1068/1410 [3:27:52<1:08:15, 11.98s/it]
                                                       
{'loss': 0.0174, 'grad_norm': 1.8442517518997192, 'learning_rate': 1.3198481894445108e-07, 'kl': 0.0231, 'entropy': -0.0383, 'ce_loss': 0.0314, 'epoch': 1.51}

 76%|███████▌  | 1068/1410 [3:27:52<1:08:15, 11.98s/it]
 76%|███████▌  | 1069/1410 [3:28:04<1:08:22, 12.03s/it]
                                                       
{'loss': 0.0197, 'grad_norm': 1.1301295757293701, 'learning_rate': 1.3125396749114256e-07, 'kl': 0.0275, 'entropy': -0.085, 'ce_loss': 0.0127, 'epoch': 1.52}

 76%|███████▌  | 1069/1410 [3:28:04<1:08:22, 12.03s/it]
 76%|███████▌  | 1070/1410 [3:28:16<1:08:21, 12.06s/it]
                                                       
{'loss': 0.0175, 'grad_norm': 1.2218097448349, 'learning_rate': 1.3052479951532047e-07, 'kl': -0.0063, 'entropy': 0.0031, 'ce_loss': 0.0542, 'epoch': 1.52}

 76%|███████▌  | 1070/1410 [3:28:16<1:08:21, 12.06s/it]
 76%|███████▌  | 1071/1410 [3:28:29<1:09:26, 12.29s/it]
                                                       
{'loss': 0.0305, 'grad_norm': 0.8013397455215454, 'learning_rate': 1.297973188681322e-07, 'kl': -0.0056, 'entropy': 0.0386, 'ce_loss': 0.1015, 'epoch': 1.52}

 76%|███████▌  | 1071/1410 [3:28:29<1:09:26, 12.29s/it]
 76%|███████▌  | 1072/1410 [3:28:41<1:08:43, 12.20s/it]
                                                       
{'loss': 0.0106, 'grad_norm': 1.085119605064392, 'learning_rate': 1.2907152939181268e-07, 'kl': 0.0209, 'entropy': -0.082, 'ce_loss': 0.0143, 'epoch': 1.52}

 76%|███████▌  | 1072/1410 [3:28:41<1:08:43, 12.20s/it]
 76%|███████▌  | 1073/1410 [3:28:53<1:07:48, 12.07s/it]
                                                       
{'loss': 0.0121, 'grad_norm': 1.1546775102615356, 'learning_rate': 1.2834743491966567e-07, 'kl': 0.0282, 'entropy': -0.0728, 'ce_loss': 0.0082, 'epoch': 1.52}

 76%|███████▌  | 1073/1410 [3:28:53<1:07:48, 12.07s/it]
 76%|███████▌  | 1074/1410 [3:29:04<1:06:48, 11.93s/it]
                                                       
{'loss': 0.0201, 'grad_norm': 0.9987554550170898, 'learning_rate': 1.2762503927604218e-07, 'kl': 0.0403, 'entropy': -0.127, 'ce_loss': 0.0106, 'epoch': 1.52}

 76%|███████▌  | 1074/1410 [3:29:04<1:06:48, 11.93s/it]
 76%|███████▌  | 1075/1410 [3:29:16<1:07:12, 12.04s/it]
                                                       
{'loss': 0.0112, 'grad_norm': 1.2792311906814575, 'learning_rate': 1.269043462763206e-07, 'kl': -0.0017, 'entropy': 0.009, 'ce_loss': 0.0826, 'epoch': 1.52}

 76%|███████▌  | 1075/1410 [3:29:17<1:07:12, 12.04s/it]
 76%|███████▋  | 1076/1410 [3:29:29<1:06:57, 12.03s/it]
                                                       
{'loss': 0.0219, 'grad_norm': 1.2486177682876587, 'learning_rate': 1.261853597268873e-07, 'kl': 0.0258, 'entropy': -0.0635, 'ce_loss': 0.0177, 'epoch': 1.53}

 76%|███████▋  | 1076/1410 [3:29:29<1:06:57, 12.03s/it]
 76%|███████▋  | 1077/1410 [3:29:40<1:06:21, 11.96s/it]
                                                       
{'loss': 0.0108, 'grad_norm': 1.059685468673706, 'learning_rate': 1.2546808342511544e-07, 'kl': 0.0233, 'entropy': -0.0688, 'ce_loss': 0.0076, 'epoch': 1.53}

 76%|███████▋  | 1077/1410 [3:29:40<1:06:21, 11.96s/it]
 76%|███████▋  | 1078/1410 [3:29:52<1:05:46, 11.89s/it]
                                                       
{'loss': 0.0059, 'grad_norm': 0.7568551898002625, 'learning_rate': 1.247525211593453e-07, 'kl': 0.0342, 'entropy': -0.0898, 'ce_loss': 0.0176, 'epoch': 1.53}

 76%|███████▋  | 1078/1410 [3:29:52<1:05:46, 11.89s/it]
 77%|███████▋  | 1079/1410 [3:30:04<1:05:05, 11.80s/it]
                                                       
{'loss': 0.012, 'grad_norm': 0.9919165968894958, 'learning_rate': 1.24038676708865e-07, 'kl': -0.0021, 'entropy': -0.0211, 'ce_loss': 0.003, 'epoch': 1.53}

 77%|███████▋  | 1079/1410 [3:30:04<1:05:05, 11.80s/it]
 77%|███████▋  | 1080/1410 [3:30:15<1:04:45, 11.77s/it]
                                                       
{'loss': 0.0172, 'grad_norm': 1.2453138828277588, 'learning_rate': 1.2332655384388923e-07, 'kl': 0.0435, 'entropy': -0.04, 'ce_loss': 0.0026, 'epoch': 1.53}

 77%|███████▋  | 1080/1410 [3:30:15<1:04:45, 11.77s/it]
 77%|███████▋  | 1081/1410 [3:30:27<1:05:01, 11.86s/it]
                                                       
{'loss': 0.0099, 'grad_norm': 1.1918364763259888, 'learning_rate': 1.2261615632554023e-07, 'kl': 0.0018, 'entropy': -0.0059, 'ce_loss': 0.0092, 'epoch': 1.53}

 77%|███████▋  | 1081/1410 [3:30:27<1:05:01, 11.86s/it]
 77%|███████▋  | 1082/1410 [3:30:40<1:05:54, 12.06s/it]
                                                       
{'loss': 0.0103, 'grad_norm': 0.7052896618843079, 'learning_rate': 1.2190748790582815e-07, 'kl': -0.0128, 'entropy': -0.0447, 'ce_loss': 0.0062, 'epoch': 1.53}

 77%|███████▋  | 1082/1410 [3:30:40<1:05:54, 12.06s/it]
 77%|███████▋  | 1083/1410 [3:30:52<1:05:44, 12.06s/it]
                                                       
{'loss': 0.0112, 'grad_norm': 1.0168483257293701, 'learning_rate': 1.2120055232762993e-07, 'kl': 0.032, 'entropy': -0.1338, 'ce_loss': 0.006, 'epoch': 1.54}

 77%|███████▋  | 1083/1410 [3:30:52<1:05:44, 12.06s/it]
 77%|███████▋  | 1084/1410 [3:31:04<1:05:17, 12.02s/it]
                                                       
{'loss': 0.0202, 'grad_norm': 1.1856529712677002, 'learning_rate': 1.2049535332467123e-07, 'kl': 0.0245, 'entropy': -0.0859, 'ce_loss': 0.0119, 'epoch': 1.54}

 77%|███████▋  | 1084/1410 [3:31:04<1:05:17, 12.02s/it]
 77%|███████▋  | 1085/1410 [3:31:16<1:04:54, 11.98s/it]
                                                       
{'loss': 0.018, 'grad_norm': 1.5647615194320679, 'learning_rate': 1.1979189462150543e-07, 'kl': 0.0156, 'entropy': -0.0747, 'ce_loss': 0.0157, 'epoch': 1.54}

 77%|███████▋  | 1085/1410 [3:31:16<1:04:54, 11.98s/it]
 77%|███████▋  | 1086/1410 [3:31:28<1:04:27, 11.94s/it]
                                                       
{'loss': 0.0129, 'grad_norm': 1.0943100452423096, 'learning_rate': 1.190901799334943e-07, 'kl': 0.0378, 'entropy': -0.0864, 'ce_loss': 0.0112, 'epoch': 1.54}

 77%|███████▋  | 1086/1410 [3:31:28<1:04:27, 11.94s/it]
 77%|███████▋  | 1087/1410 [3:31:40<1:04:23, 11.96s/it]
                                                       
{'loss': 0.0099, 'grad_norm': 0.6951851844787598, 'learning_rate': 1.1839021296678906e-07, 'kl': 0.003, 'entropy': -0.0208, 'ce_loss': 0.004, 'epoch': 1.54}

 77%|███████▋  | 1087/1410 [3:31:40<1:04:23, 11.96s/it]
 77%|███████▋  | 1088/1410 [3:31:51<1:03:36, 11.85s/it]
                                                       
{'loss': 0.0222, 'grad_norm': 1.0879595279693604, 'learning_rate': 1.1769199741830968e-07, 'kl': 0.019, 'entropy': -0.0547, 'ce_loss': 0.002, 'epoch': 1.54}

 77%|███████▋  | 1088/1410 [3:31:51<1:03:36, 11.85s/it]
 77%|███████▋  | 1089/1410 [3:32:02<1:01:44, 11.54s/it]
                                                       
{'loss': 0.0172, 'grad_norm': 1.3307747840881348, 'learning_rate': 1.1699553697572595e-07, 'kl': 0.043, 'entropy': -0.1035, 'ce_loss': 0.0106, 'epoch': 1.54}

 77%|███████▋  | 1089/1410 [3:32:02<1:01:44, 11.54s/it]
 77%|███████▋  | 1090/1410 [3:32:14<1:02:09, 11.65s/it]
                                                       
{'loss': 0.0167, 'grad_norm': 1.2184715270996094, 'learning_rate': 1.1630083531743838e-07, 'kl': 0.0214, 'entropy': -0.0654, 'ce_loss': 0.0209, 'epoch': 1.55}

 77%|███████▋  | 1090/1410 [3:32:14<1:02:09, 11.65s/it]
 77%|███████▋  | 1091/1410 [3:32:26<1:02:13, 11.70s/it]
                                                       
{'loss': 0.0114, 'grad_norm': 1.2980769872665405, 'learning_rate': 1.1560789611255782e-07, 'kl': 0.015, 'entropy': -0.0598, 'ce_loss': 0.0144, 'epoch': 1.55}

 77%|███████▋  | 1091/1410 [3:32:26<1:02:13, 11.70s/it]
 77%|███████▋  | 1092/1410 [3:32:38<1:02:47, 11.85s/it]
                                                       
{'loss': 0.0406, 'grad_norm': 1.2897409200668335, 'learning_rate': 1.14916723020887e-07, 'kl': 0.0396, 'entropy': -0.0535, 'ce_loss': 0.0035, 'epoch': 1.55}

 77%|███████▋  | 1092/1410 [3:32:38<1:02:47, 11.85s/it]
 78%|███████▊  | 1093/1410 [3:32:50<1:02:19, 11.80s/it]
                                                       
{'loss': 0.0066, 'grad_norm': 0.7172817587852478, 'learning_rate': 1.1422731969290037e-07, 'kl': 0.0096, 'entropy': -0.0413, 'ce_loss': 0.003, 'epoch': 1.55}

 78%|███████▊  | 1093/1410 [3:32:50<1:02:19, 11.80s/it]
 78%|███████▊  | 1094/1410 [3:33:01<1:01:51, 11.75s/it]
                                                       
{'loss': 0.007, 'grad_norm': 0.9814090132713318, 'learning_rate': 1.1353968976972594e-07, 'kl': -0.0022, 'entropy': 0.0155, 'ce_loss': 0.0644, 'epoch': 1.55}

 78%|███████▊  | 1094/1410 [3:33:01<1:01:51, 11.75s/it]
 78%|███████▊  | 1095/1410 [3:33:13<1:01:13, 11.66s/it]
                                                       
{'loss': 0.0123, 'grad_norm': 1.0930297374725342, 'learning_rate': 1.1285383688312476e-07, 'kl': 0.0139, 'entropy': -0.0879, 'ce_loss': 0.0169, 'epoch': 1.55}

 78%|███████▊  | 1095/1410 [3:33:13<1:01:13, 11.66s/it]
 78%|███████▊  | 1096/1410 [3:33:25<1:01:16, 11.71s/it]
                                                       
{'loss': 0.0147, 'grad_norm': 1.2069002389907837, 'learning_rate': 1.1216976465547262e-07, 'kl': 0.0247, 'entropy': -0.0698, 'ce_loss': 0.0302, 'epoch': 1.55}

 78%|███████▊  | 1096/1410 [3:33:25<1:01:16, 11.71s/it]
 78%|███████▊  | 1097/1410 [3:33:37<1:01:29, 11.79s/it]
                                                       
{'loss': 0.022, 'grad_norm': 1.2510755062103271, 'learning_rate': 1.1148747669974038e-07, 'kl': 0.0204, 'entropy': -0.0518, 'ce_loss': 0.0144, 'epoch': 1.56}

 78%|███████▊  | 1097/1410 [3:33:37<1:01:29, 11.79s/it]
 78%|███████▊  | 1098/1410 [3:33:48<1:01:10, 11.77s/it]
                                                       
{'loss': 0.0146, 'grad_norm': 1.2553032636642456, 'learning_rate': 1.1080697661947573e-07, 'kl': 0.0286, 'entropy': -0.0791, 'ce_loss': 0.0203, 'epoch': 1.56}

 78%|███████▊  | 1098/1410 [3:33:48<1:01:10, 11.77s/it]
 78%|███████▊  | 1099/1410 [3:34:00<1:01:17, 11.82s/it]
                                                       
{'loss': 0.0139, 'grad_norm': 0.7156863808631897, 'learning_rate': 1.1012826800878305e-07, 'kl': 0.0043, 'entropy': 0.103, 'ce_loss': 0.1062, 'epoch': 1.56}

 78%|███████▊  | 1099/1410 [3:34:00<1:01:17, 11.82s/it]
 78%|███████▊  | 1100/1410 [3:34:12<1:01:09, 11.84s/it]
                                                       
{'loss': 0.0246, 'grad_norm': 0.8671999573707581, 'learning_rate': 1.0945135445230496e-07, 'kl': -0.002, 'entropy': 0.1128, 'ce_loss': 0.0952, 'epoch': 1.56}

 78%|███████▊  | 1100/1410 [3:34:12<1:01:09, 11.84s/it]
 78%|███████▊  | 1101/1410 [3:34:24<1:00:45, 11.80s/it]
                                                       
{'loss': 0.0136, 'grad_norm': 1.3183059692382812, 'learning_rate': 1.087762395252038e-07, 'kl': 0.0165, 'entropy': -0.041, 'ce_loss': 0.0056, 'epoch': 1.56}

 78%|███████▊  | 1101/1410 [3:34:24<1:00:45, 11.80s/it]
 78%|███████▊  | 1102/1410 [3:34:36<1:00:51, 11.85s/it]
                                                       
{'loss': 0.0243, 'grad_norm': 0.9933121204376221, 'learning_rate': 1.0810292679314205e-07, 'kl': 0.0337, 'entropy': -0.0693, 'ce_loss': 0.0423, 'epoch': 1.56}

 78%|███████▊  | 1102/1410 [3:34:36<1:00:51, 11.85s/it]
 78%|███████▊  | 1103/1410 [3:34:48<1:00:44, 11.87s/it]
                                                       
{'loss': 0.0186, 'grad_norm': 1.334240436553955, 'learning_rate': 1.0743141981226372e-07, 'kl': 0.0366, 'entropy': -0.1016, 'ce_loss': 0.0088, 'epoch': 1.56}

 78%|███████▊  | 1103/1410 [3:34:48<1:00:44, 11.87s/it]
 78%|███████▊  | 1104/1410 [3:35:00<1:00:36, 11.88s/it]
                                                       
{'loss': 0.0177, 'grad_norm': 1.4058490991592407, 'learning_rate': 1.0676172212917567e-07, 'kl': 0.0058, 'entropy': -0.0486, 'ce_loss': 0.0094, 'epoch': 1.57}

 78%|███████▊  | 1104/1410 [3:35:00<1:00:36, 11.88s/it]
 78%|███████▊  | 1105/1410 [3:35:12<1:00:57, 11.99s/it]
                                                       
{'loss': 0.0197, 'grad_norm': 0.7789604663848877, 'learning_rate': 1.0609383728092913e-07, 'kl': 0.0242, 'entropy': -0.0225, 'ce_loss': 0.0306, 'epoch': 1.57}

 78%|███████▊  | 1105/1410 [3:35:12<1:00:57, 11.99s/it]
 78%|███████▊  | 1106/1410 [3:35:24<1:01:13, 12.08s/it]
                                                       
{'loss': 0.0193, 'grad_norm': 0.6132597327232361, 'learning_rate': 1.0542776879500034e-07, 'kl': -0.0013, 'entropy': 0.0361, 'ce_loss': 0.0854, 'epoch': 1.57}

 78%|███████▊  | 1106/1410 [3:35:24<1:01:13, 12.08s/it]
 79%|███████▊  | 1107/1410 [3:35:36<1:00:46, 12.03s/it]
                                                       
{'loss': 0.0141, 'grad_norm': 1.1160366535186768, 'learning_rate': 1.0476352018927237e-07, 'kl': 0.0352, 'entropy': -0.0767, 'ce_loss': 0.0047, 'epoch': 1.57}

 79%|███████▊  | 1107/1410 [3:35:36<1:00:46, 12.03s/it]
 79%|███████▊  | 1108/1410 [3:35:48<1:00:17, 11.98s/it]
                                                       
{'loss': 0.0213, 'grad_norm': 0.972956657409668, 'learning_rate': 1.0410109497201679e-07, 'kl': 0.0025, 'entropy': -0.031, 'ce_loss': 0.0009, 'epoch': 1.57}

 79%|███████▊  | 1108/1410 [3:35:48<1:00:17, 11.98s/it]
 79%|███████▊  | 1109/1410 [3:36:00<59:39, 11.89s/it]  
                                                     
{'loss': 0.0099, 'grad_norm': 0.9878646731376648, 'learning_rate': 1.0344049664187452e-07, 'kl': 0.0457, 'entropy': -0.0811, 'ce_loss': 0.0162, 'epoch': 1.57}

 79%|███████▊  | 1109/1410 [3:36:00<59:39, 11.89s/it]
 79%|███████▊  | 1110/1410 [3:36:11<59:05, 11.82s/it]
                                                     
{'loss': 0.0094, 'grad_norm': 0.9534783363342285, 'learning_rate': 1.027817286878376e-07, 'kl': 0.0334, 'entropy': -0.0938, 'ce_loss': 0.0172, 'epoch': 1.57}

 79%|███████▊  | 1110/1410 [3:36:11<59:05, 11.82s/it]
 79%|███████▉  | 1111/1410 [3:36:23<58:55, 11.82s/it]
                                                     
{'loss': 0.0171, 'grad_norm': 1.143884539604187, 'learning_rate': 1.0212479458923127e-07, 'kl': 0.0317, 'entropy': -0.0957, 'ce_loss': 0.0195, 'epoch': 1.58}

 79%|███████▉  | 1111/1410 [3:36:23<58:55, 11.82s/it]
 79%|███████▉  | 1112/1410 [3:36:34<58:03, 11.69s/it]
                                                     
{'loss': 0.0109, 'grad_norm': 1.2377409934997559, 'learning_rate': 1.014696978156945e-07, 'kl': 0.0161, 'entropy': -0.0403, 'ce_loss': 0.0169, 'epoch': 1.58}

 79%|███████▉  | 1112/1410 [3:36:34<58:03, 11.69s/it]
 79%|███████▉  | 1113/1410 [3:36:46<58:01, 11.72s/it]
                                                     
{'loss': 0.0202, 'grad_norm': 1.1363073587417603, 'learning_rate': 1.008164418271629e-07, 'kl': 0.0189, 'entropy': -0.0791, 'ce_loss': 0.0051, 'epoch': 1.58}

 79%|███████▉  | 1113/1410 [3:36:46<58:01, 11.72s/it]
 79%|███████▉  | 1114/1410 [3:36:58<57:53, 11.74s/it]
                                                     
{'loss': 0.0076, 'grad_norm': 0.8548643589019775, 'learning_rate': 1.0016503007384953e-07, 'kl': 0.0405, 'entropy': -0.0703, 'ce_loss': 0.0045, 'epoch': 1.58}

 79%|███████▉  | 1114/1410 [3:36:58<57:53, 11.74s/it]
 79%|███████▉  | 1115/1410 [3:37:10<57:50, 11.76s/it]
                                                     
{'loss': 0.0106, 'grad_norm': 1.0387905836105347, 'learning_rate': 9.951546599622697e-08, 'kl': -0.0074, 'entropy': -0.0156, 'ce_loss': 0.0589, 'epoch': 1.58}

 79%|███████▉  | 1115/1410 [3:37:10<57:50, 11.76s/it]
 79%|███████▉  | 1116/1410 [3:37:22<57:29, 11.73s/it]
                                                     
{'loss': 0.0187, 'grad_norm': 1.0783772468566895, 'learning_rate': 9.886775302500951e-08, 'kl': 0.0376, 'entropy': -0.0825, 'ce_loss': 0.0327, 'epoch': 1.58}

 79%|███████▉  | 1116/1410 [3:37:22<57:29, 11.73s/it]
 79%|███████▉  | 1117/1410 [3:37:33<57:29, 11.77s/it]
                                                     
{'loss': 0.0129, 'grad_norm': 1.183302402496338, 'learning_rate': 9.822189458113434e-08, 'kl': 0.0256, 'entropy': -0.1006, 'ce_loss': 0.0078, 'epoch': 1.58}

 79%|███████▉  | 1117/1410 [3:37:33<57:29, 11.77s/it]
 79%|███████▉  | 1118/1410 [3:37:45<57:08, 11.74s/it]
                                                     
{'loss': 0.0127, 'grad_norm': 0.9833526611328125, 'learning_rate': 9.757789407574399e-08, 'kl': 0.0228, 'entropy': -0.0962, 'ce_loss': 0.0103, 'epoch': 1.59}

 79%|███████▉  | 1118/1410 [3:37:45<57:08, 11.74s/it]
 79%|███████▉  | 1119/1410 [3:37:57<56:47, 11.71s/it]
                                                     
{'loss': 0.0079, 'grad_norm': 0.9644126296043396, 'learning_rate': 9.693575491016832e-08, 'kl': -0.0004, 'entropy': -0.0182, 'ce_loss': 0.0005, 'epoch': 1.59}

 79%|███████▉  | 1119/1410 [3:37:57<56:47, 11.71s/it]
 79%|███████▉  | 1120/1410 [3:38:09<56:58, 11.79s/it]
                                                     
{'loss': 0.0106, 'grad_norm': 0.8396703004837036, 'learning_rate': 9.629548047590616e-08, 'kl': 0.0325, 'entropy': -0.0449, 'ce_loss': 0.0326, 'epoch': 1.59}

 79%|███████▉  | 1120/1410 [3:38:09<56:58, 11.79s/it]
 80%|███████▉  | 1121/1410 [3:38:21<56:52, 11.81s/it]
                                                     
{'loss': 0.0116, 'grad_norm': 1.2677175998687744, 'learning_rate': 9.565707415460762e-08, 'kl': -0.0008, 'entropy': 0.0859, 'ce_loss': 0.0843, 'epoch': 1.59}

 80%|███████▉  | 1121/1410 [3:38:21<56:52, 11.81s/it]
 80%|███████▉  | 1122/1410 [3:38:33<57:04, 11.89s/it]
                                                     
{'loss': 0.0332, 'grad_norm': 0.9209173321723938, 'learning_rate': 9.50205393180568e-08, 'kl': -0.0019, 'entropy': 0.082, 'ce_loss': 0.1318, 'epoch': 1.59}

 80%|███████▉  | 1122/1410 [3:38:33<57:04, 11.89s/it]
 80%|███████▉  | 1123/1410 [3:38:44<56:14, 11.76s/it]
                                                     
{'loss': 0.0197, 'grad_norm': 1.265001654624939, 'learning_rate': 9.438587932815242e-08, 'kl': 0.0216, 'entropy': -0.0747, 'ce_loss': 0.0292, 'epoch': 1.59}

 80%|███████▉  | 1123/1410 [3:38:44<56:14, 11.76s/it]
 80%|███████▉  | 1124/1410 [3:38:56<56:24, 11.83s/it]
                                                     
{'loss': 0.0244, 'grad_norm': 1.0787782669067383, 'learning_rate': 9.375309753689199e-08, 'kl': 0.0121, 'entropy': -0.0476, 'ce_loss': 0.0037, 'epoch': 1.59}

 80%|███████▉  | 1124/1410 [3:38:56<56:24, 11.83s/it]
 80%|███████▉  | 1125/1410 [3:39:08<56:24, 11.87s/it]
                                                     
{'loss': 0.0279, 'grad_norm': 1.1004972457885742, 'learning_rate': 9.31221972863531e-08, 'kl': -0.009, 'entropy': 0.0036, 'ce_loss': 0.0814, 'epoch': 1.6}

 80%|███████▉  | 1125/1410 [3:39:08<56:24, 11.87s/it]
 80%|███████▉  | 1126/1410 [3:39:20<55:53, 11.81s/it]
                                                     
{'loss': 0.0087, 'grad_norm': 0.8828406929969788, 'learning_rate': 9.24931819086751e-08, 'kl': 0.0192, 'entropy': -0.0947, 'ce_loss': 0.0129, 'epoch': 1.6}

 80%|███████▉  | 1126/1410 [3:39:20<55:53, 11.81s/it]
 80%|███████▉  | 1127/1410 [3:39:31<55:42, 11.81s/it]
                                                     
{'loss': 0.0232, 'grad_norm': 1.5345458984375, 'learning_rate': 9.186605472604326e-08, 'kl': 0.0229, 'entropy': -0.0825, 'ce_loss': 0.0146, 'epoch': 1.6}

 80%|███████▉  | 1127/1410 [3:39:31<55:42, 11.81s/it]
 80%|████████  | 1128/1410 [3:39:43<55:26, 11.80s/it]
                                                     
{'loss': 0.0135, 'grad_norm': 1.3571845293045044, 'learning_rate': 9.124081905066955e-08, 'kl': -0.003, 'entropy': 0.0654, 'ce_loss': 0.1118, 'epoch': 1.6}

 80%|████████  | 1128/1410 [3:39:43<55:26, 11.80s/it]
 80%|████████  | 1129/1410 [3:39:55<55:13, 11.79s/it]
                                                     
{'loss': 0.0139, 'grad_norm': 0.9778938889503479, 'learning_rate': 9.061747818477595e-08, 'kl': 0.0188, 'entropy': -0.0583, 'ce_loss': 0.0171, 'epoch': 1.6}

 80%|████████  | 1129/1410 [3:39:55<55:13, 11.79s/it]
 80%|████████  | 1130/1410 [3:40:06<53:45, 11.52s/it]
                                                     
{'loss': 0.0143, 'grad_norm': 1.2214231491088867, 'learning_rate': 8.999603542057717e-08, 'kl': -0.0079, 'entropy': 0.0388, 'ce_loss': 0.161, 'epoch': 1.6}

 80%|████████  | 1130/1410 [3:40:06<53:45, 11.52s/it]
 80%|████████  | 1131/1410 [3:40:18<54:00, 11.61s/it]
                                                     
{'loss': 0.0205, 'grad_norm': 1.5868747234344482, 'learning_rate': 8.937649404026278e-08, 'kl': 0.0261, 'entropy': -0.064, 'ce_loss': 0.01, 'epoch': 1.6}

 80%|████████  | 1131/1410 [3:40:18<54:00, 11.61s/it]
 80%|████████  | 1132/1410 [3:40:30<54:15, 11.71s/it]
                                                     
{'loss': 0.039, 'grad_norm': 1.1466909646987915, 'learning_rate': 8.875885731597984e-08, 'kl': 0.0259, 'entropy': -0.0806, 'ce_loss': 0.0143, 'epoch': 1.61}

 80%|████████  | 1132/1410 [3:40:30<54:15, 11.71s/it]
 80%|████████  | 1133/1410 [3:40:42<54:19, 11.77s/it]
                                                     
{'loss': 0.0114, 'grad_norm': 0.9219392538070679, 'learning_rate': 8.814312850981634e-08, 'kl': 0.011, 'entropy': -0.0659, 'ce_loss': 0.0079, 'epoch': 1.61}

 80%|████████  | 1133/1410 [3:40:42<54:19, 11.77s/it]
 80%|████████  | 1134/1410 [3:40:54<54:25, 11.83s/it]
                                                     
{'loss': 0.0264, 'grad_norm': 1.107546091079712, 'learning_rate': 8.752931087378314e-08, 'kl': 0.0352, 'entropy': -0.0908, 'ce_loss': 0.0083, 'epoch': 1.61}

 80%|████████  | 1134/1410 [3:40:54<54:25, 11.83s/it]
 80%|████████  | 1135/1410 [3:41:05<54:04, 11.80s/it]
                                                     
{'loss': 0.0067, 'grad_norm': 1.0198466777801514, 'learning_rate': 8.691740764979703e-08, 'kl': 0.0366, 'entropy': -0.1099, 'ce_loss': 0.0087, 'epoch': 1.61}

 80%|████████  | 1135/1410 [3:41:05<54:04, 11.80s/it]
 81%|████████  | 1136/1410 [3:41:17<53:41, 11.76s/it]
                                                     
{'loss': 0.0243, 'grad_norm': 1.2089977264404297, 'learning_rate': 8.630742206966423e-08, 'kl': 0.0337, 'entropy': -0.0688, 'ce_loss': 0.0037, 'epoch': 1.61}

 81%|████████  | 1136/1410 [3:41:17<53:41, 11.76s/it]
 81%|████████  | 1137/1410 [3:41:29<53:29, 11.76s/it]
                                                     
{'loss': 0.0048, 'grad_norm': 0.9792128801345825, 'learning_rate': 8.569935735506207e-08, 'kl': -0.0059, 'entropy': 0.0334, 'ce_loss': 0.0856, 'epoch': 1.61}

 81%|████████  | 1137/1410 [3:41:29<53:29, 11.76s/it]
 81%|████████  | 1138/1410 [3:41:40<52:37, 11.61s/it]
                                                     
{'loss': 0.0157, 'grad_norm': 1.370151162147522, 'learning_rate': 8.509321671752338e-08, 'kl': 0.0498, 'entropy': -0.0674, 'ce_loss': 0.009, 'epoch': 1.61}

 81%|████████  | 1138/1410 [3:41:40<52:37, 11.61s/it]
 81%|████████  | 1139/1410 [3:41:52<52:21, 11.59s/it]
                                                     
{'loss': 0.0038, 'grad_norm': 0.5416613817214966, 'learning_rate': 8.448900335841849e-08, 'kl': 0.0041, 'entropy': -0.0371, 'ce_loss': 0.0036, 'epoch': 1.62}

 81%|████████  | 1139/1410 [3:41:52<52:21, 11.59s/it]
 81%|████████  | 1140/1410 [3:42:03<52:32, 11.67s/it]
                                                     
{'loss': 0.0156, 'grad_norm': 0.9615955352783203, 'learning_rate': 8.38867204689386e-08, 'kl': 0.0255, 'entropy': -0.0598, 'ce_loss': 0.0176, 'epoch': 1.62}

 81%|████████  | 1140/1410 [3:42:03<52:32, 11.67s/it]
 81%|████████  | 1141/1410 [3:42:15<52:47, 11.78s/it]
                                                     
{'loss': 0.0304, 'grad_norm': 1.4258067607879639, 'learning_rate': 8.328637123007948e-08, 'kl': 0.0078, 'entropy': -0.0272, 'ce_loss': 0.0009, 'epoch': 1.62}

 81%|████████  | 1141/1410 [3:42:15<52:47, 11.78s/it]
 81%|████████  | 1142/1410 [3:42:26<51:27, 11.52s/it]
                                                     
{'loss': 0.0075, 'grad_norm': 1.018634557723999, 'learning_rate': 8.268795881262373e-08, 'kl': 0.0011, 'entropy': -0.0413, 'ce_loss': 0.0022, 'epoch': 1.62}

 81%|████████  | 1142/1410 [3:42:26<51:27, 11.52s/it]
 81%|████████  | 1143/1410 [3:42:38<51:32, 11.58s/it]
                                                     
{'loss': 0.0142, 'grad_norm': 1.1264537572860718, 'learning_rate': 8.209148637712455e-08, 'kl': 0.016, 'entropy': -0.0649, 'ce_loss': 0.0115, 'epoch': 1.62}

 81%|████████  | 1143/1410 [3:42:38<51:32, 11.58s/it]
 81%|████████  | 1144/1410 [3:42:50<51:50, 11.69s/it]
                                                     
{'loss': 0.0147, 'grad_norm': 1.214585781097412, 'learning_rate': 8.14969570738894e-08, 'kl': 0.0177, 'entropy': -0.051, 'ce_loss': 0.0358, 'epoch': 1.62}

 81%|████████  | 1144/1410 [3:42:50<51:50, 11.69s/it]
 81%|████████  | 1145/1410 [3:43:02<52:05, 11.80s/it]
                                                     
{'loss': 0.0257, 'grad_norm': 1.1246322393417358, 'learning_rate': 8.09043740429626e-08, 'kl': 0.0237, 'entropy': -0.0898, 'ce_loss': 0.0237, 'epoch': 1.62}

 81%|████████  | 1145/1410 [3:43:02<52:05, 11.80s/it]
 81%|████████▏ | 1146/1410 [3:43:14<51:50, 11.78s/it]
                                                     
{'loss': 0.0106, 'grad_norm': 1.5705102682113647, 'learning_rate': 8.031374041410899e-08, 'kl': 0.041, 'entropy': -0.0996, 'ce_loss': 0.0176, 'epoch': 1.63}

 81%|████████▏ | 1146/1410 [3:43:14<51:50, 11.78s/it]
 81%|████████▏ | 1147/1410 [3:43:26<51:42, 11.80s/it]
                                                     
{'loss': 0.0096, 'grad_norm': 0.9885153770446777, 'learning_rate': 7.972505930679798e-08, 'kl': -0.0033, 'entropy': -0.0698, 'ce_loss': 0.0448, 'epoch': 1.63}

 81%|████████▏ | 1147/1410 [3:43:26<51:42, 11.80s/it]
 81%|████████▏ | 1148/1410 [3:43:38<51:52, 11.88s/it]
                                                     
{'loss': 0.0381, 'grad_norm': 1.191404104232788, 'learning_rate': 7.91383338301863e-08, 'kl': 0.0216, 'entropy': -0.0786, 'ce_loss': 0.0203, 'epoch': 1.63}

 81%|████████▏ | 1148/1410 [3:43:38<51:52, 11.88s/it]
 81%|████████▏ | 1149/1410 [3:43:50<52:43, 12.12s/it]
                                                     
{'loss': 0.0099, 'grad_norm': 0.9652087688446045, 'learning_rate': 7.855356708310195e-08, 'kl': -0.0051, 'entropy': 0.0029, 'ce_loss': 0.0953, 'epoch': 1.63}

 81%|████████▏ | 1149/1410 [3:43:50<52:43, 12.12s/it]
 82%|████████▏ | 1150/1410 [3:44:01<51:07, 11.80s/it]
                                                     
{'loss': 0.0145, 'grad_norm': 1.4064462184906006, 'learning_rate': 7.797076215402769e-08, 'kl': 0.03, 'entropy': -0.0771, 'ce_loss': 0.0034, 'epoch': 1.63}

 82%|████████▏ | 1150/1410 [3:44:01<51:07, 11.80s/it]
 82%|████████▏ | 1151/1410 [3:44:14<51:27, 11.92s/it]
                                                     
{'loss': 0.0237, 'grad_norm': 1.5455536842346191, 'learning_rate': 7.738992212108483e-08, 'kl': -0.0058, 'entropy': -0.0403, 'ce_loss': 0.063, 'epoch': 1.63}

 82%|████████▏ | 1151/1410 [3:44:14<51:27, 11.92s/it]
 82%|████████▏ | 1152/1410 [3:44:26<51:12, 11.91s/it]
                                                     
{'loss': 0.0084, 'grad_norm': 1.007320761680603, 'learning_rate': 7.681105005201728e-08, 'kl': -0.0074, 'entropy': -0.0162, 'ce_loss': 0.09, 'epoch': 1.63}

 82%|████████▏ | 1152/1410 [3:44:26<51:12, 11.91s/it]
 82%|████████▏ | 1153/1410 [3:44:37<50:56, 11.89s/it]
                                                     
{'loss': 0.0086, 'grad_norm': 1.0688375234603882, 'learning_rate': 7.623414900417467e-08, 'kl': -0.005, 'entropy': 0.0269, 'ce_loss': 0.0791, 'epoch': 1.64}

 82%|████████▏ | 1153/1410 [3:44:37<50:56, 11.89s/it]
 82%|████████▏ | 1154/1410 [3:44:49<50:28, 11.83s/it]
                                                     
{'loss': 0.0105, 'grad_norm': 1.1217597723007202, 'learning_rate': 7.565922202449644e-08, 'kl': 0.0145, 'entropy': -0.0315, 'ce_loss': 0.0006, 'epoch': 1.64}

 82%|████████▏ | 1154/1410 [3:44:49<50:28, 11.83s/it]
 82%|████████▏ | 1155/1410 [3:45:00<49:41, 11.69s/it]
                                                     
{'loss': 0.0114, 'grad_norm': 1.1090847253799438, 'learning_rate': 7.508627214949643e-08, 'kl': 0.0201, 'entropy': -0.0613, 'ce_loss': 0.0061, 'epoch': 1.64}

 82%|████████▏ | 1155/1410 [3:45:00<49:41, 11.69s/it]
 82%|████████▏ | 1156/1410 [3:45:13<50:22, 11.90s/it]
                                                     
{'loss': 0.0197, 'grad_norm': 0.9436354637145996, 'learning_rate': 7.45153024052457e-08, 'kl': 0.0004, 'entropy': 0.0513, 'ce_loss': 0.0687, 'epoch': 1.64}

 82%|████████▏ | 1156/1410 [3:45:13<50:22, 11.90s/it]
 82%|████████▏ | 1157/1410 [3:45:25<50:46, 12.04s/it]
                                                     
{'loss': 0.0184, 'grad_norm': 1.3339283466339111, 'learning_rate': 7.394631580735738e-08, 'kl': 0.0007, 'entropy': -0.0229, 'ce_loss': 0.0013, 'epoch': 1.64}

 82%|████████▏ | 1157/1410 [3:45:25<50:46, 12.04s/it]
 82%|████████▏ | 1158/1410 [3:45:37<50:22, 12.00s/it]
                                                     
{'loss': 0.0104, 'grad_norm': 1.049203872680664, 'learning_rate': 7.337931536097039e-08, 'kl': -0.0077, 'entropy': 0.085, 'ce_loss': 0.1315, 'epoch': 1.64}

 82%|████████▏ | 1158/1410 [3:45:37<50:22, 12.00s/it]
 82%|████████▏ | 1159/1410 [3:45:49<50:12, 12.00s/it]
                                                     
{'loss': 0.0195, 'grad_norm': 0.9893776178359985, 'learning_rate': 7.281430406073389e-08, 'kl': -0.0, 'entropy': -0.0139, 'ce_loss': 0.0001, 'epoch': 1.64}

 82%|████████▏ | 1159/1410 [3:45:49<50:12, 12.00s/it]
 82%|████████▏ | 1160/1410 [3:46:00<49:04, 11.78s/it]
                                                     
{'loss': 0.0248, 'grad_norm': 1.0883660316467285, 'learning_rate': 7.225128489079101e-08, 'kl': 0.0187, 'entropy': -0.0615, 'ce_loss': 0.0106, 'epoch': 1.65}

 82%|████████▏ | 1160/1410 [3:46:00<49:04, 11.78s/it]
 82%|████████▏ | 1161/1410 [3:46:12<49:17, 11.88s/it]
                                                     
{'loss': 0.0132, 'grad_norm': 1.054674506187439, 'learning_rate': 7.169026082476327e-08, 'kl': 0.0154, 'entropy': -0.0354, 'ce_loss': 0.0023, 'epoch': 1.65}

 82%|████████▏ | 1161/1410 [3:46:12<49:17, 11.88s/it]
 82%|████████▏ | 1162/1410 [3:46:24<49:09, 11.89s/it]
                                                     
{'loss': 0.0212, 'grad_norm': 1.8987184762954712, 'learning_rate': 7.113123482573517e-08, 'kl': 0.0293, 'entropy': -0.0859, 'ce_loss': 0.0327, 'epoch': 1.65}

 82%|████████▏ | 1162/1410 [3:46:24<49:09, 11.89s/it]
 82%|████████▏ | 1163/1410 [3:46:37<49:22, 11.99s/it]
                                                     
{'loss': 0.0274, 'grad_norm': 1.3119032382965088, 'learning_rate': 7.057420984623815e-08, 'kl': 0.0225, 'entropy': -0.0294, 'ce_loss': 0.0015, 'epoch': 1.65}

 82%|████████▏ | 1163/1410 [3:46:37<49:22, 11.99s/it]
 83%|████████▎ | 1164/1410 [3:46:49<49:20, 12.03s/it]
                                                     
{'loss': 0.0205, 'grad_norm': 1.5592608451843262, 'learning_rate': 7.001918882823492e-08, 'kl': 0.0361, 'entropy': -0.0918, 'ce_loss': 0.0062, 'epoch': 1.65}

 83%|████████▎ | 1164/1410 [3:46:49<49:20, 12.03s/it]
 83%|████████▎ | 1165/1410 [3:47:01<49:10, 12.04s/it]
                                                     
{'loss': 0.0314, 'grad_norm': 0.8759779334068298, 'learning_rate': 6.946617470310468e-08, 'kl': 0.027, 'entropy': -0.0889, 'ce_loss': 0.0067, 'epoch': 1.65}

 83%|████████▎ | 1165/1410 [3:47:01<49:10, 12.04s/it]
 83%|████████▎ | 1166/1410 [3:47:12<48:28, 11.92s/it]
                                                     
{'loss': 0.0083, 'grad_norm': 1.029316782951355, 'learning_rate': 6.891517039162664e-08, 'kl': -0.0037, 'entropy': 0.019, 'ce_loss': 0.065, 'epoch': 1.65}

 83%|████████▎ | 1166/1410 [3:47:12<48:28, 11.92s/it]
 83%|████████▎ | 1167/1410 [3:47:24<48:07, 11.88s/it]
                                                     
{'loss': 0.0091, 'grad_norm': 1.1374099254608154, 'learning_rate': 6.836617880396521e-08, 'kl': 0.0023, 'entropy': -0.0144, 'ce_loss': 0.002, 'epoch': 1.66}

 83%|████████▎ | 1167/1410 [3:47:24<48:07, 11.88s/it]
 83%|████████▎ | 1168/1410 [3:47:36<47:50, 11.86s/it]
                                                     
{'loss': 0.0172, 'grad_norm': 1.0625026226043701, 'learning_rate': 6.781920283965441e-08, 'kl': 0.0311, 'entropy': -0.0251, 'ce_loss': 0.0735, 'epoch': 1.66}

 83%|████████▎ | 1168/1410 [3:47:36<47:50, 11.86s/it]
 83%|████████▎ | 1169/1410 [3:47:48<48:05, 11.97s/it]
                                                     
{'loss': 0.0152, 'grad_norm': 1.2501038312911987, 'learning_rate': 6.727424538758258e-08, 'kl': -0.0025, 'entropy': 0.0383, 'ce_loss': 0.0886, 'epoch': 1.66}

 83%|████████▎ | 1169/1410 [3:47:48<48:05, 11.97s/it]
 83%|████████▎ | 1170/1410 [3:48:00<47:54, 11.98s/it]
                                                     
{'loss': 0.0286, 'grad_norm': 1.114723563194275, 'learning_rate': 6.67313093259774e-08, 'kl': 0.0002, 'entropy': -0.0232, 'ce_loss': 0.0006, 'epoch': 1.66}

 83%|████████▎ | 1170/1410 [3:48:00<47:54, 11.98s/it]
 83%|████████▎ | 1171/1410 [3:48:12<47:42, 11.98s/it]
                                                     
{'loss': 0.027, 'grad_norm': 1.2427277565002441, 'learning_rate': 6.619039752239023e-08, 'kl': 0.0295, 'entropy': -0.0203, 'ce_loss': 0.0665, 'epoch': 1.66}

 83%|████████▎ | 1171/1410 [3:48:12<47:42, 11.98s/it]
 83%|████████▎ | 1172/1410 [3:48:24<47:40, 12.02s/it]
                                                     
{'loss': 0.018, 'grad_norm': 1.0295885801315308, 'learning_rate': 6.56515128336811e-08, 'kl': -0.0001, 'entropy': -0.0129, 'ce_loss': 0.0003, 'epoch': 1.66}

 83%|████████▎ | 1172/1410 [3:48:24<47:40, 12.02s/it]
 83%|████████▎ | 1173/1410 [3:48:36<47:27, 12.01s/it]
                                                     
{'loss': 0.0046, 'grad_norm': 0.5069164633750916, 'learning_rate': 6.511465810600411e-08, 'kl': 0.0262, 'entropy': -0.0854, 'ce_loss': 0.0123, 'epoch': 1.66}

 83%|████████▎ | 1173/1410 [3:48:36<47:27, 12.01s/it]
 83%|████████▎ | 1174/1410 [3:48:48<47:11, 12.00s/it]
                                                     
{'loss': 0.0161, 'grad_norm': 1.3052181005477905, 'learning_rate': 6.457983617479156e-08, 'kl': -0.0059, 'entropy': -0.0157, 'ce_loss': 0.0962, 'epoch': 1.67}

 83%|████████▎ | 1174/1410 [3:48:48<47:11, 12.00s/it]
 83%|████████▎ | 1175/1410 [3:49:00<47:02, 12.01s/it]
                                                     
{'loss': 0.0314, 'grad_norm': 1.0255171060562134, 'learning_rate': 6.404704986473945e-08, 'kl': 0.0111, 'entropy': -0.0308, 'ce_loss': 0.0062, 'epoch': 1.67}

 83%|████████▎ | 1175/1410 [3:49:00<47:02, 12.01s/it]
 83%|████████▎ | 1176/1410 [3:49:13<47:40, 12.22s/it]
                                                     
{'loss': 0.0221, 'grad_norm': 0.9249253869056702, 'learning_rate': 6.351630198979301e-08, 'kl': 0.0182, 'entropy': -0.0693, 'ce_loss': 0.0041, 'epoch': 1.67}

 83%|████████▎ | 1176/1410 [3:49:13<47:40, 12.22s/it]
 83%|████████▎ | 1177/1410 [3:49:25<47:36, 12.26s/it]
                                                     
{'loss': 0.0303, 'grad_norm': 1.1085631847381592, 'learning_rate': 6.298759535313052e-08, 'kl': -0.0081, 'entropy': -0.0664, 'ce_loss': 0.0565, 'epoch': 1.67}

 83%|████████▎ | 1177/1410 [3:49:25<47:36, 12.26s/it]
 84%|████████▎ | 1178/1410 [3:49:37<46:56, 12.14s/it]
                                                     
{'loss': 0.0115, 'grad_norm': 1.3053863048553467, 'learning_rate': 6.246093274714993e-08, 'kl': 0.0156, 'entropy': -0.0674, 'ce_loss': 0.0174, 'epoch': 1.67}

 84%|████████▎ | 1178/1410 [3:49:37<46:56, 12.14s/it]
 84%|████████▎ | 1179/1410 [3:49:50<46:58, 12.20s/it]
                                                     
{'loss': 0.0113, 'grad_norm': 1.1059846878051758, 'learning_rate': 6.193631695345366e-08, 'kl': 0.0275, 'entropy': -0.1138, 'ce_loss': 0.0159, 'epoch': 1.67}

 84%|████████▎ | 1179/1410 [3:49:50<46:58, 12.20s/it]
 84%|████████▎ | 1180/1410 [3:50:02<47:01, 12.27s/it]
                                                     
{'loss': 0.0094, 'grad_norm': 1.140214204788208, 'learning_rate': 6.1413750742833e-08, 'kl': -0.0005, 'entropy': -0.0266, 'ce_loss': 0.0449, 'epoch': 1.67}

 84%|████████▎ | 1180/1410 [3:50:02<47:01, 12.27s/it]
 84%|████████▍ | 1181/1410 [3:50:14<46:18, 12.14s/it]
                                                     
{'loss': 0.0156, 'grad_norm': 1.2803568840026855, 'learning_rate': 6.089323687525505e-08, 'kl': 0.0261, 'entropy': -0.0781, 'ce_loss': 0.0141, 'epoch': 1.68}

 84%|████████▍ | 1181/1410 [3:50:14<46:18, 12.14s/it]
 84%|████████▍ | 1182/1410 [3:50:26<46:18, 12.19s/it]
                                                     
{'loss': 0.0212, 'grad_norm': 0.8729901909828186, 'learning_rate': 6.037477809984679e-08, 'kl': -0.0077, 'entropy': -0.0334, 'ce_loss': 0.0388, 'epoch': 1.68}

 84%|████████▍ | 1182/1410 [3:50:26<46:18, 12.19s/it]
 84%|████████▍ | 1183/1410 [3:50:38<45:31, 12.03s/it]
                                                     
{'loss': 0.0142, 'grad_norm': 1.1481738090515137, 'learning_rate': 5.985837715488128e-08, 'kl': 0.0195, 'entropy': -0.0427, 'ce_loss': 0.0029, 'epoch': 1.68}

 84%|████████▍ | 1183/1410 [3:50:38<45:31, 12.03s/it]
 84%|████████▍ | 1184/1410 [3:50:50<45:19, 12.03s/it]
                                                     
{'loss': 0.0227, 'grad_norm': 1.1365046501159668, 'learning_rate': 5.934403676776317e-08, 'kl': 0.0016, 'entropy': -0.0288, 'ce_loss': 0.0012, 'epoch': 1.68}

 84%|████████▍ | 1184/1410 [3:50:50<45:19, 12.03s/it]
 84%|████████▍ | 1185/1410 [3:51:02<44:51, 11.96s/it]
                                                     
{'loss': 0.0174, 'grad_norm': 1.1667660474777222, 'learning_rate': 5.883175965501391e-08, 'kl': -0.0081, 'entropy': -0.0101, 'ce_loss': 0.1089, 'epoch': 1.68}

 84%|████████▍ | 1185/1410 [3:51:02<44:51, 11.96s/it]
 84%|████████▍ | 1186/1410 [3:51:13<44:19, 11.87s/it]
                                                     
{'loss': 0.0076, 'grad_norm': 1.0512034893035889, 'learning_rate': 5.832154852225764e-08, 'kl': -0.0004, 'entropy': -0.0234, 'ce_loss': 0.0005, 'epoch': 1.68}

 84%|████████▍ | 1186/1410 [3:51:13<44:19, 11.87s/it]
 84%|████████▍ | 1187/1410 [3:51:25<43:39, 11.75s/it]
                                                     
{'loss': 0.0207, 'grad_norm': 1.5971040725708008, 'learning_rate': 5.7813406064207155e-08, 'kl': -0.0096, 'entropy': 0.0025, 'ce_loss': 0.0809, 'epoch': 1.68}

 84%|████████▍ | 1187/1410 [3:51:25<43:39, 11.75s/it]
 84%|████████▍ | 1188/1410 [3:51:37<43:34, 11.78s/it]
                                                     
{'loss': 0.0061, 'grad_norm': 0.8092554211616516, 'learning_rate': 5.7307334964649096e-08, 'kl': 0.0192, 'entropy': -0.084, 'ce_loss': 0.0118, 'epoch': 1.69}

 84%|████████▍ | 1188/1410 [3:51:37<43:34, 11.78s/it]
 84%|████████▍ | 1189/1410 [3:51:49<43:57, 11.93s/it]
                                                     
{'loss': 0.0185, 'grad_norm': 1.1359943151474, 'learning_rate': 5.680333789643021e-08, 'kl': -0.0001, 'entropy': -0.062, 'ce_loss': 0.0039, 'epoch': 1.69}

 84%|████████▍ | 1189/1410 [3:51:49<43:57, 11.93s/it]
 84%|████████▍ | 1190/1410 [3:52:01<43:38, 11.90s/it]
                                                     
{'loss': 0.0214, 'grad_norm': 1.3101754188537598, 'learning_rate': 5.63014175214434e-08, 'kl': -0.0054, 'entropy': -0.0017, 'ce_loss': 0.0543, 'epoch': 1.69}

 84%|████████▍ | 1190/1410 [3:52:01<43:38, 11.90s/it]
 84%|████████▍ | 1191/1410 [3:52:12<43:12, 11.84s/it]
                                                     
{'loss': 0.0149, 'grad_norm': 1.2116109132766724, 'learning_rate': 5.58015764906127e-08, 'kl': 0.0347, 'entropy': -0.05, 'ce_loss': 0.0046, 'epoch': 1.69}

 84%|████████▍ | 1191/1410 [3:52:12<43:12, 11.84s/it]
 85%|████████▍ | 1192/1410 [3:52:24<42:58, 11.83s/it]
                                                     
{'loss': 0.0082, 'grad_norm': 0.8081415891647339, 'learning_rate': 5.5303817443880576e-08, 'kl': 0.022, 'entropy': -0.0654, 'ce_loss': 0.0194, 'epoch': 1.69}

 85%|████████▍ | 1192/1410 [3:52:24<42:58, 11.83s/it]
 85%|████████▍ | 1193/1410 [3:52:36<42:21, 11.71s/it]
                                                     
{'loss': 0.0128, 'grad_norm': 1.107349157333374, 'learning_rate': 5.480814301019338e-08, 'kl': 0.0294, 'entropy': -0.0894, 'ce_loss': 0.009, 'epoch': 1.69}

 85%|████████▍ | 1193/1410 [3:52:36<42:21, 11.71s/it]
 85%|████████▍ | 1194/1410 [3:52:48<42:21, 11.76s/it]
                                                     
{'loss': 0.0171, 'grad_norm': 1.4599676132202148, 'learning_rate': 5.431455580748684e-08, 'kl': 0.0309, 'entropy': -0.0767, 'ce_loss': 0.0354, 'epoch': 1.69}

 85%|████████▍ | 1194/1410 [3:52:48<42:21, 11.76s/it]
 85%|████████▍ | 1195/1410 [3:53:00<42:25, 11.84s/it]
                                                     
{'loss': 0.0088, 'grad_norm': 0.9787486791610718, 'learning_rate': 5.3823058442673484e-08, 'kl': 0.0081, 'entropy': -0.0151, 'ce_loss': 0.0001, 'epoch': 1.7}

 85%|████████▍ | 1195/1410 [3:53:00<42:25, 11.84s/it]
 85%|████████▍ | 1196/1410 [3:53:11<42:00, 11.78s/it]
                                                     
{'loss': 0.0104, 'grad_norm': 1.0637410879135132, 'learning_rate': 5.3333653511627936e-08, 'kl': -0.0048, 'entropy': 0.1738, 'ce_loss': 0.1994, 'epoch': 1.7}

 85%|████████▍ | 1196/1410 [3:53:11<42:00, 11.78s/it]
 85%|████████▍ | 1197/1410 [3:53:23<42:00, 11.83s/it]
                                                     
{'loss': 0.0105, 'grad_norm': 0.8671389222145081, 'learning_rate': 5.284634359917334e-08, 'kl': -0.0001, 'entropy': 0.0292, 'ce_loss': 0.0848, 'epoch': 1.7}

 85%|████████▍ | 1197/1410 [3:53:23<42:00, 11.83s/it]
 85%|████████▍ | 1198/1410 [3:53:36<42:38, 12.07s/it]
                                                     
{'loss': 0.0181, 'grad_norm': 0.8301557302474976, 'learning_rate': 5.236113127906832e-08, 'kl': -0.0079, 'entropy': -0.0148, 'ce_loss': 0.086, 'epoch': 1.7}

 85%|████████▍ | 1198/1410 [3:53:36<42:38, 12.07s/it]
 85%|████████▌ | 1199/1410 [3:53:47<41:54, 11.92s/it]
                                                     
{'loss': 0.0302, 'grad_norm': 1.5736465454101562, 'learning_rate': 5.187801911399252e-08, 'kl': 0.0054, 'entropy': -0.0302, 'ce_loss': 0.0019, 'epoch': 1.7}

 85%|████████▌ | 1199/1410 [3:53:47<41:54, 11.92s/it]
 85%|████████▌ | 1200/1410 [3:53:58<40:34, 11.59s/it]
                                                     
{'loss': 0.0103, 'grad_norm': 1.05115807056427, 'learning_rate': 5.13970096555336e-08, 'kl': 0.0186, 'entropy': -0.0354, 'ce_loss': 0.0045, 'epoch': 1.7}

 85%|████████▌ | 1200/1410 [3:53:58<40:34, 11.59s/it]
 85%|████████▌ | 1201/1410 [3:54:10<40:11, 11.54s/it]
                                                     
{'loss': 0.021, 'grad_norm': 1.4608416557312012, 'learning_rate': 5.091810544417383e-08, 'kl': -0.0002, 'entropy': -0.0242, 'ce_loss': 0.0005, 'epoch': 1.7}

 85%|████████▌ | 1201/1410 [3:54:10<40:11, 11.54s/it]
 85%|████████▌ | 1202/1410 [3:54:22<40:39, 11.73s/it]
                                                     
{'loss': 0.0309, 'grad_norm': 0.9850816130638123, 'learning_rate': 5.044130900927628e-08, 'kl': -0.0071, 'entropy': 0.0197, 'ce_loss': 0.1237, 'epoch': 1.7}

 85%|████████▌ | 1202/1410 [3:54:22<40:39, 11.73s/it]
 85%|████████▌ | 1203/1410 [3:54:34<40:30, 11.74s/it]
                                                     
{'loss': 0.012, 'grad_norm': 1.1738905906677246, 'learning_rate': 4.996662286907162e-08, 'kl': 0.0079, 'entropy': -0.033, 'ce_loss': 0.0058, 'epoch': 1.71}

 85%|████████▌ | 1203/1410 [3:54:34<40:30, 11.74s/it]
 85%|████████▌ | 1204/1410 [3:54:45<39:53, 11.62s/it]
                                                     
{'loss': 0.0171, 'grad_norm': 0.9058971405029297, 'learning_rate': 4.9494049530644885e-08, 'kl': 0.0026, 'entropy': 0.1748, 'ce_loss': 0.2217, 'epoch': 1.71}

 85%|████████▌ | 1204/1410 [3:54:45<39:53, 11.62s/it]
 85%|████████▌ | 1205/1410 [3:54:56<39:07, 11.45s/it]
                                                     
{'loss': 0.0087, 'grad_norm': 1.0411734580993652, 'learning_rate': 4.902359148992251e-08, 'kl': 0.0325, 'entropy': -0.0737, 'ce_loss': 0.0073, 'epoch': 1.71}

 85%|████████▌ | 1205/1410 [3:54:56<39:07, 11.45s/it]
 86%|████████▌ | 1206/1410 [3:55:08<39:53, 11.74s/it]
                                                     
{'loss': 0.0282, 'grad_norm': 1.0263692140579224, 'learning_rate': 4.8555251231658526e-08, 'kl': -0.0125, 'entropy': -0.0248, 'ce_loss': 0.0921, 'epoch': 1.71}

 86%|████████▌ | 1206/1410 [3:55:08<39:53, 11.74s/it]
 86%|████████▌ | 1207/1410 [3:55:20<39:41, 11.73s/it]
                                                     
{'loss': 0.0091, 'grad_norm': 0.9889789819717407, 'learning_rate': 4.808903122942184e-08, 'kl': 0.0109, 'entropy': -0.0266, 'ce_loss': 0.0024, 'epoch': 1.71}

 86%|████████▌ | 1207/1410 [3:55:20<39:41, 11.73s/it]
 86%|████████▌ | 1208/1410 [3:55:31<39:02, 11.60s/it]
                                                     
{'loss': 0.0168, 'grad_norm': 1.3220468759536743, 'learning_rate': 4.7624933945583014e-08, 'kl': 0.0048, 'entropy': -0.0469, 'ce_loss': 0.0022, 'epoch': 1.71}

 86%|████████▌ | 1208/1410 [3:55:31<39:02, 11.60s/it]
 86%|████████▌ | 1209/1410 [3:55:43<39:12, 11.70s/it]
                                                     
{'loss': 0.0085, 'grad_norm': 1.0767688751220703, 'learning_rate': 4.716296183130162e-08, 'kl': -0.0065, 'entropy': 0.0005, 'ce_loss': 0.0889, 'epoch': 1.71}

 86%|████████▌ | 1209/1410 [3:55:43<39:12, 11.70s/it]
 86%|████████▌ | 1210/1410 [3:55:56<39:43, 11.92s/it]
                                                     
{'loss': 0.0118, 'grad_norm': 1.1106853485107422, 'learning_rate': 4.670311732651271e-08, 'kl': -0.0097, 'entropy': -0.0354, 'ce_loss': 0.0307, 'epoch': 1.72}

 86%|████████▌ | 1210/1410 [3:55:56<39:43, 11.92s/it]
 86%|████████▌ | 1211/1410 [3:56:07<38:47, 11.70s/it]
                                                     
{'loss': 0.0149, 'grad_norm': 1.3216423988342285, 'learning_rate': 4.624540285991428e-08, 'kl': 0.0212, 'entropy': -0.0801, 'ce_loss': 0.0259, 'epoch': 1.72}

 86%|████████▌ | 1211/1410 [3:56:07<38:47, 11.70s/it]
 86%|████████▌ | 1212/1410 [3:56:19<38:53, 11.79s/it]
                                                     
{'loss': 0.0274, 'grad_norm': 1.4762372970581055, 'learning_rate': 4.578982084895467e-08, 'kl': 0.0292, 'entropy': -0.0903, 'ce_loss': 0.0122, 'epoch': 1.72}

 86%|████████▌ | 1212/1410 [3:56:19<38:53, 11.79s/it]
 86%|████████▌ | 1213/1410 [3:56:30<37:46, 11.51s/it]
                                                     
{'loss': 0.0065, 'grad_norm': 0.9320425987243652, 'learning_rate': 4.533637369981913e-08, 'kl': 0.0306, 'entropy': -0.0957, 'ce_loss': 0.0164, 'epoch': 1.72}

 86%|████████▌ | 1213/1410 [3:56:30<37:46, 11.51s/it]
 86%|████████▌ | 1214/1410 [3:56:42<38:01, 11.64s/it]
                                                     
{'loss': 0.0191, 'grad_norm': 1.4099289178848267, 'learning_rate': 4.4885063807417784e-08, 'kl': 0.0177, 'entropy': -0.0574, 'ce_loss': 0.0241, 'epoch': 1.72}

 86%|████████▌ | 1214/1410 [3:56:42<38:01, 11.64s/it]
 86%|████████▌ | 1215/1410 [3:56:54<37:57, 11.68s/it]
                                                     
{'loss': 0.0145, 'grad_norm': 0.7937688827514648, 'learning_rate': 4.443589355537243e-08, 'kl': 0.02, 'entropy': -0.0723, 'ce_loss': 0.0266, 'epoch': 1.72}

 86%|████████▌ | 1215/1410 [3:56:54<37:57, 11.68s/it]
 86%|████████▌ | 1216/1410 [3:57:05<37:56, 11.73s/it]
                                                     
{'loss': 0.0225, 'grad_norm': 1.3138387203216553, 'learning_rate': 4.3988865316004604e-08, 'kl': 0.0315, 'entropy': -0.0664, 'ce_loss': 0.0164, 'epoch': 1.72}

 86%|████████▌ | 1216/1410 [3:57:05<37:56, 11.73s/it]
 86%|████████▋ | 1217/1410 [3:57:17<37:58, 11.80s/it]
                                                     
{'loss': 0.008, 'grad_norm': 1.0031591653823853, 'learning_rate': 4.35439814503223e-08, 'kl': 0.0049, 'entropy': -0.019, 'ce_loss': 0.0035, 'epoch': 1.73}

 86%|████████▋ | 1217/1410 [3:57:17<37:58, 11.80s/it]
 86%|████████▋ | 1218/1410 [3:57:29<37:47, 11.81s/it]
                                                     
{'loss': 0.0047, 'grad_norm': 0.8352823257446289, 'learning_rate': 4.310124430800801e-08, 'kl': 0.0265, 'entropy': -0.1055, 'ce_loss': 0.0172, 'epoch': 1.73}

 86%|████████▋ | 1218/1410 [3:57:29<37:47, 11.81s/it]
 86%|████████▋ | 1219/1410 [3:57:41<37:32, 11.79s/it]
                                                     
{'loss': 0.0073, 'grad_norm': 1.5376285314559937, 'learning_rate': 4.266065622740626e-08, 'kl': 0.0322, 'entropy': -0.0947, 'ce_loss': 0.0139, 'epoch': 1.73}

 86%|████████▋ | 1219/1410 [3:57:41<37:32, 11.79s/it]
 87%|████████▋ | 1220/1410 [3:57:53<37:35, 11.87s/it]
                                                     
{'loss': 0.0073, 'grad_norm': 0.7987399101257324, 'learning_rate': 4.2222219535511e-08, 'kl': 0.0157, 'entropy': -0.0693, 'ce_loss': 0.0186, 'epoch': 1.73}

 87%|████████▋ | 1220/1410 [3:57:53<37:35, 11.87s/it]
 87%|████████▋ | 1221/1410 [3:58:05<37:17, 11.84s/it]
                                                     
{'loss': 0.0085, 'grad_norm': 1.066060185432434, 'learning_rate': 4.178593654795365e-08, 'kl': 0.0102, 'entropy': -0.0493, 'ce_loss': 0.0067, 'epoch': 1.73}

 87%|████████▋ | 1221/1410 [3:58:05<37:17, 11.84s/it]
 87%|████████▋ | 1222/1410 [3:58:17<37:10, 11.87s/it]
                                                     
{'loss': 0.0191, 'grad_norm': 1.1178227663040161, 'learning_rate': 4.135180956899051e-08, 'kl': 0.0146, 'entropy': -0.0581, 'ce_loss': 0.0131, 'epoch': 1.73}

 87%|████████▋ | 1222/1410 [3:58:17<37:10, 11.87s/it]
 87%|████████▋ | 1223/1410 [3:58:28<36:37, 11.75s/it]
                                                     
{'loss': 0.014, 'grad_norm': 0.9797238707542419, 'learning_rate': 4.091984089149087e-08, 'kl': 0.0022, 'entropy': -0.0056, 'ce_loss': 0.0019, 'epoch': 1.73}

 87%|████████▋ | 1223/1410 [3:58:28<36:37, 11.75s/it]
 87%|████████▋ | 1224/1410 [3:58:40<36:06, 11.65s/it]
                                                     
{'loss': 0.0095, 'grad_norm': 1.161865472793579, 'learning_rate': 4.0490032796924944e-08, 'kl': 0.0425, 'entropy': -0.1118, 'ce_loss': 0.0157, 'epoch': 1.74}

 87%|████████▋ | 1224/1410 [3:58:40<36:06, 11.65s/it]
 87%|████████▋ | 1225/1410 [3:58:50<35:01, 11.36s/it]
                                                     
{'loss': 0.0128, 'grad_norm': 1.2767016887664795, 'learning_rate': 4.0062387555351526e-08, 'kl': 0.0354, 'entropy': -0.0811, 'ce_loss': 0.023, 'epoch': 1.74}

 87%|████████▋ | 1225/1410 [3:58:50<35:01, 11.36s/it]
 87%|████████▋ | 1226/1410 [3:59:02<35:12, 11.48s/it]
                                                     
{'loss': 0.0098, 'grad_norm': 1.1438755989074707, 'learning_rate': 3.9636907425406184e-08, 'kl': -0.0043, 'entropy': -0.014, 'ce_loss': 0.0945, 'epoch': 1.74}

 87%|████████▋ | 1226/1410 [3:59:02<35:12, 11.48s/it]
 87%|████████▋ | 1227/1410 [3:59:13<34:40, 11.37s/it]
                                                     
{'loss': 0.0129, 'grad_norm': 1.2048448324203491, 'learning_rate': 3.921359465428941e-08, 'kl': 0.0403, 'entropy': -0.0962, 'ce_loss': 0.0074, 'epoch': 1.74}

 87%|████████▋ | 1227/1410 [3:59:13<34:40, 11.37s/it]
 87%|████████▋ | 1228/1410 [3:59:25<34:58, 11.53s/it]
                                                     
{'loss': 0.0317, 'grad_norm': 1.601261019706726, 'learning_rate': 3.8792451477754616e-08, 'kl': 0.0003, 'entropy': 0.0027, 'ce_loss': 0.0669, 'epoch': 1.74}

 87%|████████▋ | 1228/1410 [3:59:25<34:58, 11.53s/it]
 87%|████████▋ | 1229/1410 [3:59:37<34:57, 11.59s/it]
                                                     
{'loss': 0.0087, 'grad_norm': 0.9783957600593567, 'learning_rate': 3.837348012009612e-08, 'kl': 0.0549, 'entropy': -0.0991, 'ce_loss': 0.0055, 'epoch': 1.74}

 87%|████████▋ | 1229/1410 [3:59:37<34:57, 11.59s/it]
 87%|████████▋ | 1230/1410 [3:59:49<35:28, 11.82s/it]
                                                     
{'loss': 0.0128, 'grad_norm': 0.9979757070541382, 'learning_rate': 3.7956682794138186e-08, 'kl': 0.0176, 'entropy': -0.0435, 'ce_loss': 0.0034, 'epoch': 1.74}

 87%|████████▋ | 1230/1410 [3:59:49<35:28, 11.82s/it]
 87%|████████▋ | 1231/1410 [4:00:01<35:08, 11.78s/it]
                                                     
{'loss': 0.0204, 'grad_norm': 1.1500730514526367, 'learning_rate': 3.7542061701222124e-08, 'kl': 0.041, 'entropy': -0.0923, 'ce_loss': 0.0672, 'epoch': 1.75}

 87%|████████▋ | 1231/1410 [4:00:01<35:08, 11.78s/it]
 87%|████████▋ | 1232/1410 [4:00:13<35:02, 11.81s/it]
                                                     
{'loss': 0.0191, 'grad_norm': 0.9949401021003723, 'learning_rate': 3.7129619031195783e-08, 'kl': 0.0522, 'entropy': -0.0422, 'ce_loss': 0.0025, 'epoch': 1.75}

 87%|████████▋ | 1232/1410 [4:00:13<35:02, 11.81s/it]
 87%|████████▋ | 1233/1410 [4:00:25<34:53, 11.83s/it]
                                                     
{'loss': 0.0175, 'grad_norm': 1.246429443359375, 'learning_rate': 3.6719356962401634e-08, 'kl': 0.0154, 'entropy': -0.0249, 'ce_loss': 0.0026, 'epoch': 1.75}

 87%|████████▋ | 1233/1410 [4:00:25<34:53, 11.83s/it]
 88%|████████▊ | 1234/1410 [4:00:36<34:35, 11.79s/it]
                                                     
{'loss': 0.0087, 'grad_norm': 0.8372634053230286, 'learning_rate': 3.631127766166453e-08, 'kl': -0.0014, 'entropy': -0.054, 'ce_loss': 0.0308, 'epoch': 1.75}

 88%|████████▊ | 1234/1410 [4:00:36<34:35, 11.79s/it]
 88%|████████▊ | 1235/1410 [4:00:48<34:32, 11.84s/it]
                                                     
{'loss': 0.0221, 'grad_norm': 1.3899493217468262, 'learning_rate': 3.590538328428157e-08, 'kl': 0.0058, 'entropy': 0.0105, 'ce_loss': 0.0194, 'epoch': 1.75}

 88%|████████▊ | 1235/1410 [4:00:48<34:32, 11.84s/it]
 88%|████████▊ | 1236/1410 [4:01:00<34:27, 11.88s/it]
                                                     
{'loss': 0.0466, 'grad_norm': 1.311062216758728, 'learning_rate': 3.550167597400988e-08, 'kl': -0.0009, 'entropy': -0.0074, 'ce_loss': 0.0005, 'epoch': 1.75}

 88%|████████▊ | 1236/1410 [4:01:00<34:27, 11.88s/it]
 88%|████████▊ | 1237/1410 [4:01:12<34:18, 11.90s/it]
                                                     
{'loss': 0.0257, 'grad_norm': 1.547267198562622, 'learning_rate': 3.510015786305501e-08, 'kl': 0.0012, 'entropy': -0.0197, 'ce_loss': 0.0009, 'epoch': 1.75}

 88%|████████▊ | 1237/1410 [4:01:12<34:18, 11.90s/it]
 88%|████████▊ | 1238/1410 [4:01:24<33:59, 11.86s/it]
                                                     
{'loss': 0.0051, 'grad_norm': 0.612858235836029, 'learning_rate': 3.4700831072060775e-08, 'kl': -0.0054, 'entropy': 0.0211, 'ce_loss': 0.0688, 'epoch': 1.76}

 88%|████████▊ | 1238/1410 [4:01:24<33:59, 11.86s/it]
 88%|████████▊ | 1239/1410 [4:01:36<33:50, 11.87s/it]
                                                     
{'loss': 0.0101, 'grad_norm': 1.0736407041549683, 'learning_rate': 3.43036977100969e-08, 'kl': 0.0342, 'entropy': -0.0801, 'ce_loss': 0.0116, 'epoch': 1.76}

 88%|████████▊ | 1239/1410 [4:01:36<33:50, 11.87s/it]
 88%|████████▊ | 1240/1410 [4:01:48<33:31, 11.83s/it]
                                                     
{'loss': 0.0096, 'grad_norm': 1.8277345895767212, 'learning_rate': 3.390875987464852e-08, 'kl': 0.0028, 'entropy': -0.0271, 'ce_loss': 0.0043, 'epoch': 1.76}

 88%|████████▊ | 1240/1410 [4:01:48<33:31, 11.83s/it]
 88%|████████▊ | 1241/1410 [4:01:59<33:15, 11.81s/it]
                                                     
{'loss': 0.0097, 'grad_norm': 1.1628860235214233, 'learning_rate': 3.3516019651605185e-08, 'kl': 0.0175, 'entropy': -0.0148, 'ce_loss': 0.0025, 'epoch': 1.76}

 88%|████████▊ | 1241/1410 [4:01:59<33:15, 11.81s/it]
 88%|████████▊ | 1242/1410 [4:02:11<33:10, 11.85s/it]
                                                     
{'loss': 0.0199, 'grad_norm': 1.463987112045288, 'learning_rate': 3.3125479115249206e-08, 'kl': -0.0084, 'entropy': -0.0757, 'ce_loss': 0.0296, 'epoch': 1.76}

 88%|████████▊ | 1242/1410 [4:02:11<33:10, 11.85s/it]
 88%|████████▊ | 1243/1410 [4:02:23<32:59, 11.85s/it]
                                                     
{'loss': 0.0241, 'grad_norm': 1.1013222932815552, 'learning_rate': 3.273714032824541e-08, 'kl': -0.0062, 'entropy': -0.0053, 'ce_loss': 0.0866, 'epoch': 1.76}

 88%|████████▊ | 1243/1410 [4:02:23<32:59, 11.85s/it]
 88%|████████▊ | 1244/1410 [4:02:35<33:03, 11.95s/it]
                                                     
{'loss': 0.0274, 'grad_norm': 1.1760209798812866, 'learning_rate': 3.235100534162994e-08, 'kl': 0.0312, 'entropy': -0.0923, 'ce_loss': 0.0124, 'epoch': 1.76}

 88%|████████▊ | 1244/1410 [4:02:35<33:03, 11.95s/it]
 88%|████████▊ | 1245/1410 [4:02:47<32:42, 11.89s/it]
                                                     
{'loss': 0.0062, 'grad_norm': 0.9781205058097839, 'learning_rate': 3.196707619479906e-08, 'kl': 0.0042, 'entropy': -0.021, 'ce_loss': 0.0053, 'epoch': 1.77}

 88%|████████▊ | 1245/1410 [4:02:47<32:42, 11.89s/it]
 88%|████████▊ | 1246/1410 [4:02:59<32:20, 11.83s/it]
                                                     
{'loss': 0.012, 'grad_norm': 1.0381953716278076, 'learning_rate': 3.158535491549927e-08, 'kl': 0.0082, 'entropy': -0.0178, 'ce_loss': 0.0001, 'epoch': 1.77}

 88%|████████▊ | 1246/1410 [4:02:59<32:20, 11.83s/it]
 88%|████████▊ | 1247/1410 [4:03:11<32:06, 11.82s/it]
                                                     
{'loss': 0.0148, 'grad_norm': 1.1844068765640259, 'learning_rate': 3.120584351981589e-08, 'kl': -0.0001, 'entropy': -0.0151, 'ce_loss': 0.0001, 'epoch': 1.77}

 88%|████████▊ | 1247/1410 [4:03:11<32:06, 11.82s/it]
 89%|████████▊ | 1248/1410 [4:03:22<31:49, 11.79s/it]
                                                     
{'loss': 0.0087, 'grad_norm': 1.1131277084350586, 'learning_rate': 3.0828544012162286e-08, 'kl': 0.0132, 'entropy': -0.0289, 'ce_loss': 0.0013, 'epoch': 1.77}

 89%|████████▊ | 1248/1410 [4:03:22<31:49, 11.79s/it]
 89%|████████▊ | 1249/1410 [4:03:33<31:09, 11.61s/it]
                                                     
{'loss': 0.013, 'grad_norm': 1.227138876914978, 'learning_rate': 3.045345838527005e-08, 'kl': 0.0005, 'entropy': -0.0092, 'ce_loss': 0.0002, 'epoch': 1.77}

 89%|████████▊ | 1249/1410 [4:03:33<31:09, 11.61s/it]
 89%|████████▊ | 1250/1410 [4:03:45<31:11, 11.70s/it]
                                                     
{'loss': 0.0203, 'grad_norm': 1.322933554649353, 'learning_rate': 3.0080588620177815e-08, 'kl': 0.0087, 'entropy': -0.0219, 'ce_loss': 0.004, 'epoch': 1.77}

 89%|████████▊ | 1250/1410 [4:03:45<31:11, 11.70s/it]
 89%|████████▊ | 1251/1410 [4:03:57<31:06, 11.74s/it]
                                                     
{'loss': 0.0079, 'grad_norm': 1.0402247905731201, 'learning_rate': 2.9709936686220872e-08, 'kl': -0.0111, 'entropy': 0.0347, 'ce_loss': 0.1271, 'epoch': 1.77}

 89%|████████▊ | 1251/1410 [4:03:57<31:06, 11.74s/it]
 89%|████████▉ | 1252/1410 [4:04:09<30:56, 11.75s/it]
                                                     
{'loss': 0.0089, 'grad_norm': 1.3714884519577026, 'learning_rate': 2.9341504541021235e-08, 'kl': -0.0086, 'entropy': -0.0234, 'ce_loss': 0.0516, 'epoch': 1.78}

 89%|████████▉ | 1252/1410 [4:04:09<30:56, 11.75s/it]
 89%|████████▉ | 1253/1410 [4:04:20<30:16, 11.57s/it]
                                                     
{'loss': 0.0104, 'grad_norm': 0.9411522746086121, 'learning_rate': 2.8975294130476645e-08, 'kl': 0.0009, 'entropy': -0.024, 'ce_loss': 0.0658, 'epoch': 1.78}

 89%|████████▉ | 1253/1410 [4:04:20<30:16, 11.57s/it]
 89%|████████▉ | 1254/1410 [4:04:32<30:23, 11.69s/it]
                                                     
{'loss': 0.0157, 'grad_norm': 1.4755699634552002, 'learning_rate': 2.8611307388750687e-08, 'kl': 0.05, 'entropy': -0.1245, 'ce_loss': 0.0095, 'epoch': 1.78}

 89%|████████▉ | 1254/1410 [4:04:32<30:23, 11.69s/it]
 89%|████████▉ | 1255/1410 [4:04:44<30:12, 11.70s/it]
                                                     
{'loss': 0.009, 'grad_norm': 0.9848698377609253, 'learning_rate': 2.8249546238262635e-08, 'kl': -0.0125, 'entropy': -0.0037, 'ce_loss': 0.0867, 'epoch': 1.78}

 89%|████████▉ | 1255/1410 [4:04:44<30:12, 11.70s/it]
 89%|████████▉ | 1256/1410 [4:04:55<29:59, 11.69s/it]
                                                     
{'loss': 0.0096, 'grad_norm': 1.1563606262207031, 'learning_rate': 2.789001258967701e-08, 'kl': 0.0317, 'entropy': -0.0889, 'ce_loss': 0.0118, 'epoch': 1.78}

 89%|████████▉ | 1256/1410 [4:04:55<29:59, 11.69s/it]
 89%|████████▉ | 1257/1410 [4:05:07<29:49, 11.70s/it]
                                                     
{'loss': 0.0153, 'grad_norm': 1.2196428775787354, 'learning_rate': 2.753270834189365e-08, 'kl': 0.0227, 'entropy': -0.0435, 'ce_loss': 0.0048, 'epoch': 1.78}

 89%|████████▉ | 1257/1410 [4:05:07<29:49, 11.70s/it]
 89%|████████▉ | 1258/1410 [4:05:19<29:36, 11.68s/it]
                                                     
{'loss': 0.0146, 'grad_norm': 1.2836451530456543, 'learning_rate': 2.7177635382037816e-08, 'kl': 0.0337, 'entropy': -0.0302, 'ce_loss': 0.003, 'epoch': 1.78}

 89%|████████▉ | 1258/1410 [4:05:19<29:36, 11.68s/it]
 89%|████████▉ | 1259/1410 [4:05:30<28:52, 11.47s/it]
                                                     
{'loss': 0.0277, 'grad_norm': 1.4132015705108643, 'learning_rate': 2.6824795585449937e-08, 'kl': 0.0317, 'entropy': -0.0869, 'ce_loss': 0.0078, 'epoch': 1.79}

 89%|████████▉ | 1259/1410 [4:05:30<28:52, 11.47s/it]
 89%|████████▉ | 1260/1410 [4:05:41<28:48, 11.52s/it]
                                                     
{'loss': 0.0103, 'grad_norm': 1.3770495653152466, 'learning_rate': 2.6474190815675884e-08, 'kl': -0.0007, 'entropy': 0.0203, 'ce_loss': 0.082, 'epoch': 1.79}

 89%|████████▉ | 1260/1410 [4:05:41<28:48, 11.52s/it]
 89%|████████▉ | 1261/1410 [4:05:53<28:47, 11.59s/it]
                                                     
{'loss': 0.0178, 'grad_norm': 1.3525177240371704, 'learning_rate': 2.6125822924457113e-08, 'kl': 0.0388, 'entropy': -0.0835, 'ce_loss': 0.0085, 'epoch': 1.79}

 89%|████████▉ | 1261/1410 [4:05:53<28:47, 11.59s/it]
 90%|████████▉ | 1262/1410 [4:06:05<28:48, 11.68s/it]
                                                     
{'loss': 0.0221, 'grad_norm': 0.9137647151947021, 'learning_rate': 2.577969375172069e-08, 'kl': 0.0137, 'entropy': -0.0469, 'ce_loss': 0.0018, 'epoch': 1.79}

 90%|████████▉ | 1262/1410 [4:06:05<28:48, 11.68s/it]
 90%|████████▉ | 1263/1410 [4:06:16<28:04, 11.46s/it]
                                                     
{'loss': 0.0099, 'grad_norm': 1.1469905376434326, 'learning_rate': 2.5435805125570124e-08, 'kl': 0.0124, 'entropy': -0.0488, 'ce_loss': 0.0026, 'epoch': 1.79}

 90%|████████▉ | 1263/1410 [4:06:16<28:04, 11.46s/it]
 90%|████████▉ | 1264/1410 [4:06:27<27:38, 11.36s/it]
                                                     
{'loss': 0.0099, 'grad_norm': 1.1976951360702515, 'learning_rate': 2.509415886227501e-08, 'kl': 0.0242, 'entropy': -0.1011, 'ce_loss': 0.0054, 'epoch': 1.79}

 90%|████████▉ | 1264/1410 [4:06:27<27:38, 11.36s/it]
 90%|████████▉ | 1265/1410 [4:06:39<27:35, 11.42s/it]
                                                     
{'loss': 0.0128, 'grad_norm': 1.4051584005355835, 'learning_rate': 2.475475676626185e-08, 'kl': 0.0352, 'entropy': -0.0845, 'ce_loss': 0.0079, 'epoch': 1.79}

 90%|████████▉ | 1265/1410 [4:06:39<27:35, 11.42s/it]
 90%|████████▉ | 1266/1410 [4:06:50<27:28, 11.45s/it]
                                                     
{'loss': 0.0069, 'grad_norm': 0.7914636135101318, 'learning_rate': 2.441760063010449e-08, 'kl': 0.002, 'entropy': -0.026, 'ce_loss': 0.0019, 'epoch': 1.8}

 90%|████████▉ | 1266/1410 [4:06:50<27:28, 11.45s/it]
 90%|████████▉ | 1267/1410 [4:07:01<26:58, 11.32s/it]
                                                     
{'loss': 0.0141, 'grad_norm': 1.321414589881897, 'learning_rate': 2.408269223451455e-08, 'kl': 0.0378, 'entropy': -0.0781, 'ce_loss': 0.0224, 'epoch': 1.8}

 90%|████████▉ | 1267/1410 [4:07:01<26:58, 11.32s/it]
 90%|████████▉ | 1268/1410 [4:07:13<27:04, 11.44s/it]
                                                     
{'loss': 0.0142, 'grad_norm': 1.0448782444000244, 'learning_rate': 2.375003334833201e-08, 'kl': 0.0312, 'entropy': -0.061, 'ce_loss': 0.0123, 'epoch': 1.8}

 90%|████████▉ | 1268/1410 [4:07:13<27:04, 11.44s/it]
 90%|█████████ | 1269/1410 [4:07:25<27:10, 11.56s/it]
                                                     
{'loss': 0.0189, 'grad_norm': 1.206379771232605, 'learning_rate': 2.3419625728515995e-08, 'kl': 0.0254, 'entropy': -0.0845, 'ce_loss': 0.0108, 'epoch': 1.8}

 90%|█████████ | 1269/1410 [4:07:25<27:10, 11.56s/it]
 90%|█████████ | 1270/1410 [4:07:37<27:13, 11.67s/it]
                                                     
{'loss': 0.0241, 'grad_norm': 1.0206190347671509, 'learning_rate': 2.3091471120135496e-08, 'kl': 0.0184, 'entropy': -0.0659, 'ce_loss': 0.0069, 'epoch': 1.8}

 90%|█████████ | 1270/1410 [4:07:37<27:13, 11.67s/it]
 90%|█████████ | 1271/1410 [4:07:48<27:03, 11.68s/it]
                                                     
{'loss': 0.0123, 'grad_norm': 1.2141735553741455, 'learning_rate': 2.276557125635987e-08, 'kl': 0.0427, 'entropy': -0.106, 'ce_loss': 0.0145, 'epoch': 1.8}

 90%|█████████ | 1271/1410 [4:07:48<27:03, 11.68s/it]
 90%|█████████ | 1272/1410 [4:08:00<26:47, 11.65s/it]
                                                     
{'loss': 0.0053, 'grad_norm': 0.6556879878044128, 'learning_rate': 2.2441927858450055e-08, 'kl': -0.003, 'entropy': -0.0581, 'ce_loss': 0.0255, 'epoch': 1.8}

 90%|█████████ | 1272/1410 [4:08:00<26:47, 11.65s/it]
 90%|█████████ | 1273/1410 [4:08:11<26:12, 11.48s/it]
                                                     
{'loss': 0.008, 'grad_norm': 0.9689332842826843, 'learning_rate': 2.2120542635749363e-08, 'kl': 0.0386, 'entropy': -0.1089, 'ce_loss': 0.0121, 'epoch': 1.81}

 90%|█████████ | 1273/1410 [4:08:11<26:12, 11.48s/it]
 90%|█████████ | 1274/1410 [4:08:22<25:54, 11.43s/it]
                                                     
{'loss': 0.0153, 'grad_norm': 1.2728352546691895, 'learning_rate': 2.18014172856743e-08, 'kl': 0.0194, 'entropy': -0.0513, 'ce_loss': 0.0019, 'epoch': 1.81}

 90%|█████████ | 1274/1410 [4:08:22<25:54, 11.43s/it]
 90%|█████████ | 1275/1410 [4:08:34<25:53, 11.51s/it]
                                                     
{'loss': 0.0091, 'grad_norm': 0.9925817251205444, 'learning_rate': 2.1484553493705677e-08, 'kl': 0.011, 'entropy': -0.0557, 'ce_loss': 0.0125, 'epoch': 1.81}

 90%|█████████ | 1275/1410 [4:08:34<25:53, 11.51s/it]
 90%|█████████ | 1276/1410 [4:08:46<25:51, 11.58s/it]
                                                     
{'loss': 0.008, 'grad_norm': 0.9785524606704712, 'learning_rate': 2.11699529333798e-08, 'kl': 0.0228, 'entropy': -0.0732, 'ce_loss': 0.0188, 'epoch': 1.81}

 90%|█████████ | 1276/1410 [4:08:46<25:51, 11.58s/it]
 91%|█████████ | 1277/1410 [4:08:58<25:56, 11.70s/it]
                                                     
{'loss': 0.016, 'grad_norm': 1.2421338558197021, 'learning_rate': 2.0857617266279687e-08, 'kl': 0.0352, 'entropy': -0.0884, 'ce_loss': 0.0069, 'epoch': 1.81}

 91%|█████████ | 1277/1410 [4:08:58<25:56, 11.70s/it]
 91%|█████████ | 1278/1410 [4:09:10<25:50, 11.74s/it]
                                                     
{'loss': 0.0101, 'grad_norm': 1.5123893022537231, 'learning_rate': 2.0547548142026032e-08, 'kl': 0.0049, 'entropy': -0.0109, 'ce_loss': 0.0001, 'epoch': 1.81}

 91%|█████████ | 1278/1410 [4:09:10<25:50, 11.74s/it]
 91%|█████████ | 1279/1410 [4:09:22<25:47, 11.81s/it]
                                                     
{'loss': 0.0165, 'grad_norm': 0.8720173835754395, 'learning_rate': 2.023974719826865e-08, 'kl': 0.0178, 'entropy': 0.0008, 'ce_loss': 0.039, 'epoch': 1.81}

 91%|█████████ | 1279/1410 [4:09:22<25:47, 11.81s/it]
 91%|█████████ | 1280/1410 [4:09:34<25:41, 11.86s/it]
                                                     
{'loss': 0.0258, 'grad_norm': 0.9950017333030701, 'learning_rate': 1.9934216060677795e-08, 'kl': -0.0017, 'entropy': -0.0227, 'ce_loss': 0.053, 'epoch': 1.82}

 91%|█████████ | 1280/1410 [4:09:34<25:41, 11.86s/it]
 91%|█████████ | 1281/1410 [4:09:45<25:08, 11.69s/it]
                                                     
{'loss': 0.0135, 'grad_norm': 1.2477537393569946, 'learning_rate': 1.9630956342935696e-08, 'kl': -0.0002, 'entropy': -0.0138, 'ce_loss': 0.0001, 'epoch': 1.82}

 91%|█████████ | 1281/1410 [4:09:45<25:08, 11.69s/it]
 91%|█████████ | 1282/1410 [4:09:56<24:43, 11.59s/it]
                                                     
{'loss': 0.0133, 'grad_norm': 1.2879024744033813, 'learning_rate': 1.932996964672794e-08, 'kl': 0.027, 'entropy': -0.0574, 'ce_loss': 0.0093, 'epoch': 1.82}

 91%|█████████ | 1282/1410 [4:09:56<24:43, 11.59s/it]
 91%|█████████ | 1283/1410 [4:10:08<24:32, 11.60s/it]
                                                     
{'loss': 0.01, 'grad_norm': 1.2846568822860718, 'learning_rate': 1.9031257561734816e-08, 'kl': 0.011, 'entropy': -0.0432, 'ce_loss': 0.0062, 'epoch': 1.82}

 91%|█████████ | 1283/1410 [4:10:08<24:32, 11.60s/it]
 91%|█████████ | 1284/1410 [4:10:20<24:26, 11.64s/it]
                                                     
{'loss': 0.0129, 'grad_norm': 1.119553565979004, 'learning_rate': 1.873482166562342e-08, 'kl': -0.0089, 'entropy': 0.0216, 'ce_loss': 0.1022, 'epoch': 1.82}

 91%|█████████ | 1284/1410 [4:10:20<24:26, 11.64s/it]
 91%|█████████ | 1285/1410 [4:10:31<24:22, 11.70s/it]
                                                     
{'loss': 0.0075, 'grad_norm': 0.9521317481994629, 'learning_rate': 1.8440663524038662e-08, 'kl': 0.0264, 'entropy': -0.0669, 'ce_loss': 0.0114, 'epoch': 1.82}

 91%|█████████ | 1285/1410 [4:10:31<24:22, 11.70s/it]
 91%|█████████ | 1286/1410 [4:10:43<24:20, 11.78s/it]
                                                     
{'loss': 0.0177, 'grad_norm': 0.8150120377540588, 'learning_rate': 1.8148784690595537e-08, 'kl': 0.0308, 'entropy': -0.085, 'ce_loss': 0.022, 'epoch': 1.82}

 91%|█████████ | 1286/1410 [4:10:43<24:20, 11.78s/it]
 91%|█████████▏| 1287/1410 [4:10:56<24:23, 11.90s/it]
                                                     
{'loss': 0.0313, 'grad_norm': 1.7799023389816284, 'learning_rate': 1.7859186706870856e-08, 'kl': -0.0034, 'entropy': 0.0029, 'ce_loss': 0.0888, 'epoch': 1.83}

 91%|█████████▏| 1287/1410 [4:10:56<24:23, 11.90s/it]
 91%|█████████▏| 1288/1410 [4:11:08<24:37, 12.11s/it]
                                                     
{'loss': 0.016, 'grad_norm': 1.4324833154678345, 'learning_rate': 1.757187110239462e-08, 'kl': 0.0432, 'entropy': -0.0898, 'ce_loss': 0.0064, 'epoch': 1.83}

 91%|█████████▏| 1288/1410 [4:11:08<24:37, 12.11s/it]
 91%|█████████▏| 1289/1410 [4:11:20<24:10, 11.99s/it]
                                                     
{'loss': 0.0075, 'grad_norm': 0.8205345273017883, 'learning_rate': 1.728683939464263e-08, 'kl': 0.0091, 'entropy': -0.0167, 'ce_loss': 0.0002, 'epoch': 1.83}

 91%|█████████▏| 1289/1410 [4:11:20<24:10, 11.99s/it]
 91%|█████████▏| 1290/1410 [4:11:32<23:50, 11.92s/it]
                                                     
{'loss': 0.0032, 'grad_norm': 0.5433408617973328, 'learning_rate': 1.700409308902807e-08, 'kl': 0.002, 'entropy': 0.0109, 'ce_loss': 0.0976, 'epoch': 1.83}

 91%|█████████▏| 1290/1410 [4:11:32<23:50, 11.92s/it]
 92%|█████████▏| 1291/1410 [4:11:43<23:31, 11.86s/it]
                                                     
{'loss': 0.0146, 'grad_norm': 1.2990856170654297, 'learning_rate': 1.67236336788934e-08, 'kl': 0.0322, 'entropy': -0.1021, 'ce_loss': 0.0288, 'epoch': 1.83}

 92%|█████████▏| 1291/1410 [4:11:43<23:31, 11.86s/it]
 92%|█████████▏| 1292/1410 [4:11:55<23:18, 11.85s/it]
                                                     
{'loss': 0.0129, 'grad_norm': 1.1441468000411987, 'learning_rate': 1.6445462645503028e-08, 'kl': -0.0129, 'entropy': -0.0469, 'ce_loss': 0.0335, 'epoch': 1.83}

 92%|█████████▏| 1292/1410 [4:11:55<23:18, 11.85s/it]
 92%|█████████▏| 1293/1410 [4:12:07<23:02, 11.81s/it]
                                                     
{'loss': 0.0157, 'grad_norm': 1.4180516004562378, 'learning_rate': 1.6169581458034983e-08, 'kl': 0.0457, 'entropy': -0.0898, 'ce_loss': 0.0175, 'epoch': 1.83}

 92%|█████████▏| 1293/1410 [4:12:07<23:02, 11.81s/it]
 92%|█████████▏| 1294/1410 [4:12:19<22:54, 11.85s/it]
                                                     
{'loss': 0.0151, 'grad_norm': 1.009212851524353, 'learning_rate': 1.5895991573573164e-08, 'kl': 0.0133, 'entropy': -0.0415, 'ce_loss': 0.0102, 'epoch': 1.84}

 92%|█████████▏| 1294/1410 [4:12:19<22:54, 11.85s/it]
 92%|█████████▏| 1295/1410 [4:12:31<22:39, 11.82s/it]
                                                     
{'loss': 0.0108, 'grad_norm': 1.0807095766067505, 'learning_rate': 1.562469443710021e-08, 'kl': 0.0269, 'entropy': -0.0859, 'ce_loss': 0.0175, 'epoch': 1.84}

 92%|█████████▏| 1295/1410 [4:12:31<22:39, 11.82s/it]
 92%|█████████▏| 1296/1410 [4:12:42<22:22, 11.78s/it]
                                                     
{'loss': 0.01, 'grad_norm': 1.1026238203048706, 'learning_rate': 1.5355691481489142e-08, 'kl': 0.0129, 'entropy': -0.0527, 'ce_loss': 0.0061, 'epoch': 1.84}

 92%|█████████▏| 1296/1410 [4:12:42<22:22, 11.78s/it]
 92%|█████████▏| 1297/1410 [4:12:54<22:09, 11.77s/it]
                                                     
{'loss': 0.0171, 'grad_norm': 1.2976899147033691, 'learning_rate': 1.508898412749612e-08, 'kl': 0.0134, 'entropy': -0.0311, 'ce_loss': 0.0023, 'epoch': 1.84}

 92%|█████████▏| 1297/1410 [4:12:54<22:09, 11.77s/it]
 92%|█████████▏| 1298/1410 [4:13:06<21:53, 11.73s/it]
                                                     
{'loss': 0.006, 'grad_norm': 0.7322930097579956, 'learning_rate': 1.4824573783753303e-08, 'kl': 0.0396, 'entropy': -0.082, 'ce_loss': 0.0182, 'epoch': 1.84}

 92%|█████████▏| 1298/1410 [4:13:06<21:53, 11.73s/it]
 92%|█████████▏| 1299/1410 [4:13:17<21:39, 11.70s/it]
                                                     
{'loss': 0.007, 'grad_norm': 0.8423061370849609, 'learning_rate': 1.456246184676046e-08, 'kl': -0.01, 'entropy': -0.0173, 'ce_loss': 0.1043, 'epoch': 1.84}

 92%|█████████▏| 1299/1410 [4:13:17<21:39, 11.70s/it]
 92%|█████████▏| 1300/1410 [4:13:29<21:16, 11.60s/it]
                                                     
{'loss': 0.0141, 'grad_norm': 1.3138309717178345, 'learning_rate': 1.4302649700878616e-08, 'kl': 0.0359, 'entropy': -0.106, 'ce_loss': 0.0058, 'epoch': 1.84}

 92%|█████████▏| 1300/1410 [4:13:29<21:16, 11.60s/it]
 92%|█████████▏| 1301/1410 [4:13:41<21:13, 11.69s/it]
                                                     
{'loss': 0.0258, 'grad_norm': 0.8320556879043579, 'learning_rate': 1.4045138718322314e-08, 'kl': 0.026, 'entropy': -0.0388, 'ce_loss': 0.0015, 'epoch': 1.85}

 92%|█████████▏| 1301/1410 [4:13:41<21:13, 11.69s/it]
 92%|█████████▏| 1302/1410 [4:13:51<20:21, 11.31s/it]
                                                     
{'loss': 0.0109, 'grad_norm': 1.0772387981414795, 'learning_rate': 1.3789930259152015e-08, 'kl': 0.0021, 'entropy': -0.0208, 'ce_loss': 0.0001, 'epoch': 1.85}

 92%|█████████▏| 1302/1410 [4:13:51<20:21, 11.31s/it]
 92%|█████████▏| 1303/1410 [4:14:03<20:26, 11.46s/it]
                                                     
{'loss': 0.0118, 'grad_norm': 1.1682777404785156, 'learning_rate': 1.3537025671267566e-08, 'kl': 0.0359, 'entropy': -0.0776, 'ce_loss': 0.0108, 'epoch': 1.85}

 92%|█████████▏| 1303/1410 [4:14:03<20:26, 11.46s/it]
 92%|█████████▏| 1304/1410 [4:14:14<20:05, 11.38s/it]
                                                     
{'loss': 0.0147, 'grad_norm': 1.2638726234436035, 'learning_rate': 1.3286426290400737e-08, 'kl': 0.0405, 'entropy': -0.1113, 'ce_loss': 0.0077, 'epoch': 1.85}

 92%|█████████▏| 1304/1410 [4:14:14<20:05, 11.38s/it]
 93%|█████████▎| 1305/1410 [4:14:25<19:48, 11.32s/it]
                                                     
{'loss': 0.0167, 'grad_norm': 1.4725147485733032, 'learning_rate': 1.3038133440107945e-08, 'kl': 0.0197, 'entropy': -0.0371, 'ce_loss': 0.0036, 'epoch': 1.85}

 93%|█████████▎| 1305/1410 [4:14:25<19:48, 11.32s/it]
 93%|█████████▎| 1306/1410 [4:14:37<19:51, 11.46s/it]
                                                     
{'loss': 0.0109, 'grad_norm': 0.9024627804756165, 'learning_rate': 1.2792148431763799e-08, 'kl': 0.0278, 'entropy': -0.1064, 'ce_loss': 0.0109, 'epoch': 1.85}

 93%|█████████▎| 1306/1410 [4:14:37<19:51, 11.46s/it]
 93%|█████████▎| 1307/1410 [4:14:49<19:47, 11.53s/it]
                                                     
{'loss': 0.0153, 'grad_norm': 0.848213791847229, 'learning_rate': 1.2548472564553858e-08, 'kl': 0.0195, 'entropy': -0.0439, 'ce_loss': 0.0032, 'epoch': 1.85}

 93%|█████████▎| 1307/1410 [4:14:49<19:47, 11.53s/it]
 93%|█████████▎| 1308/1410 [4:15:01<19:46, 11.63s/it]
                                                     
{'loss': 0.0123, 'grad_norm': 1.0429877042770386, 'learning_rate': 1.2307107125467635e-08, 'kl': -0.0095, 'entropy': -0.0659, 'ce_loss': 0.053, 'epoch': 1.86}

 93%|█████████▎| 1308/1410 [4:15:01<19:46, 11.63s/it]
 93%|█████████▎| 1309/1410 [4:15:12<19:42, 11.71s/it]
                                                     
{'loss': 0.0144, 'grad_norm': 1.1391881704330444, 'learning_rate': 1.206805338929221e-08, 'kl': 0.0425, 'entropy': -0.1196, 'ce_loss': 0.0231, 'epoch': 1.86}

 93%|█████████▎| 1309/1410 [4:15:12<19:42, 11.71s/it]
 93%|█████████▎| 1310/1410 [4:15:24<19:29, 11.70s/it]
                                                     
{'loss': 0.0141, 'grad_norm': 0.8472716212272644, 'learning_rate': 1.1831312618605021e-08, 'kl': 0.033, 'entropy': -0.0854, 'ce_loss': 0.0186, 'epoch': 1.86}

 93%|█████████▎| 1310/1410 [4:15:24<19:29, 11.70s/it]
 93%|█████████▎| 1311/1410 [4:15:35<18:55, 11.47s/it]
                                                     
{'loss': 0.0134, 'grad_norm': 1.3704532384872437, 'learning_rate': 1.1596886063767537e-08, 'kl': -0.0094, 'entropy': -0.0258, 'ce_loss': 0.0039, 'epoch': 1.86}

 93%|█████████▎| 1311/1410 [4:15:35<18:55, 11.47s/it]
 93%|█████████▎| 1312/1410 [4:15:47<18:49, 11.53s/it]
                                                     
{'loss': 0.0077, 'grad_norm': 1.2553271055221558, 'learning_rate': 1.1364774962918644e-08, 'kl': 0.0286, 'entropy': -0.0369, 'ce_loss': 0.0027, 'epoch': 1.86}

 93%|█████████▎| 1312/1410 [4:15:47<18:49, 11.53s/it]
 93%|█████████▎| 1313/1410 [4:15:58<18:37, 11.52s/it]
                                                     
{'loss': 0.0204, 'grad_norm': 1.3409430980682373, 'learning_rate': 1.113498054196782e-08, 'kl': 0.0608, 'entropy': -0.1079, 'ce_loss': 0.0055, 'epoch': 1.86}

 93%|█████████▎| 1313/1410 [4:15:58<18:37, 11.52s/it]
 93%|█████████▎| 1314/1410 [4:16:10<18:36, 11.63s/it]
                                                     
{'loss': 0.0182, 'grad_norm': 0.6815531849861145, 'learning_rate': 1.0907504014588974e-08, 'kl': -0.0083, 'entropy': -0.0133, 'ce_loss': 0.0555, 'epoch': 1.86}

 93%|█████████▎| 1314/1410 [4:16:10<18:36, 11.63s/it]
 93%|█████████▎| 1315/1410 [4:16:22<18:29, 11.68s/it]
                                                     
{'loss': 0.017, 'grad_norm': 1.3715459108352661, 'learning_rate': 1.0682346582213913e-08, 'kl': 0.0337, 'entropy': -0.085, 'ce_loss': 0.0189, 'epoch': 1.87}

 93%|█████████▎| 1315/1410 [4:16:22<18:29, 11.68s/it]
 93%|█████████▎| 1316/1410 [4:16:34<18:25, 11.76s/it]
                                                     
{'loss': 0.0152, 'grad_norm': 1.0914820432662964, 'learning_rate': 1.0459509434025937e-08, 'kl': 0.0225, 'entropy': -0.0854, 'ce_loss': 0.0155, 'epoch': 1.87}

 93%|█████████▎| 1316/1410 [4:16:34<18:25, 11.76s/it]
 93%|█████████▎| 1317/1410 [4:16:46<18:13, 11.76s/it]
                                                     
{'loss': 0.0082, 'grad_norm': 1.0371742248535156, 'learning_rate': 1.023899374695365e-08, 'kl': 0.0225, 'entropy': -0.052, 'ce_loss': 0.0041, 'epoch': 1.87}

 93%|█████████▎| 1317/1410 [4:16:46<18:13, 11.76s/it]
 93%|█████████▎| 1318/1410 [4:16:58<18:06, 11.81s/it]
                                                     
{'loss': 0.0216, 'grad_norm': 1.2381176948547363, 'learning_rate': 1.0020800685664715e-08, 'kl': 0.014, 'entropy': -0.0569, 'ce_loss': 0.0033, 'epoch': 1.87}

 93%|█████████▎| 1318/1410 [4:16:58<18:06, 11.81s/it]
 94%|█████████▎| 1319/1410 [4:17:09<17:56, 11.82s/it]
                                                     
{'loss': 0.0193, 'grad_norm': 1.2400152683258057, 'learning_rate': 9.804931402559658e-09, 'kl': 0.0286, 'entropy': -0.1025, 'ce_loss': 0.0069, 'epoch': 1.87}

 94%|█████████▎| 1319/1410 [4:17:09<17:56, 11.82s/it]
 94%|█████████▎| 1320/1410 [4:17:21<17:46, 11.85s/it]
                                                     
{'loss': 0.0239, 'grad_norm': 1.6582698822021484, 'learning_rate': 9.59138703776587e-09, 'kl': 0.0259, 'entropy': -0.0466, 'ce_loss': 0.0039, 'epoch': 1.87}

 94%|█████████▎| 1320/1410 [4:17:21<17:46, 11.85s/it]
 94%|█████████▎| 1321/1410 [4:17:33<17:31, 11.82s/it]
                                                     
{'loss': 0.0086, 'grad_norm': 0.9217672944068909, 'learning_rate': 9.380168719131615e-09, 'kl': -0.0046, 'entropy': 0.0113, 'ce_loss': 0.0699, 'epoch': 1.87}

 94%|█████████▎| 1321/1410 [4:17:33<17:31, 11.82s/it]
 94%|█████████▍| 1322/1410 [4:17:45<17:14, 11.76s/it]
                                                     
{'loss': 0.0089, 'grad_norm': 1.2033507823944092, 'learning_rate': 9.171277562219737e-09, 'kl': 0.0347, 'entropy': -0.1167, 'ce_loss': 0.0098, 'epoch': 1.88}

 94%|█████████▍| 1322/1410 [4:17:45<17:14, 11.76s/it]
 94%|█████████▍| 1323/1410 [4:17:56<17:01, 11.74s/it]
                                                     
{'loss': 0.0104, 'grad_norm': 1.131693720817566, 'learning_rate': 8.96471467030236e-09, 'kl': 0.0166, 'entropy': -0.0708, 'ce_loss': 0.0218, 'epoch': 1.88}

 94%|█████████▍| 1323/1410 [4:17:56<17:01, 11.74s/it]
 94%|█████████▍| 1324/1410 [4:18:08<16:40, 11.63s/it]
                                                     
{'loss': 0.0152, 'grad_norm': 1.467455506324768, 'learning_rate': 8.760481134354492e-09, 'kl': 0.0349, 'entropy': -0.0287, 'ce_loss': 0.0008, 'epoch': 1.88}

 94%|█████████▍| 1324/1410 [4:18:08<16:40, 11.63s/it]
 94%|█████████▍| 1325/1410 [4:18:20<16:34, 11.70s/it]
                                                     
{'loss': 0.0131, 'grad_norm': 1.2313392162322998, 'learning_rate': 8.558578033048585e-09, 'kl': 0.0315, 'entropy': -0.0742, 'ce_loss': 0.0037, 'epoch': 1.88}

 94%|█████████▍| 1325/1410 [4:18:20<16:34, 11.70s/it]
 94%|█████████▍| 1326/1410 [4:18:31<16:26, 11.75s/it]
                                                     
{'loss': 0.0171, 'grad_norm': 1.1002470254898071, 'learning_rate': 8.359006432748689e-09, 'kl': 0.0204, 'entropy': -0.0525, 'ce_loss': 0.0321, 'epoch': 1.88}

 94%|█████████▍| 1326/1410 [4:18:31<16:26, 11.75s/it]
 94%|█████████▍| 1327/1410 [4:18:43<16:14, 11.74s/it]
                                                     
{'loss': 0.0142, 'grad_norm': 1.2259588241577148, 'learning_rate': 8.161767387504948e-09, 'kl': 0.0079, 'entropy': -0.0398, 'ce_loss': 0.0044, 'epoch': 1.88}

 94%|█████████▍| 1327/1410 [4:18:43<16:14, 11.74s/it]
 94%|█████████▍| 1328/1410 [4:18:54<15:44, 11.52s/it]
                                                     
{'loss': 0.0109, 'grad_norm': 1.184155821800232, 'learning_rate': 7.966861939048014e-09, 'kl': 0.016, 'entropy': -0.0293, 'ce_loss': 0.0143, 'epoch': 1.88}

 94%|█████████▍| 1328/1410 [4:18:54<15:44, 11.52s/it]
 94%|█████████▍| 1329/1410 [4:19:06<15:48, 11.71s/it]
                                                     
{'loss': 0.0344, 'grad_norm': 1.2903276681900024, 'learning_rate': 7.77429111678335e-09, 'kl': 0.0356, 'entropy': 0.015, 'ce_loss': 0.1065, 'epoch': 1.89}

 94%|█████████▍| 1329/1410 [4:19:06<15:48, 11.71s/it]
 94%|█████████▍| 1330/1410 [4:19:18<15:38, 11.73s/it]
                                                     
{'loss': 0.0043, 'grad_norm': 0.6169547438621521, 'learning_rate': 7.584055937786077e-09, 'kl': -0.0034, 'entropy': 0.1206, 'ce_loss': 0.0977, 'epoch': 1.89}

 94%|█████████▍| 1330/1410 [4:19:18<15:38, 11.73s/it]
 94%|█████████▍| 1331/1410 [4:19:30<15:27, 11.75s/it]
                                                     
{'loss': 0.014, 'grad_norm': 1.1508457660675049, 'learning_rate': 7.396157406795439e-09, 'kl': 0.0182, 'entropy': -0.0383, 'ce_loss': 0.0033, 'epoch': 1.89}

 94%|█████████▍| 1331/1410 [4:19:30<15:27, 11.75s/it]
 94%|█████████▍| 1332/1410 [4:19:42<15:16, 11.76s/it]
                                                     
{'loss': 0.0103, 'grad_norm': 1.196691632270813, 'learning_rate': 7.21059651620945e-09, 'kl': -0.0057, 'entropy': 0.0289, 'ce_loss': 0.0977, 'epoch': 1.89}

 94%|█████████▍| 1332/1410 [4:19:42<15:16, 11.76s/it]
 95%|█████████▍| 1333/1410 [4:19:54<15:12, 11.85s/it]
                                                     
{'loss': 0.021, 'grad_norm': 0.7469039559364319, 'learning_rate': 7.027374246079798e-09, 'kl': -0.0096, 'entropy': 0.0615, 'ce_loss': 0.094, 'epoch': 1.89}

 95%|█████████▍| 1333/1410 [4:19:54<15:12, 11.85s/it]
 95%|█████████▍| 1334/1410 [4:20:06<15:00, 11.84s/it]
                                                     
{'loss': 0.0134, 'grad_norm': 0.8475077152252197, 'learning_rate': 6.846491564106555e-09, 'kl': 0.0432, 'entropy': -0.0356, 'ce_loss': 0.002, 'epoch': 1.89}

 95%|█████████▍| 1334/1410 [4:20:06<15:00, 11.84s/it]
 95%|█████████▍| 1335/1410 [4:20:17<14:36, 11.69s/it]
                                                     
{'loss': 0.0145, 'grad_norm': 1.28443443775177, 'learning_rate': 6.667949425633224e-09, 'kl': 0.0312, 'entropy': -0.0918, 'ce_loss': 0.0157, 'epoch': 1.89}

 95%|█████████▍| 1335/1410 [4:20:17<14:36, 11.69s/it]
 95%|█████████▍| 1336/1410 [4:20:29<14:26, 11.71s/it]
                                                     
{'loss': 0.0113, 'grad_norm': 2.006416082382202, 'learning_rate': 6.491748773641398e-09, 'kl': -0.0014, 'entropy': 0.02, 'ce_loss': 0.0629, 'epoch': 1.9}

 95%|█████████▍| 1336/1410 [4:20:29<14:26, 11.71s/it]
 95%|█████████▍| 1337/1410 [4:20:41<14:22, 11.82s/it]
                                                     
{'loss': 0.0405, 'grad_norm': 1.7245837450027466, 'learning_rate': 6.317890538746063e-09, 'kl': 0.02, 'entropy': -0.051, 'ce_loss': 0.0256, 'epoch': 1.9}

 95%|█████████▍| 1337/1410 [4:20:41<14:22, 11.82s/it]
 95%|█████████▍| 1338/1410 [4:20:53<14:10, 11.81s/it]
                                                     
{'loss': 0.0154, 'grad_norm': 1.0705682039260864, 'learning_rate': 6.146375639190599e-09, 'kl': 0.0147, 'entropy': -0.0571, 'ce_loss': 0.0119, 'epoch': 1.9}

 95%|█████████▍| 1338/1410 [4:20:53<14:10, 11.81s/it]
 95%|█████████▍| 1339/1410 [4:21:04<13:59, 11.83s/it]
                                                     
{'loss': 0.0171, 'grad_norm': 1.127750277519226, 'learning_rate': 5.977204980841888e-09, 'kl': -0.0086, 'entropy': 0.0942, 'ce_loss': 0.098, 'epoch': 1.9}

 95%|█████████▍| 1339/1410 [4:21:04<13:59, 11.83s/it]
 95%|█████████▌| 1340/1410 [4:21:16<13:43, 11.76s/it]
                                                     
{'loss': 0.0211, 'grad_norm': 1.2147213220596313, 'learning_rate': 5.810379457185516e-09, 'kl': 0.041, 'entropy': -0.0981, 'ce_loss': 0.0221, 'epoch': 1.9}

 95%|█████████▌| 1340/1410 [4:21:16<13:43, 11.76s/it]
 95%|█████████▌| 1341/1410 [4:21:28<13:27, 11.70s/it]
                                                     
{'loss': 0.0137, 'grad_norm': 1.5819120407104492, 'learning_rate': 5.645899949321176e-09, 'kl': 0.0369, 'entropy': -0.0923, 'ce_loss': 0.0153, 'epoch': 1.9}

 95%|█████████▌| 1341/1410 [4:21:28<13:27, 11.70s/it]
 95%|█████████▌| 1342/1410 [4:21:39<13:14, 11.68s/it]
                                                     
{'loss': 0.0174, 'grad_norm': 1.2793939113616943, 'learning_rate': 5.4837673259577225e-09, 'kl': 0.0447, 'entropy': -0.0854, 'ce_loss': 0.0029, 'epoch': 1.9}

 95%|█████████▌| 1342/1410 [4:21:39<13:14, 11.68s/it]
 95%|█████████▌| 1343/1410 [4:21:51<13:03, 11.69s/it]
                                                     
{'loss': 0.0263, 'grad_norm': 1.2795751094818115, 'learning_rate': 5.323982443409075e-09, 'kl': 0.0036, 'entropy': -0.0227, 'ce_loss': 0.0013, 'epoch': 1.9}

 95%|█████████▌| 1343/1410 [4:21:51<13:03, 11.69s/it]
 95%|█████████▌| 1344/1410 [4:22:03<12:55, 11.76s/it]
                                                     
{'loss': 0.0136, 'grad_norm': 0.8348223567008972, 'learning_rate': 5.166546145589323e-09, 'kl': -0.0042, 'entropy': -0.0139, 'ce_loss': 0.0548, 'epoch': 1.91}

 95%|█████████▌| 1344/1410 [4:22:03<12:55, 11.76s/it]
 95%|█████████▌| 1345/1410 [4:22:15<12:48, 11.82s/it]
                                                     
{'loss': 0.0305, 'grad_norm': 1.382495403289795, 'learning_rate': 5.011459264008128e-09, 'kl': 0.0305, 'entropy': -0.0801, 'ce_loss': 0.0303, 'epoch': 1.91}

 95%|█████████▌| 1345/1410 [4:22:15<12:48, 11.82s/it]
 95%|█████████▌| 1346/1410 [4:22:27<12:39, 11.86s/it]
                                                     
{'loss': 0.018, 'grad_norm': 1.1068236827850342, 'learning_rate': 4.858722617766875e-09, 'kl': 0.0312, 'entropy': -0.0815, 'ce_loss': 0.0124, 'epoch': 1.91}

 95%|█████████▌| 1346/1410 [4:22:27<12:39, 11.86s/it]
 96%|█████████▌| 1347/1410 [4:22:39<12:28, 11.88s/it]
                                                     
{'loss': 0.0258, 'grad_norm': 1.2015135288238525, 'learning_rate': 4.708337013553881e-09, 'kl': 0.0089, 'entropy': -0.0618, 'ce_loss': 0.0025, 'epoch': 1.91}

 96%|█████████▌| 1347/1410 [4:22:39<12:28, 11.88s/it]
 96%|█████████▌| 1348/1410 [4:22:50<12:11, 11.80s/it]
                                                     
{'loss': 0.0179, 'grad_norm': 1.3737080097198486, 'learning_rate': 4.5603032456401955e-09, 'kl': 0.0339, 'entropy': -0.1318, 'ce_loss': 0.0108, 'epoch': 1.91}

 96%|█████████▌| 1348/1410 [4:22:50<12:11, 11.80s/it]
 96%|█████████▌| 1349/1410 [4:23:02<11:59, 11.80s/it]
                                                     
{'loss': 0.0076, 'grad_norm': 1.0885467529296875, 'learning_rate': 4.414622095875753e-09, 'kl': 0.0056, 'entropy': -0.0012, 'ce_loss': 0.0045, 'epoch': 1.91}

 96%|█████████▌| 1349/1410 [4:23:02<11:59, 11.80s/it]
 96%|█████████▌| 1350/1410 [4:23:14<11:50, 11.84s/it]
                                                     
{'loss': 0.022, 'grad_norm': 1.239841341972351, 'learning_rate': 4.2712943336847775e-09, 'kl': 0.0082, 'entropy': -0.03, 'ce_loss': 0.0027, 'epoch': 1.91}

 96%|█████████▌| 1350/1410 [4:23:14<11:50, 11.84s/it]
 96%|█████████▌| 1351/1410 [4:23:26<11:38, 11.83s/it]
                                                     
{'loss': 0.0088, 'grad_norm': 1.2158174514770508, 'learning_rate': 4.1303207160618855e-09, 'kl': -0.0161, 'entropy': -0.0522, 'ce_loss': 0.0273, 'epoch': 1.92}

 96%|█████████▌| 1351/1410 [4:23:26<11:38, 11.83s/it]
 96%|█████████▌| 1352/1410 [4:23:38<11:28, 11.88s/it]
                                                     
{'loss': 0.0183, 'grad_norm': 1.196810245513916, 'learning_rate': 3.991701987568391e-09, 'kl': -0.0123, 'entropy': 0.0322, 'ce_loss': 0.0797, 'epoch': 1.92}

 96%|█████████▌| 1352/1410 [4:23:38<11:28, 11.88s/it]
 96%|█████████▌| 1353/1410 [4:23:49<11:13, 11.81s/it]
                                                     
{'loss': 0.0073, 'grad_norm': 0.8691873550415039, 'learning_rate': 3.855438880327755e-09, 'kl': 0.0347, 'entropy': -0.082, 'ce_loss': 0.0194, 'epoch': 1.92}

 96%|█████████▌| 1353/1410 [4:23:49<11:13, 11.81s/it]
 96%|█████████▌| 1354/1410 [4:24:01<11:02, 11.82s/it]
                                                     
{'loss': 0.0173, 'grad_norm': 1.2282917499542236, 'learning_rate': 3.721532114022391e-09, 'kl': -0.0046, 'entropy': -0.0242, 'ce_loss': 0.0527, 'epoch': 1.92}

 96%|█████████▌| 1354/1410 [4:24:01<11:02, 11.82s/it]
 96%|█████████▌| 1355/1410 [4:24:13<10:44, 11.71s/it]
                                                     
{'loss': 0.0256, 'grad_norm': 1.0353710651397705, 'learning_rate': 3.589982395889468e-09, 'kl': -0.0009, 'entropy': 0.0383, 'ce_loss': 0.0761, 'epoch': 1.92}

 96%|█████████▌| 1355/1410 [4:24:13<10:44, 11.71s/it]
 96%|█████████▌| 1356/1410 [4:24:25<10:33, 11.73s/it]
                                                     
{'loss': 0.0144, 'grad_norm': 1.1551687717437744, 'learning_rate': 3.4607904207170636e-09, 'kl': 0.0212, 'entropy': -0.054, 'ce_loss': 0.0171, 'epoch': 1.92}

 96%|█████████▌| 1356/1410 [4:24:25<10:33, 11.73s/it]
 96%|█████████▌| 1357/1410 [4:24:36<10:24, 11.79s/it]
                                                     
{'loss': 0.0051, 'grad_norm': 0.7406342029571533, 'learning_rate': 3.333956870840865e-09, 'kl': -0.0023, 'entropy': -0.0549, 'ce_loss': 0.0281, 'epoch': 1.92}

 96%|█████████▌| 1357/1410 [4:24:36<10:24, 11.79s/it]
 96%|█████████▋| 1358/1410 [4:24:46<09:43, 11.22s/it]
                                                     
{'loss': 0.0105, 'grad_norm': 1.3301331996917725, 'learning_rate': 3.2094824161404245e-09, 'kl': 0.0493, 'entropy': -0.0684, 'ce_loss': 0.0066, 'epoch': 1.93}

 96%|█████████▋| 1358/1410 [4:24:46<09:43, 11.22s/it]
 96%|█████████▋| 1359/1410 [4:24:58<09:43, 11.44s/it]
                                                     
{'loss': 0.0101, 'grad_norm': 0.6768611669540405, 'learning_rate': 3.0873677140354604e-09, 'kl': 0.0206, 'entropy': -0.0845, 'ce_loss': 0.0107, 'epoch': 1.93}

 96%|█████████▋| 1359/1410 [4:24:58<09:43, 11.44s/it]
 96%|█████████▋| 1360/1410 [4:25:10<09:31, 11.43s/it]
                                                     
{'loss': 0.0151, 'grad_norm': 0.8995928168296814, 'learning_rate': 2.967613409482511e-09, 'kl': -0.0019, 'entropy': -0.0698, 'ce_loss': 0.045, 'epoch': 1.93}

 96%|█████████▋| 1360/1410 [4:25:10<09:31, 11.43s/it]
 97%|█████████▋| 1361/1410 [4:25:22<09:26, 11.57s/it]
                                                     
{'loss': 0.026, 'grad_norm': 1.4326205253601074, 'learning_rate': 2.850220134971687e-09, 'kl': 0.0049, 'entropy': -0.0229, 'ce_loss': 0.0033, 'epoch': 1.93}

 97%|█████████▋| 1361/1410 [4:25:22<09:26, 11.57s/it]
 97%|█████████▋| 1362/1410 [4:25:34<09:21, 11.70s/it]
                                                     
{'loss': 0.0258, 'grad_norm': 1.1398707628250122, 'learning_rate': 2.7351885105229234e-09, 'kl': 0.0067, 'entropy': -0.033, 'ce_loss': 0.0006, 'epoch': 1.93}

 97%|█████████▋| 1362/1410 [4:25:34<09:21, 11.70s/it]
 97%|█████████▋| 1363/1410 [4:25:45<09:09, 11.70s/it]
                                                     
{'loss': 0.0084, 'grad_norm': 1.1233627796173096, 'learning_rate': 2.622519143683133e-09, 'kl': 0.0496, 'entropy': -0.0991, 'ce_loss': 0.0061, 'epoch': 1.93}

 97%|█████████▋| 1363/1410 [4:25:45<09:09, 11.70s/it]
 97%|█████████▋| 1364/1410 [4:25:57<09:00, 11.74s/it]
                                                     
{'loss': 0.0132, 'grad_norm': 1.3416153192520142, 'learning_rate': 2.5122126295227584e-09, 'kl': 0.0234, 'entropy': -0.0688, 'ce_loss': 0.0046, 'epoch': 1.93}

 97%|█████████▋| 1364/1410 [4:25:57<09:00, 11.74s/it]
 97%|█████████▋| 1365/1410 [4:26:09<08:48, 11.75s/it]
                                                     
{'loss': 0.0067, 'grad_norm': 0.8725802898406982, 'learning_rate': 2.404269550632676e-09, 'kl': 0.0135, 'entropy': -0.0522, 'ce_loss': 0.0217, 'epoch': 1.94}

 97%|█████████▋| 1365/1410 [4:26:09<08:48, 11.75s/it]
 97%|█████████▋| 1366/1410 [4:26:21<08:41, 11.86s/it]
                                                     
{'loss': 0.0255, 'grad_norm': 1.1244703531265259, 'learning_rate': 2.2986904771211957e-09, 'kl': 0.0073, 'entropy': -0.0261, 'ce_loss': 0.0036, 'epoch': 1.94}

 97%|█████████▋| 1366/1410 [4:26:21<08:41, 11.86s/it]
 97%|█████████▋| 1367/1410 [4:26:32<08:21, 11.66s/it]
                                                     
{'loss': 0.0098, 'grad_norm': 1.1899785995483398, 'learning_rate': 2.1954759666108156e-09, 'kl': 0.0287, 'entropy': -0.1055, 'ce_loss': 0.0133, 'epoch': 1.94}

 97%|█████████▋| 1367/1410 [4:26:32<08:21, 11.66s/it]
 97%|█████████▋| 1368/1410 [4:26:44<08:10, 11.67s/it]
                                                     
{'loss': 0.0175, 'grad_norm': 1.1740330457687378, 'learning_rate': 2.094626564235624e-09, 'kl': 0.0352, 'entropy': -0.0908, 'ce_loss': 0.0105, 'epoch': 1.94}

 97%|█████████▋| 1368/1410 [4:26:44<08:10, 11.67s/it]
 97%|█████████▋| 1369/1410 [4:26:56<08:01, 11.75s/it]
                                                     
{'loss': 0.0141, 'grad_norm': 1.0597925186157227, 'learning_rate': 1.996142802638151e-09, 'kl': 0.0035, 'entropy': 0.1602, 'ce_loss': 0.177, 'epoch': 1.94}

 97%|█████████▋| 1369/1410 [4:26:56<08:01, 11.75s/it]
 97%|█████████▋| 1370/1410 [4:27:08<07:50, 11.75s/it]
                                                     
{'loss': 0.0054, 'grad_norm': 0.8831906318664551, 'learning_rate': 1.900025201966621e-09, 'kl': 0.0176, 'entropy': -0.0532, 'ce_loss': 0.001, 'epoch': 1.94}

 97%|█████████▋| 1370/1410 [4:27:08<07:50, 11.75s/it]
 97%|█████████▋| 1371/1410 [4:27:19<07:39, 11.77s/it]
                                                     
{'loss': 0.0158, 'grad_norm': 1.302896499633789, 'learning_rate': 1.8062742698723542e-09, 'kl': 0.0178, 'entropy': -0.0176, 'ce_loss': 0.0044, 'epoch': 1.94}

 97%|█████████▋| 1371/1410 [4:27:19<07:39, 11.77s/it]
 97%|█████████▋| 1372/1410 [4:27:31<07:27, 11.77s/it]
                                                     
{'loss': 0.0138, 'grad_norm': 1.34892737865448, 'learning_rate': 1.7148905015068216e-09, 'kl': 0.0422, 'entropy': -0.1118, 'ce_loss': 0.0178, 'epoch': 1.95}

 97%|█████████▋| 1372/1410 [4:27:31<07:27, 11.77s/it]
 97%|█████████▋| 1373/1410 [4:27:42<07:06, 11.53s/it]
                                                     
{'loss': 0.0147, 'grad_norm': 1.1687042713165283, 'learning_rate': 1.6258743795191932e-09, 'kl': -0.0104, 'entropy': -0.0245, 'ce_loss': 0.0869, 'epoch': 1.95}

 97%|█████████▋| 1373/1410 [4:27:42<07:06, 11.53s/it]
 97%|█████████▋| 1374/1410 [4:27:54<06:59, 11.65s/it]
                                                     
{'loss': 0.0148, 'grad_norm': 0.8062624931335449, 'learning_rate': 1.5392263740538424e-09, 'kl': 0.003, 'entropy': 0.02, 'ce_loss': 0.0587, 'epoch': 1.95}

 97%|█████████▋| 1374/1410 [4:27:54<06:59, 11.65s/it]
 98%|█████████▊| 1375/1410 [4:28:06<06:48, 11.67s/it]
                                                     
{'loss': 0.0037, 'grad_norm': 0.5585755705833435, 'learning_rate': 1.4549469427477478e-09, 'kl': 0.0435, 'entropy': -0.1138, 'ce_loss': 0.0119, 'epoch': 1.95}

 98%|█████████▊| 1375/1410 [4:28:06<06:48, 11.67s/it]
 98%|█████████▊| 1376/1410 [4:28:18<06:40, 11.79s/it]
                                                     
{'loss': 0.0126, 'grad_norm': 1.4540290832519531, 'learning_rate': 1.3730365307279945e-09, 'kl': -0.0001, 'entropy': -0.0104, 'ce_loss': 0.0001, 'epoch': 1.95}

 98%|█████████▊| 1376/1410 [4:28:18<06:40, 11.79s/it]
 98%|█████████▊| 1377/1410 [4:28:30<06:28, 11.79s/it]
                                                     
{'loss': 0.0087, 'grad_norm': 0.8971464037895203, 'learning_rate': 1.293495570609776e-09, 'kl': 0.0315, 'entropy': -0.0559, 'ce_loss': 0.0234, 'epoch': 1.95}

 98%|█████████▊| 1377/1410 [4:28:30<06:28, 11.79s/it]
 98%|█████████▊| 1378/1410 [4:28:41<06:16, 11.75s/it]
                                                     
{'loss': 0.0146, 'grad_norm': 1.2643685340881348, 'learning_rate': 1.216324482493647e-09, 'kl': 0.014, 'entropy': -0.0238, 'ce_loss': 0.0047, 'epoch': 1.95}

 98%|█████████▊| 1378/1410 [4:28:41<06:16, 11.75s/it]
 98%|█████████▊| 1379/1410 [4:28:53<06:06, 11.83s/it]
                                                     
{'loss': 0.0122, 'grad_norm': 0.7387881875038147, 'learning_rate': 1.1415236739636237e-09, 'kl': -0.0061, 'entropy': 0.0688, 'ce_loss': 0.1589, 'epoch': 1.96}

 98%|█████████▊| 1379/1410 [4:28:53<06:06, 11.83s/it]
 98%|█████████▊| 1380/1410 [4:29:05<05:56, 11.89s/it]
                                                     
{'loss': 0.0053, 'grad_norm': 0.6947910189628601, 'learning_rate': 1.0690935400850366e-09, 'kl': 0.0034, 'entropy': -0.0104, 'ce_loss': 0.0002, 'epoch': 1.96}

 98%|█████████▊| 1380/1410 [4:29:05<05:56, 11.89s/it]
 98%|█████████▊| 1381/1410 [4:29:17<05:46, 11.95s/it]
                                                     
{'loss': 0.0159, 'grad_norm': 0.8579176068305969, 'learning_rate': 9.990344634021819e-10, 'kl': 0.0374, 'entropy': -0.0845, 'ce_loss': 0.0083, 'epoch': 1.96}

 98%|█████████▊| 1381/1410 [4:29:17<05:46, 11.95s/it]
 98%|█████████▊| 1382/1410 [4:29:29<05:33, 11.90s/it]
                                                     
{'loss': 0.0101, 'grad_norm': 0.8808637857437134, 'learning_rate': 9.313468139365233e-10, 'kl': -0.0067, 'entropy': -0.0537, 'ce_loss': 0.0377, 'epoch': 1.96}

 98%|█████████▊| 1382/1410 [4:29:29<05:33, 11.90s/it]
 98%|█████████▊| 1383/1410 [4:29:41<05:21, 11.91s/it]
                                                     
{'loss': 0.0209, 'grad_norm': 1.3927040100097656, 'learning_rate': 8.660309491847428e-10, 'kl': 0.0289, 'entropy': -0.0767, 'ce_loss': 0.0089, 'epoch': 1.96}

 98%|█████████▊| 1383/1410 [4:29:41<05:21, 11.91s/it]
 98%|█████████▊| 1384/1410 [4:29:53<05:08, 11.86s/it]
                                                     
{'loss': 0.0094, 'grad_norm': 0.9981158375740051, 'learning_rate': 8.030872141167433e-10, 'kl': 0.0452, 'entropy': -0.0977, 'ce_loss': 0.0079, 'epoch': 1.96}

 98%|█████████▊| 1384/1410 [4:29:53<05:08, 11.86s/it]
 98%|█████████▊| 1385/1410 [4:30:04<04:54, 11.77s/it]
                                                     
{'loss': 0.0094, 'grad_norm': 1.5137401819229126, 'learning_rate': 7.425159411738491e-10, 'kl': 0.041, 'entropy': -0.1104, 'ce_loss': 0.0104, 'epoch': 1.96}

 98%|█████████▊| 1385/1410 [4:30:04<04:54, 11.77s/it]
 98%|█████████▊| 1386/1410 [4:30:16<04:42, 11.77s/it]
                                                     
{'loss': 0.0043, 'grad_norm': 0.5724989175796509, 'learning_rate': 6.843174502671079e-10, 'kl': 0.0208, 'entropy': -0.0684, 'ce_loss': 0.0017, 'epoch': 1.97}

 98%|█████████▊| 1386/1410 [4:30:16<04:42, 11.77s/it]
 98%|█████████▊| 1387/1410 [4:30:28<04:30, 11.77s/it]
                                                     
{'loss': 0.0132, 'grad_norm': 1.2183082103729248, 'learning_rate': 6.284920487755418e-10, 'kl': 0.0469, 'entropy': -0.0747, 'ce_loss': 0.0189, 'epoch': 1.97}

 98%|█████████▊| 1387/1410 [4:30:28<04:30, 11.77s/it]
 98%|█████████▊| 1388/1410 [4:30:39<04:14, 11.56s/it]
                                                     
{'loss': 0.0089, 'grad_norm': 1.1836719512939453, 'learning_rate': 5.750400315445992e-10, 'kl': 0.015, 'entropy': -0.0703, 'ce_loss': 0.0088, 'epoch': 1.97}

 98%|█████████▊| 1388/1410 [4:30:39<04:14, 11.56s/it]
 99%|█████████▊| 1389/1410 [4:30:51<04:03, 11.60s/it]
                                                     
{'loss': 0.0077, 'grad_norm': 0.9223316311836243, 'learning_rate': 5.239616808845049e-10, 'kl': 0.0208, 'entropy': -0.0457, 'ce_loss': 0.0035, 'epoch': 1.97}

 99%|█████████▊| 1389/1410 [4:30:51<04:03, 11.60s/it]
 99%|█████████▊| 1390/1410 [4:31:03<03:54, 11.73s/it]
                                                     
{'loss': 0.0222, 'grad_norm': 1.0707849264144897, 'learning_rate': 4.752572665688126e-10, 'kl': 0.0009, 'entropy': -0.0029, 'ce_loss': 0.0008, 'epoch': 1.97}

 99%|█████████▊| 1390/1410 [4:31:03<03:54, 11.73s/it]
 99%|█████████▊| 1391/1410 [4:31:14<03:40, 11.61s/it]
                                                     
{'loss': 0.0108, 'grad_norm': 1.383495569229126, 'learning_rate': 4.2892704583295527e-10, 'kl': 0.0371, 'entropy': -0.0854, 'ce_loss': 0.0173, 'epoch': 1.97}

 99%|█████████▊| 1391/1410 [4:31:14<03:40, 11.61s/it]
 99%|█████████▊| 1392/1410 [4:31:26<03:27, 11.55s/it]
                                                     
{'loss': 0.0217, 'grad_norm': 1.2518837451934814, 'learning_rate': 3.849712633730462e-10, 'kl': 0.0334, 'entropy': -0.0874, 'ce_loss': 0.0126, 'epoch': 1.97}

 99%|█████████▊| 1392/1410 [4:31:26<03:27, 11.55s/it]
 99%|█████████▉| 1393/1410 [4:31:37<03:14, 11.43s/it]
                                                     
{'loss': 0.0174, 'grad_norm': 1.6328994035720825, 'learning_rate': 3.433901513442805e-10, 'kl': 0.0135, 'entropy': -0.0386, 'ce_loss': 0.0088, 'epoch': 1.98}

 99%|█████████▉| 1393/1410 [4:31:37<03:14, 11.43s/it]
 99%|█████████▉| 1394/1410 [4:31:47<02:59, 11.24s/it]
                                                     
{'loss': 0.0121, 'grad_norm': 1.17063570022583, 'learning_rate': 3.0418392935993577e-10, 'kl': 0.025, 'entropy': -0.0972, 'ce_loss': 0.012, 'epoch': 1.98}

 99%|█████████▉| 1394/1410 [4:31:47<02:59, 11.24s/it]
 99%|█████████▉| 1395/1410 [4:31:59<02:51, 11.45s/it]
                                                     
{'loss': 0.0366, 'grad_norm': 1.2916736602783203, 'learning_rate': 2.673528044902229e-10, 'kl': 0.001, 'entropy': -0.0522, 'ce_loss': 0.045, 'epoch': 1.98}

 99%|█████████▉| 1395/1410 [4:31:59<02:51, 11.45s/it]
 99%|█████████▉| 1396/1410 [4:32:11<02:42, 11.58s/it]
                                                     
{'loss': 0.0123, 'grad_norm': 1.0960687398910522, 'learning_rate': 2.3289697126103735e-10, 'kl': -0.0047, 'entropy': -0.0461, 'ce_loss': 0.0435, 'epoch': 1.98}

 99%|█████████▉| 1396/1410 [4:32:11<02:42, 11.58s/it]
 99%|█████████▉| 1397/1410 [4:32:23<02:30, 11.61s/it]
                                                     
{'loss': 0.0054, 'grad_norm': 0.727576732635498, 'learning_rate': 2.008166116531096e-10, 'kl': 0.0197, 'entropy': -0.0535, 'ce_loss': 0.005, 'epoch': 1.98}

 99%|█████████▉| 1397/1410 [4:32:23<02:30, 11.61s/it]
 99%|█████████▉| 1398/1410 [4:32:35<02:19, 11.66s/it]
                                                     
{'loss': 0.0095, 'grad_norm': 1.4503662586212158, 'learning_rate': 1.711118951008561e-10, 'kl': -0.0086, 'entropy': 0.0245, 'ce_loss': 0.083, 'epoch': 1.98}

 99%|█████████▉| 1398/1410 [4:32:35<02:19, 11.66s/it]
 99%|█████████▉| 1399/1410 [4:32:47<02:08, 11.69s/it]
                                                     
{'loss': 0.0136, 'grad_norm': 1.3851889371871948, 'learning_rate': 1.4378297849158004e-10, 'kl': -0.002, 'entropy': 0.024, 'ce_loss': 0.0526, 'epoch': 1.98}

 99%|█████████▉| 1399/1410 [4:32:47<02:08, 11.69s/it]
 99%|█████████▉| 1400/1410 [4:32:58<01:57, 11.73s/it]
                                                     
{'loss': 0.0235, 'grad_norm': 0.9709588885307312, 'learning_rate': 1.1883000616482174e-10, 'kl': 0.0201, 'entropy': -0.0967, 'ce_loss': 0.008, 'epoch': 1.99}

 99%|█████████▉| 1400/1410 [4:32:58<01:57, 11.73s/it]
 99%|█████████▉| 1401/1410 [4:33:10<01:45, 11.73s/it]
                                                     
{'loss': 0.0093, 'grad_norm': 0.9388182163238525, 'learning_rate': 9.625310991120961e-11, 'kl': -0.0043, 'entropy': -0.0123, 'ce_loss': 0.0008, 'epoch': 1.99}

 99%|█████████▉| 1401/1410 [4:33:10<01:45, 11.73s/it]
 99%|█████████▉| 1402/1410 [4:33:22<01:34, 11.76s/it]
                                                     
{'loss': 0.0319, 'grad_norm': 1.8124635219573975, 'learning_rate': 7.60524089721104e-11, 'kl': 0.0139, 'entropy': 0.0771, 'ce_loss': 0.1224, 'epoch': 1.99}

 99%|█████████▉| 1402/1410 [4:33:22<01:34, 11.76s/it]
100%|█████████▉| 1403/1410 [4:33:34<01:22, 11.81s/it]
                                                     
{'loss': 0.0329, 'grad_norm': 1.554975152015686, 'learning_rate': 5.822801003877998e-11, 'kl': -0.004, 'entropy': 0.0708, 'ce_loss': 0.0904, 'epoch': 1.99}

100%|█████████▉| 1403/1410 [4:33:34<01:22, 11.81s/it]
100%|█████████▉| 1404/1410 [4:33:46<01:10, 11.77s/it]
                                                     
{'loss': 0.0119, 'grad_norm': 1.0269269943237305, 'learning_rate': 4.2780007251913556e-11, 'kl': -0.0021, 'entropy': 0.0588, 'ce_loss': 0.1366, 'epoch': 1.99}

100%|█████████▉| 1404/1410 [4:33:46<01:10, 11.77s/it]
100%|█████████▉| 1405/1410 [4:33:57<00:58, 11.67s/it]
                                                     
{'loss': 0.009, 'grad_norm': 1.0339914560317993, 'learning_rate': 2.970848220109623e-11, 'kl': 0.0474, 'entropy': -0.0791, 'ce_loss': 0.0091, 'epoch': 1.99}

100%|█████████▉| 1405/1410 [4:33:57<00:58, 11.67s/it]
100%|█████████▉| 1406/1410 [4:34:09<00:46, 11.66s/it]
                                                     
{'loss': 0.0098, 'grad_norm': 0.8506991863250732, 'learning_rate': 1.901350392450318e-11, 'kl': 0.0126, 'entropy': -0.0127, 'ce_loss': 0.0033, 'epoch': 1.99}

100%|█████████▉| 1406/1410 [4:34:09<00:46, 11.66s/it]
100%|█████████▉| 1407/1410 [4:34:20<00:35, 11.67s/it]
                                                     
{'loss': 0.0095, 'grad_norm': 1.0562430620193481, 'learning_rate': 1.0695128908300155e-11, 'kl': -0.0054, 'entropy': 0.0198, 'ce_loss': 0.0873, 'epoch': 2.0}

100%|█████████▉| 1407/1410 [4:34:20<00:35, 11.67s/it]
100%|█████████▉| 1408/1410 [4:34:32<00:23, 11.65s/it]
                                                     
{'loss': 0.0099, 'grad_norm': 1.3244811296463013, 'learning_rate': 4.75340108649358e-12, 'kl': 0.0496, 'entropy': -0.1123, 'ce_loss': 0.0114, 'epoch': 2.0}

100%|█████████▉| 1408/1410 [4:34:32<00:23, 11.65s/it]
100%|█████████▉| 1409/1410 [4:34:43<00:11, 11.45s/it]
                                                     
{'loss': 0.0144, 'grad_norm': 1.4075050354003906, 'learning_rate': 1.1883518407307392e-12, 'kl': 0.004, 'entropy': -0.0388, 'ce_loss': 0.0028, 'epoch': 2.0}

100%|█████████▉| 1409/1410 [4:34:43<00:11, 11.45s/it]
100%|██████████| 1410/1410 [4:34:54<00:00, 11.48s/it]
                                                     
{'loss': 0.0167, 'grad_norm': 1.4677404165267944, 'learning_rate': 0.0, 'kl': 0.0452, 'entropy': -0.0762, 'ce_loss': 0.0107, 'epoch': 2.0}

100%|██████████| 1410/1410 [4:34:54<00:00, 11.48s/it][INFO|trainer.py:2665] 2025-04-27 19:10:42,438 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)



                                                     
{'train_runtime': 16494.942, 'train_samples_per_second': 1.026, 'train_steps_per_second': 0.085, 'train_loss': 0.022131871465586973, 'epoch': 2.0}

100%|██████████| 1410/1410 [4:34:54<00:00, 11.48s/it]
100%|██████████| 1410/1410 [4:34:54<00:00, 11.70s/it]
[INFO|trainer.py:3966] 2025-04-27 19:10:51,571 >> Saving model checkpoint to /scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1
[INFO|configuration_utils.py:423] 2025-04-27 19:10:51,585 >> Configuration saved in /scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1/config.json
[INFO|configuration_utils.py:908] 2025-04-27 19:10:51,586 >> Configuration saved in /scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1/generation_config.json
[2025-04-27 19:10:59,328] [INFO] [launch.py:351:main] Process 2201285 exits successfully.
[2025-04-27 19:11:02,343] [INFO] [launch.py:351:main] Process 2201284 exits successfully.
[2025-04-27 19:11:04,353] [INFO] [launch.py:351:main] Process 2201282 exits successfully.
[2025-04-27 19:11:07,368] [INFO] [launch.py:351:main] Process 2201281 exits successfully.
[2025-04-27 19:11:10,383] [INFO] [launch.py:351:main] Process 2201283 exits successfully.
[INFO|modeling_utils.py:3594] 2025-04-27 19:11:10,418 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 6 checkpoint shards. You can find where each parameters has been saved in the index located at /scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1/model.safetensors.index.json.
[INFO|tokenization_utils_base.py:2510] 2025-04-27 19:11:10,421 >> tokenizer config file saved in /scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1/tokenizer_config.json
[INFO|tokenization_utils_base.py:2519] 2025-04-27 19:11:10,421 >> Special tokens file saved in /scratch/gpfs/jg9904/saved_models/Qwen2.5-14B-Instruct-RL-2.1/special_tokens_map.json
***** train metrics *****
  epoch                    =        2.0
  total_flos               =   421479GF
  train_loss               =     0.0221
  train_runtime            = 4:34:54.94
  train_samples            =       8460
  train_samples_per_second =      1.026
  train_steps_per_second   =      0.085
[2025-04-27 19:11:17,418] [INFO] [launch.py:351:main] Process 2201280 exits successfully.