Hugging Face's logo

bibekyess
/
solar-checkpoint-2000

PEFT
TensorBoard
Safetensors
trl
sft
Generated from Trainer
Model card Files
xet
Metrics Community
bibekyess commited on
Commit
b451f4e
·
verified ·
1 Parent(s): b1a7e7e

Model save

Browse files
Files changed (6) hide show
  1. README.md +56 -0
  2. adapter_model.safetensors +1 -1
  3. wandb/debug-internal.log +192 -0
  4. wandb/debug.log +3 -0
  5. wandb/run-20240310_065024-lx2gw13k/logs/debug-internal.log +192 -0
  6. wandb/run-20240310_065024-lx2gw13k/logs/debug.log +3 -0
README.md ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ library_name: peft
4
+ tags:
5
+ - trl
6
+ - sft
7
+ - generated_from_trainer
8
+ datasets:
9
+ - generator
10
+ base_model: Edentns/DataVortexS-10.7B-dpo-v1.11
11
+ model-index:
12
+ - name: best-one
13
+ results: []
14
+ ---
15
+
16
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
17
+ should probably proofread and complete it, then remove this comment. -->
18
+
19
+ # best-one
20
+
21
+ This model is a fine-tuned version of [Edentns/DataVortexS-10.7B-dpo-v1.11](https://huggingface.co/Edentns/DataVortexS-10.7B-dpo-v1.11) on the generator dataset.
22
+
23
+ ## Model description
24
+
25
+ More information needed
26
+
27
+ ## Intended uses & limitations
28
+
29
+ More information needed
30
+
31
+ ## Training and evaluation data
32
+
33
+ More information needed
34
+
35
+ ## Training procedure
36
+
37
+ ### Training hyperparameters
38
+
39
+ The following hyperparameters were used during training:
40
+ - learning_rate: 0.0002
41
+ - train_batch_size: 1
42
+ - eval_batch_size: 8
43
+ - seed: 42
44
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
45
+ - lr_scheduler_type: constant
46
+ - lr_scheduler_warmup_ratio: 0.03
47
+ - num_epochs: 4
48
+ - mixed_precision_training: Native AMP
49
+
50
+ ### Framework versions
51
+
52
+ - PEFT 0.9.0
53
+ - Transformers 4.38.2
54
+ - Pytorch 2.1.0+cu121
55
+ - Datasets 2.18.0
56
+ - Tokenizers 0.15.2
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:6eb282a193e1395598ece0a26d494255e4a01998e06adeb1111c57000f4f8db1
3
  size 1006723888
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5f61c685cdbdcc4b5a782f6f2d2cb780b49f51c2bf86d7279cb11bc06dbe34f6
3
  size 1006723888
wandb/debug-internal.log CHANGED
@@ -3481,3 +3481,195 @@
3481
  2024-03-10 09:04:13,470 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3482
  2024-03-10 09:04:13,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3483
  2024-03-10 09:04:15,682 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3481
  2024-03-10 09:04:13,470 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3482
  2024-03-10 09:04:13,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3483
  2024-03-10 09:04:15,682 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3484
+ 2024-03-10 09:04:20,683 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3485
+ 2024-03-10 09:04:25,684 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3486
+ 2024-03-10 09:04:28,101 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3487
+ 2024-03-10 09:04:28,469 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3488
+ 2024-03-10 09:04:28,470 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3489
+ 2024-03-10 09:04:28,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3490
+ 2024-03-10 09:04:31,635 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3491
+ 2024-03-10 09:04:36,635 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3492
+ 2024-03-10 09:04:41,636 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3493
+ 2024-03-10 09:04:43,470 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3494
+ 2024-03-10 09:04:43,470 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3495
+ 2024-03-10 09:04:43,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3496
+ 2024-03-10 09:04:46,682 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3497
+ 2024-03-10 09:04:51,683 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3498
+ 2024-03-10 09:04:56,683 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3499
+ 2024-03-10 09:04:58,102 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3500
+ 2024-03-10 09:04:58,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3501
+ 2024-03-10 09:04:58,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3502
+ 2024-03-10 09:04:58,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3503
+ 2024-03-10 09:05:02,638 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3504
+ 2024-03-10 09:05:07,640 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3505
+ 2024-03-10 09:05:12,641 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3506
+ 2024-03-10 09:05:13,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3507
+ 2024-03-10 09:05:13,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3508
+ 2024-03-10 09:05:13,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3509
+ 2024-03-10 09:05:17,698 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3510
+ 2024-03-10 09:05:22,699 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3511
+ 2024-03-10 09:05:27,699 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3512
+ 2024-03-10 09:05:28,106 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3513
+ 2024-03-10 09:05:28,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3514
+ 2024-03-10 09:05:28,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3515
+ 2024-03-10 09:05:28,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3516
+ 2024-03-10 09:05:33,696 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3517
+ 2024-03-10 09:05:38,697 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3518
+ 2024-03-10 09:05:43,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3519
+ 2024-03-10 09:05:43,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3520
+ 2024-03-10 09:05:43,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3521
+ 2024-03-10 09:05:44,628 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3522
+ 2024-03-10 09:05:49,629 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3523
+ 2024-03-10 09:05:54,629 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3524
+ 2024-03-10 09:05:58,105 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3525
+ 2024-03-10 09:05:58,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3526
+ 2024-03-10 09:05:58,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3527
+ 2024-03-10 09:05:58,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3528
+ 2024-03-10 09:05:59,637 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3529
+ 2024-03-10 09:06:04,638 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3530
+ 2024-03-10 09:06:09,639 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3531
+ 2024-03-10 09:06:13,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3532
+ 2024-03-10 09:06:13,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3533
+ 2024-03-10 09:06:13,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3534
+ 2024-03-10 09:06:15,623 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3535
+ 2024-03-10 09:06:20,623 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3536
+ 2024-03-10 09:06:25,624 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3537
+ 2024-03-10 09:06:28,106 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3538
+ 2024-03-10 09:06:28,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3539
+ 2024-03-10 09:06:28,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3540
+ 2024-03-10 09:06:28,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3541
+ 2024-03-10 09:06:30,645 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3542
+ 2024-03-10 09:06:35,646 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3543
+ 2024-03-10 09:06:40,646 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3544
+ 2024-03-10 09:06:43,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3545
+ 2024-03-10 09:06:43,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3546
+ 2024-03-10 09:06:43,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3547
+ 2024-03-10 09:06:46,616 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3548
+ 2024-03-10 09:06:51,617 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3549
+ 2024-03-10 09:06:56,618 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3550
+ 2024-03-10 09:06:58,112 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3551
+ 2024-03-10 09:06:58,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3552
+ 2024-03-10 09:06:58,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3553
+ 2024-03-10 09:06:58,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3554
+ 2024-03-10 09:07:01,702 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3555
+ 2024-03-10 09:07:06,703 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3556
+ 2024-03-10 09:07:11,704 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3557
+ 2024-03-10 09:07:13,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3558
+ 2024-03-10 09:07:13,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3559
+ 2024-03-10 09:07:13,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3560
+ 2024-03-10 09:07:17,706 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3561
+ 2024-03-10 09:07:22,707 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3562
+ 2024-03-10 09:07:27,708 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3563
+ 2024-03-10 09:07:28,112 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3564
+ 2024-03-10 09:07:28,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3565
+ 2024-03-10 09:07:28,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3566
+ 2024-03-10 09:07:28,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3567
+ 2024-03-10 09:07:33,697 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3568
+ 2024-03-10 09:07:38,697 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3569
+ 2024-03-10 09:07:43,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3570
+ 2024-03-10 09:07:43,474 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3571
+ 2024-03-10 09:07:43,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3572
+ 2024-03-10 09:07:44,613 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3573
+ 2024-03-10 09:07:49,613 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3574
+ 2024-03-10 09:07:54,614 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3575
+ 2024-03-10 09:07:58,113 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3576
+ 2024-03-10 09:07:58,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3577
+ 2024-03-10 09:07:58,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3578
+ 2024-03-10 09:07:58,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3579
+ 2024-03-10 09:07:59,642 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3580
+ 2024-03-10 09:08:04,642 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3581
+ 2024-03-10 09:08:09,643 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3582
+ 2024-03-10 09:08:13,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3583
+ 2024-03-10 09:08:13,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3584
+ 2024-03-10 09:08:13,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3585
+ 2024-03-10 09:08:15,642 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3586
+ 2024-03-10 09:08:20,643 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3587
+ 2024-03-10 09:08:25,643 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3588
+ 2024-03-10 09:08:28,114 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3589
+ 2024-03-10 09:08:28,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3590
+ 2024-03-10 09:08:28,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3591
+ 2024-03-10 09:08:28,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3592
+ 2024-03-10 09:08:30,714 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3593
+ 2024-03-10 09:08:35,714 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3594
+ 2024-03-10 09:08:40,715 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3595
+ 2024-03-10 09:08:43,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3596
+ 2024-03-10 09:08:43,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3597
+ 2024-03-10 09:08:43,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3598
+ 2024-03-10 09:08:46,624 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3599
+ 2024-03-10 09:08:51,625 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3600
+ 2024-03-10 09:08:56,625 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3601
+ 2024-03-10 09:08:58,114 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3602
+ 2024-03-10 09:08:58,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3603
+ 2024-03-10 09:08:58,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3604
+ 2024-03-10 09:08:58,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3605
+ 2024-03-10 09:09:02,608 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3606
+ 2024-03-10 09:09:07,608 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3607
+ 2024-03-10 09:09:12,609 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3608
+ 2024-03-10 09:09:13,473 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3609
+ 2024-03-10 09:09:13,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3610
+ 2024-03-10 09:09:13,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3611
+ 2024-03-10 09:09:17,645 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3612
+ 2024-03-10 09:09:22,650 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3613
+ 2024-03-10 09:09:27,651 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3614
+ 2024-03-10 09:09:28,115 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3615
+ 2024-03-10 09:09:28,473 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3616
+ 2024-03-10 09:09:28,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3617
+ 2024-03-10 09:09:28,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3618
+ 2024-03-10 09:09:32,705 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3619
+ 2024-03-10 09:09:37,706 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3620
+ 2024-03-10 09:09:42,707 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3621
+ 2024-03-10 09:09:43,473 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3622
+ 2024-03-10 09:09:43,474 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3623
+ 2024-03-10 09:09:43,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3624
+ 2024-03-10 09:09:48,687 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3625
+ 2024-03-10 09:09:53,687 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3626
+ 2024-03-10 09:09:58,116 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3627
+ 2024-03-10 09:09:58,473 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3628
+ 2024-03-10 09:09:58,474 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3629
+ 2024-03-10 09:09:58,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3630
+ 2024-03-10 09:09:58,694 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3631
+ 2024-03-10 09:09:59,313 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: pause
3632
+ 2024-03-10 09:09:59,314 INFO HandlerThread:4057 [handler.py:handle_request_pause():708] stopping system metrics thread
3633
+ 2024-03-10 09:09:59,314 INFO HandlerThread:4057 [system_monitor.py:finish():203] Stopping system monitor
3634
+ 2024-03-10 09:09:59,314 DEBUG SystemMonitor:4057 [system_monitor.py:_start():179] Finished system metrics aggregation loop
3635
+ 2024-03-10 09:09:59,316 INFO HandlerThread:4057 [interfaces.py:finish():202] Joined cpu monitor
3636
+ 2024-03-10 09:09:59,317 DEBUG SystemMonitor:4057 [system_monitor.py:_start():183] Publishing last batch of metrics
3637
+ 2024-03-10 09:09:59,318 INFO HandlerThread:4057 [interfaces.py:finish():202] Joined disk monitor
3638
+ 2024-03-10 09:09:59,328 INFO HandlerThread:4057 [interfaces.py:finish():202] Joined gpu monitor
3639
+ 2024-03-10 09:09:59,329 INFO HandlerThread:4057 [interfaces.py:finish():202] Joined memory monitor
3640
+ 2024-03-10 09:09:59,329 INFO HandlerThread:4057 [interfaces.py:finish():202] Joined network monitor
3641
+ 2024-03-10 09:09:59,329 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3642
+ 2024-03-10 09:10:04,332 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3643
+ 2024-03-10 09:10:09,333 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3644
+ 2024-03-10 09:10:13,473 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3645
+ 2024-03-10 09:10:13,474 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3646
+ 2024-03-10 09:10:13,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3647
+ 2024-03-10 09:10:14,683 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3648
+ 2024-03-10 09:10:16,588 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: resume
3649
+ 2024-03-10 09:10:16,588 INFO HandlerThread:4057 [handler.py:handle_request_resume():699] starting system metrics thread
3650
+ 2024-03-10 09:10:16,589 INFO HandlerThread:4057 [system_monitor.py:start():194] Starting system monitor
3651
+ 2024-03-10 09:10:16,606 INFO SystemMonitor:4057 [system_monitor.py:_start():158] Starting system asset monitoring threads
3652
+ 2024-03-10 09:10:16,616 INFO SystemMonitor:4057 [interfaces.py:start():190] Started cpu monitoring
3653
+ 2024-03-10 09:10:16,617 INFO SystemMonitor:4057 [interfaces.py:start():190] Started disk monitoring
3654
+ 2024-03-10 09:10:16,622 INFO SystemMonitor:4057 [interfaces.py:start():190] Started gpu monitoring
3655
+ 2024-03-10 09:10:16,627 INFO SystemMonitor:4057 [interfaces.py:start():190] Started memory monitoring
3656
+ 2024-03-10 09:10:16,638 INFO SystemMonitor:4057 [interfaces.py:start():190] Started network monitoring
3657
+ 2024-03-10 09:10:19,685 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3658
+ 2024-03-10 09:10:24,686 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3659
+ 2024-03-10 09:10:29,687 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3660
+ 2024-03-10 09:10:33,933 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3661
+ 2024-03-10 09:10:33,935 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3662
+ 2024-03-10 09:10:33,936 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3663
+ 2024-03-10 09:10:35,095 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3664
+ 2024-03-10 09:10:40,096 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3665
+ 2024-03-10 09:10:45,097 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3666
+ 2024-03-10 09:10:48,931 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3667
+ 2024-03-10 09:10:48,933 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3668
+ 2024-03-10 09:10:48,972 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3669
+ 2024-03-10 09:10:50,136 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3670
+ 2024-03-10 09:10:55,137 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3671
+ 2024-03-10 09:11:00,137 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3672
+ 2024-03-10 09:11:03,931 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3673
+ 2024-03-10 09:11:03,932 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3674
+ 2024-03-10 09:11:03,972 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3675
+ 2024-03-10 09:11:06,106 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
wandb/debug.log CHANGED
@@ -28,3 +28,6 @@ config: {}
28
  2024-03-10 06:50:28,340 INFO MainThread:929 [wandb_run.py:_redirect():2186] Redirects installed.
29
  2024-03-10 06:50:28,343 INFO MainThread:929 [wandb_init.py:init():847] run started, returning control to user process
30
  2024-03-10 06:50:28,351 INFO MainThread:929 [wandb_run.py:_config_callback():1343] config_cb None None {'vocab_size': 48000, 'max_position_embeddings': 4096, 'hidden_size': 4096, 'intermediate_size': 14336, 'num_hidden_layers': 48, 'num_attention_heads': 32, 'num_key_value_heads': 8, 'hidden_act': 'silu', 'initializer_range': 0.02, 'rms_norm_eps': 1e-05, 'pretraining_tp': 1, 'use_cache': True, 'rope_theta': 10000.0, 'rope_scaling': None, 'attention_bias': False, 'attention_dropout': 0.0, 'return_dict': True, 'output_hidden_states': False, 'output_attentions': False, 'torchscript': False, 'torch_dtype': 'float16', 'use_bfloat16': False, 'tf_legacy_loss': False, 'pruned_heads': {}, 'tie_word_embeddings': False, 'chunk_size_feed_forward': 0, 'is_encoder_decoder': False, 'is_decoder': False, 'cross_attention_hidden_size': None, 'add_cross_attention': False, 'tie_encoder_decoder': False, 'max_length': 20, 'min_length': 0, 'do_sample': False, 'early_stopping': False, 'num_beams': 1, 'num_beam_groups': 1, 'diversity_penalty': 0.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0, 'typical_p': 1.0, 'repetition_penalty': 1.0, 'length_penalty': 1.0, 'no_repeat_ngram_size': 0, 'encoder_no_repeat_ngram_size': 0, 'bad_words_ids': None, 'num_return_sequences': 1, 'output_scores': False, 'return_dict_in_generate': False, 'forced_bos_token_id': None, 'forced_eos_token_id': None, 'remove_invalid_values': False, 'exponential_decay_length_penalty': None, 'suppress_tokens': None, 'begin_suppress_tokens': None, 'architectures': ['LlamaForCausalLM'], 'finetuning_task': None, 'id2label': {0: 'LABEL_0', 1: 'LABEL_1'}, 'label2id': {'LABEL_0': 0, 'LABEL_1': 1}, 'tokenizer_class': None, 'prefix': None, 'bos_token_id': 1, 'pad_token_id': 2, 'eos_token_id': 32000, 'sep_token_id': None, 'decoder_start_token_id': None, 'task_specific_params': None, 'problem_type': None, '_name_or_path': 'Edentns/DataVortexS-10.7B-dpo-v1.11', 'transformers_version': '4.38.2', 'model_type': 'llama', 'quantization_config': {'quant_method': 'QuantizationMethod.BITS_AND_BYTES', '_load_in_8bit': False, '_load_in_4bit': True, 'llm_int8_threshold': 6.0, 'llm_int8_skip_modules': None, 'llm_int8_enable_fp32_cpu_offload': False, 'llm_int8_has_fp16_weight': False, 'bnb_4bit_quant_type': 'nf4', 'bnb_4bit_use_double_quant': True, 'bnb_4bit_compute_dtype': 'float16', 'load_in_4bit': True, 'load_in_8bit': False}, 'output_dir': '/content/drive/MyDrive/best-one', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'evaluation_strategy': 'no', 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 8, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 1, 'eval_accumulation_steps': None, 'eval_delay': 0, 'learning_rate': 0.0002, 'weight_decay': 0.0, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 0.3, 'num_train_epochs': 4, 'max_steps': -1, 'lr_scheduler_type': 'constant', 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.03, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': '/content/drive/MyDrive/best-one/runs/Mar10_06-48-48_17daf5749447', 'logging_strategy': 'steps', 'logging_first_step': False, 'logging_steps': 500, 'logging_nan_inf_filter': True, 'save_strategy': 'steps', 'save_steps': 100, 'save_total_limit': None, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': False, 'fp16': True, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': '/content/drive/MyDrive/best-one', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True}, 'deepspeed': None, 'label_smoothing_factor': 0.0, 'optim': 'adamw_torch_fused', 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['tensorboard', 'wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': True, 'resume_from_checkpoint': '/content/drive/MyDrive/best-one/checkpoint-1000', 'hub_model_id': None, 'hub_strategy': 'every_save', 'hub_token': '<HUB_TOKEN>', 'hub_private_repo': False, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': '<PUSH_TO_HUB_TOKEN>', 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'dispatch_batches': None, 'split_batches': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None}
 
 
 
 
28
  2024-03-10 06:50:28,340 INFO MainThread:929 [wandb_run.py:_redirect():2186] Redirects installed.
29
  2024-03-10 06:50:28,343 INFO MainThread:929 [wandb_init.py:init():847] run started, returning control to user process
30
  2024-03-10 06:50:28,351 INFO MainThread:929 [wandb_run.py:_config_callback():1343] config_cb None None {'vocab_size': 48000, 'max_position_embeddings': 4096, 'hidden_size': 4096, 'intermediate_size': 14336, 'num_hidden_layers': 48, 'num_attention_heads': 32, 'num_key_value_heads': 8, 'hidden_act': 'silu', 'initializer_range': 0.02, 'rms_norm_eps': 1e-05, 'pretraining_tp': 1, 'use_cache': True, 'rope_theta': 10000.0, 'rope_scaling': None, 'attention_bias': False, 'attention_dropout': 0.0, 'return_dict': True, 'output_hidden_states': False, 'output_attentions': False, 'torchscript': False, 'torch_dtype': 'float16', 'use_bfloat16': False, 'tf_legacy_loss': False, 'pruned_heads': {}, 'tie_word_embeddings': False, 'chunk_size_feed_forward': 0, 'is_encoder_decoder': False, 'is_decoder': False, 'cross_attention_hidden_size': None, 'add_cross_attention': False, 'tie_encoder_decoder': False, 'max_length': 20, 'min_length': 0, 'do_sample': False, 'early_stopping': False, 'num_beams': 1, 'num_beam_groups': 1, 'diversity_penalty': 0.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0, 'typical_p': 1.0, 'repetition_penalty': 1.0, 'length_penalty': 1.0, 'no_repeat_ngram_size': 0, 'encoder_no_repeat_ngram_size': 0, 'bad_words_ids': None, 'num_return_sequences': 1, 'output_scores': False, 'return_dict_in_generate': False, 'forced_bos_token_id': None, 'forced_eos_token_id': None, 'remove_invalid_values': False, 'exponential_decay_length_penalty': None, 'suppress_tokens': None, 'begin_suppress_tokens': None, 'architectures': ['LlamaForCausalLM'], 'finetuning_task': None, 'id2label': {0: 'LABEL_0', 1: 'LABEL_1'}, 'label2id': {'LABEL_0': 0, 'LABEL_1': 1}, 'tokenizer_class': None, 'prefix': None, 'bos_token_id': 1, 'pad_token_id': 2, 'eos_token_id': 32000, 'sep_token_id': None, 'decoder_start_token_id': None, 'task_specific_params': None, 'problem_type': None, '_name_or_path': 'Edentns/DataVortexS-10.7B-dpo-v1.11', 'transformers_version': '4.38.2', 'model_type': 'llama', 'quantization_config': {'quant_method': 'QuantizationMethod.BITS_AND_BYTES', '_load_in_8bit': False, '_load_in_4bit': True, 'llm_int8_threshold': 6.0, 'llm_int8_skip_modules': None, 'llm_int8_enable_fp32_cpu_offload': False, 'llm_int8_has_fp16_weight': False, 'bnb_4bit_quant_type': 'nf4', 'bnb_4bit_use_double_quant': True, 'bnb_4bit_compute_dtype': 'float16', 'load_in_4bit': True, 'load_in_8bit': False}, 'output_dir': '/content/drive/MyDrive/best-one', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'evaluation_strategy': 'no', 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 8, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 1, 'eval_accumulation_steps': None, 'eval_delay': 0, 'learning_rate': 0.0002, 'weight_decay': 0.0, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 0.3, 'num_train_epochs': 4, 'max_steps': -1, 'lr_scheduler_type': 'constant', 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.03, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': '/content/drive/MyDrive/best-one/runs/Mar10_06-48-48_17daf5749447', 'logging_strategy': 'steps', 'logging_first_step': False, 'logging_steps': 500, 'logging_nan_inf_filter': True, 'save_strategy': 'steps', 'save_steps': 100, 'save_total_limit': None, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': False, 'fp16': True, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': '/content/drive/MyDrive/best-one', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True}, 'deepspeed': None, 'label_smoothing_factor': 0.0, 'optim': 'adamw_torch_fused', 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['tensorboard', 'wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': True, 'resume_from_checkpoint': '/content/drive/MyDrive/best-one/checkpoint-1000', 'hub_model_id': None, 'hub_strategy': 'every_save', 'hub_token': '<HUB_TOKEN>', 'hub_private_repo': False, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': '<PUSH_TO_HUB_TOKEN>', 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'dispatch_batches': None, 'split_batches': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None}
31
+ 2024-03-10 09:09:59,311 INFO MainThread:929 [jupyter.py:save_ipynb():373] not saving jupyter notebook
32
+ 2024-03-10 09:09:59,313 INFO MainThread:929 [wandb_init.py:_pause_backend():437] pausing backend
33
+ 2024-03-10 09:10:16,586 INFO MainThread:929 [wandb_init.py:_resume_backend():442] resuming backend
wandb/run-20240310_065024-lx2gw13k/logs/debug-internal.log CHANGED
@@ -3481,3 +3481,195 @@
3481
  2024-03-10 09:04:13,470 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3482
  2024-03-10 09:04:13,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3483
  2024-03-10 09:04:15,682 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3481
  2024-03-10 09:04:13,470 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3482
  2024-03-10 09:04:13,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3483
  2024-03-10 09:04:15,682 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3484
+ 2024-03-10 09:04:20,683 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3485
+ 2024-03-10 09:04:25,684 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3486
+ 2024-03-10 09:04:28,101 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3487
+ 2024-03-10 09:04:28,469 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3488
+ 2024-03-10 09:04:28,470 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3489
+ 2024-03-10 09:04:28,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3490
+ 2024-03-10 09:04:31,635 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3491
+ 2024-03-10 09:04:36,635 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3492
+ 2024-03-10 09:04:41,636 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3493
+ 2024-03-10 09:04:43,470 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3494
+ 2024-03-10 09:04:43,470 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3495
+ 2024-03-10 09:04:43,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3496
+ 2024-03-10 09:04:46,682 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3497
+ 2024-03-10 09:04:51,683 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3498
+ 2024-03-10 09:04:56,683 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3499
+ 2024-03-10 09:04:58,102 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3500
+ 2024-03-10 09:04:58,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3501
+ 2024-03-10 09:04:58,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3502
+ 2024-03-10 09:04:58,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3503
+ 2024-03-10 09:05:02,638 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3504
+ 2024-03-10 09:05:07,640 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3505
+ 2024-03-10 09:05:12,641 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3506
+ 2024-03-10 09:05:13,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3507
+ 2024-03-10 09:05:13,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3508
+ 2024-03-10 09:05:13,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3509
+ 2024-03-10 09:05:17,698 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3510
+ 2024-03-10 09:05:22,699 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3511
+ 2024-03-10 09:05:27,699 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3512
+ 2024-03-10 09:05:28,106 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3513
+ 2024-03-10 09:05:28,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3514
+ 2024-03-10 09:05:28,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3515
+ 2024-03-10 09:05:28,536 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3516
+ 2024-03-10 09:05:33,696 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3517
+ 2024-03-10 09:05:38,697 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3518
+ 2024-03-10 09:05:43,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3519
+ 2024-03-10 09:05:43,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3520
+ 2024-03-10 09:05:43,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3521
+ 2024-03-10 09:05:44,628 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3522
+ 2024-03-10 09:05:49,629 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3523
+ 2024-03-10 09:05:54,629 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3524
+ 2024-03-10 09:05:58,105 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3525
+ 2024-03-10 09:05:58,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3526
+ 2024-03-10 09:05:58,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3527
+ 2024-03-10 09:05:58,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3528
+ 2024-03-10 09:05:59,637 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3529
+ 2024-03-10 09:06:04,638 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3530
+ 2024-03-10 09:06:09,639 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3531
+ 2024-03-10 09:06:13,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3532
+ 2024-03-10 09:06:13,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3533
+ 2024-03-10 09:06:13,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3534
+ 2024-03-10 09:06:15,623 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3535
+ 2024-03-10 09:06:20,623 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3536
+ 2024-03-10 09:06:25,624 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3537
+ 2024-03-10 09:06:28,106 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3538
+ 2024-03-10 09:06:28,471 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3539
+ 2024-03-10 09:06:28,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3540
+ 2024-03-10 09:06:28,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3541
+ 2024-03-10 09:06:30,645 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3542
+ 2024-03-10 09:06:35,646 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3543
+ 2024-03-10 09:06:40,646 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3544
+ 2024-03-10 09:06:43,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3545
+ 2024-03-10 09:06:43,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3546
+ 2024-03-10 09:06:43,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3547
+ 2024-03-10 09:06:46,616 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3548
+ 2024-03-10 09:06:51,617 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3549
+ 2024-03-10 09:06:56,618 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3550
+ 2024-03-10 09:06:58,112 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3551
+ 2024-03-10 09:06:58,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3552
+ 2024-03-10 09:06:58,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3553
+ 2024-03-10 09:06:58,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3554
+ 2024-03-10 09:07:01,702 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3555
+ 2024-03-10 09:07:06,703 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3556
+ 2024-03-10 09:07:11,704 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3557
+ 2024-03-10 09:07:13,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3558
+ 2024-03-10 09:07:13,472 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3559
+ 2024-03-10 09:07:13,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3560
+ 2024-03-10 09:07:17,706 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3561
+ 2024-03-10 09:07:22,707 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3562
+ 2024-03-10 09:07:27,708 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3563
+ 2024-03-10 09:07:28,112 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3564
+ 2024-03-10 09:07:28,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3565
+ 2024-03-10 09:07:28,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3566
+ 2024-03-10 09:07:28,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3567
+ 2024-03-10 09:07:33,697 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3568
+ 2024-03-10 09:07:38,697 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3569
+ 2024-03-10 09:07:43,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3570
+ 2024-03-10 09:07:43,474 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3571
+ 2024-03-10 09:07:43,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3572
+ 2024-03-10 09:07:44,613 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3573
+ 2024-03-10 09:07:49,613 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3574
+ 2024-03-10 09:07:54,614 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3575
+ 2024-03-10 09:07:58,113 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3576
+ 2024-03-10 09:07:58,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3577
+ 2024-03-10 09:07:58,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3578
+ 2024-03-10 09:07:58,537 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3579
+ 2024-03-10 09:07:59,642 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3580
+ 2024-03-10 09:08:04,642 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3581
+ 2024-03-10 09:08:09,643 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3582
+ 2024-03-10 09:08:13,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3583
+ 2024-03-10 09:08:13,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3584
+ 2024-03-10 09:08:13,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3585
+ 2024-03-10 09:08:15,642 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3586
+ 2024-03-10 09:08:20,643 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3587
+ 2024-03-10 09:08:25,643 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3588
+ 2024-03-10 09:08:28,114 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3589
+ 2024-03-10 09:08:28,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3590
+ 2024-03-10 09:08:28,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3591
+ 2024-03-10 09:08:28,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3592
+ 2024-03-10 09:08:30,714 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3593
+ 2024-03-10 09:08:35,714 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3594
+ 2024-03-10 09:08:40,715 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3595
+ 2024-03-10 09:08:43,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3596
+ 2024-03-10 09:08:43,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3597
+ 2024-03-10 09:08:43,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3598
+ 2024-03-10 09:08:46,624 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3599
+ 2024-03-10 09:08:51,625 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3600
+ 2024-03-10 09:08:56,625 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3601
+ 2024-03-10 09:08:58,114 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3602
+ 2024-03-10 09:08:58,472 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3603
+ 2024-03-10 09:08:58,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3604
+ 2024-03-10 09:08:58,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3605
+ 2024-03-10 09:09:02,608 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3606
+ 2024-03-10 09:09:07,608 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3607
+ 2024-03-10 09:09:12,609 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3608
+ 2024-03-10 09:09:13,473 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3609
+ 2024-03-10 09:09:13,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3610
+ 2024-03-10 09:09:13,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3611
+ 2024-03-10 09:09:17,645 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3612
+ 2024-03-10 09:09:22,650 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3613
+ 2024-03-10 09:09:27,651 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3614
+ 2024-03-10 09:09:28,115 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3615
+ 2024-03-10 09:09:28,473 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3616
+ 2024-03-10 09:09:28,473 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3617
+ 2024-03-10 09:09:28,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3618
+ 2024-03-10 09:09:32,705 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3619
+ 2024-03-10 09:09:37,706 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3620
+ 2024-03-10 09:09:42,707 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3621
+ 2024-03-10 09:09:43,473 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3622
+ 2024-03-10 09:09:43,474 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3623
+ 2024-03-10 09:09:43,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3624
+ 2024-03-10 09:09:48,687 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3625
+ 2024-03-10 09:09:53,687 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3626
+ 2024-03-10 09:09:58,116 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3627
+ 2024-03-10 09:09:58,473 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3628
+ 2024-03-10 09:09:58,474 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3629
+ 2024-03-10 09:09:58,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3630
+ 2024-03-10 09:09:58,694 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3631
+ 2024-03-10 09:09:59,313 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: pause
3632
+ 2024-03-10 09:09:59,314 INFO HandlerThread:4057 [handler.py:handle_request_pause():708] stopping system metrics thread
3633
+ 2024-03-10 09:09:59,314 INFO HandlerThread:4057 [system_monitor.py:finish():203] Stopping system monitor
3634
+ 2024-03-10 09:09:59,314 DEBUG SystemMonitor:4057 [system_monitor.py:_start():179] Finished system metrics aggregation loop
3635
+ 2024-03-10 09:09:59,316 INFO HandlerThread:4057 [interfaces.py:finish():202] Joined cpu monitor
3636
+ 2024-03-10 09:09:59,317 DEBUG SystemMonitor:4057 [system_monitor.py:_start():183] Publishing last batch of metrics
3637
+ 2024-03-10 09:09:59,318 INFO HandlerThread:4057 [interfaces.py:finish():202] Joined disk monitor
3638
+ 2024-03-10 09:09:59,328 INFO HandlerThread:4057 [interfaces.py:finish():202] Joined gpu monitor
3639
+ 2024-03-10 09:09:59,329 INFO HandlerThread:4057 [interfaces.py:finish():202] Joined memory monitor
3640
+ 2024-03-10 09:09:59,329 INFO HandlerThread:4057 [interfaces.py:finish():202] Joined network monitor
3641
+ 2024-03-10 09:09:59,329 DEBUG SenderThread:4057 [sender.py:send():378] send: stats
3642
+ 2024-03-10 09:10:04,332 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3643
+ 2024-03-10 09:10:09,333 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3644
+ 2024-03-10 09:10:13,473 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3645
+ 2024-03-10 09:10:13,474 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3646
+ 2024-03-10 09:10:13,538 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3647
+ 2024-03-10 09:10:14,683 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3648
+ 2024-03-10 09:10:16,588 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: resume
3649
+ 2024-03-10 09:10:16,588 INFO HandlerThread:4057 [handler.py:handle_request_resume():699] starting system metrics thread
3650
+ 2024-03-10 09:10:16,589 INFO HandlerThread:4057 [system_monitor.py:start():194] Starting system monitor
3651
+ 2024-03-10 09:10:16,606 INFO SystemMonitor:4057 [system_monitor.py:_start():158] Starting system asset monitoring threads
3652
+ 2024-03-10 09:10:16,616 INFO SystemMonitor:4057 [interfaces.py:start():190] Started cpu monitoring
3653
+ 2024-03-10 09:10:16,617 INFO SystemMonitor:4057 [interfaces.py:start():190] Started disk monitoring
3654
+ 2024-03-10 09:10:16,622 INFO SystemMonitor:4057 [interfaces.py:start():190] Started gpu monitoring
3655
+ 2024-03-10 09:10:16,627 INFO SystemMonitor:4057 [interfaces.py:start():190] Started memory monitoring
3656
+ 2024-03-10 09:10:16,638 INFO SystemMonitor:4057 [interfaces.py:start():190] Started network monitoring
3657
+ 2024-03-10 09:10:19,685 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3658
+ 2024-03-10 09:10:24,686 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3659
+ 2024-03-10 09:10:29,687 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3660
+ 2024-03-10 09:10:33,933 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3661
+ 2024-03-10 09:10:33,935 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3662
+ 2024-03-10 09:10:33,936 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3663
+ 2024-03-10 09:10:35,095 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3664
+ 2024-03-10 09:10:40,096 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3665
+ 2024-03-10 09:10:45,097 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3666
+ 2024-03-10 09:10:48,931 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3667
+ 2024-03-10 09:10:48,933 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3668
+ 2024-03-10 09:10:48,972 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3669
+ 2024-03-10 09:10:50,136 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3670
+ 2024-03-10 09:10:55,137 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3671
+ 2024-03-10 09:11:00,137 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
3672
+ 2024-03-10 09:11:03,931 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: stop_status
3673
+ 2024-03-10 09:11:03,932 DEBUG SenderThread:4057 [sender.py:send_request():405] send_request: stop_status
3674
+ 2024-03-10 09:11:03,972 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: internal_messages
3675
+ 2024-03-10 09:11:06,106 DEBUG HandlerThread:4057 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240310_065024-lx2gw13k/logs/debug.log CHANGED
@@ -28,3 +28,6 @@ config: {}
28
  2024-03-10 06:50:28,340 INFO MainThread:929 [wandb_run.py:_redirect():2186] Redirects installed.
29
  2024-03-10 06:50:28,343 INFO MainThread:929 [wandb_init.py:init():847] run started, returning control to user process
30
  2024-03-10 06:50:28,351 INFO MainThread:929 [wandb_run.py:_config_callback():1343] config_cb None None {'vocab_size': 48000, 'max_position_embeddings': 4096, 'hidden_size': 4096, 'intermediate_size': 14336, 'num_hidden_layers': 48, 'num_attention_heads': 32, 'num_key_value_heads': 8, 'hidden_act': 'silu', 'initializer_range': 0.02, 'rms_norm_eps': 1e-05, 'pretraining_tp': 1, 'use_cache': True, 'rope_theta': 10000.0, 'rope_scaling': None, 'attention_bias': False, 'attention_dropout': 0.0, 'return_dict': True, 'output_hidden_states': False, 'output_attentions': False, 'torchscript': False, 'torch_dtype': 'float16', 'use_bfloat16': False, 'tf_legacy_loss': False, 'pruned_heads': {}, 'tie_word_embeddings': False, 'chunk_size_feed_forward': 0, 'is_encoder_decoder': False, 'is_decoder': False, 'cross_attention_hidden_size': None, 'add_cross_attention': False, 'tie_encoder_decoder': False, 'max_length': 20, 'min_length': 0, 'do_sample': False, 'early_stopping': False, 'num_beams': 1, 'num_beam_groups': 1, 'diversity_penalty': 0.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0, 'typical_p': 1.0, 'repetition_penalty': 1.0, 'length_penalty': 1.0, 'no_repeat_ngram_size': 0, 'encoder_no_repeat_ngram_size': 0, 'bad_words_ids': None, 'num_return_sequences': 1, 'output_scores': False, 'return_dict_in_generate': False, 'forced_bos_token_id': None, 'forced_eos_token_id': None, 'remove_invalid_values': False, 'exponential_decay_length_penalty': None, 'suppress_tokens': None, 'begin_suppress_tokens': None, 'architectures': ['LlamaForCausalLM'], 'finetuning_task': None, 'id2label': {0: 'LABEL_0', 1: 'LABEL_1'}, 'label2id': {'LABEL_0': 0, 'LABEL_1': 1}, 'tokenizer_class': None, 'prefix': None, 'bos_token_id': 1, 'pad_token_id': 2, 'eos_token_id': 32000, 'sep_token_id': None, 'decoder_start_token_id': None, 'task_specific_params': None, 'problem_type': None, '_name_or_path': 'Edentns/DataVortexS-10.7B-dpo-v1.11', 'transformers_version': '4.38.2', 'model_type': 'llama', 'quantization_config': {'quant_method': 'QuantizationMethod.BITS_AND_BYTES', '_load_in_8bit': False, '_load_in_4bit': True, 'llm_int8_threshold': 6.0, 'llm_int8_skip_modules': None, 'llm_int8_enable_fp32_cpu_offload': False, 'llm_int8_has_fp16_weight': False, 'bnb_4bit_quant_type': 'nf4', 'bnb_4bit_use_double_quant': True, 'bnb_4bit_compute_dtype': 'float16', 'load_in_4bit': True, 'load_in_8bit': False}, 'output_dir': '/content/drive/MyDrive/best-one', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'evaluation_strategy': 'no', 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 8, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 1, 'eval_accumulation_steps': None, 'eval_delay': 0, 'learning_rate': 0.0002, 'weight_decay': 0.0, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 0.3, 'num_train_epochs': 4, 'max_steps': -1, 'lr_scheduler_type': 'constant', 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.03, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': '/content/drive/MyDrive/best-one/runs/Mar10_06-48-48_17daf5749447', 'logging_strategy': 'steps', 'logging_first_step': False, 'logging_steps': 500, 'logging_nan_inf_filter': True, 'save_strategy': 'steps', 'save_steps': 100, 'save_total_limit': None, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': False, 'fp16': True, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': '/content/drive/MyDrive/best-one', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True}, 'deepspeed': None, 'label_smoothing_factor': 0.0, 'optim': 'adamw_torch_fused', 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['tensorboard', 'wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': True, 'resume_from_checkpoint': '/content/drive/MyDrive/best-one/checkpoint-1000', 'hub_model_id': None, 'hub_strategy': 'every_save', 'hub_token': '<HUB_TOKEN>', 'hub_private_repo': False, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': '<PUSH_TO_HUB_TOKEN>', 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'dispatch_batches': None, 'split_batches': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None}
 
 
 
 
28
  2024-03-10 06:50:28,340 INFO MainThread:929 [wandb_run.py:_redirect():2186] Redirects installed.
29
  2024-03-10 06:50:28,343 INFO MainThread:929 [wandb_init.py:init():847] run started, returning control to user process
30
  2024-03-10 06:50:28,351 INFO MainThread:929 [wandb_run.py:_config_callback():1343] config_cb None None {'vocab_size': 48000, 'max_position_embeddings': 4096, 'hidden_size': 4096, 'intermediate_size': 14336, 'num_hidden_layers': 48, 'num_attention_heads': 32, 'num_key_value_heads': 8, 'hidden_act': 'silu', 'initializer_range': 0.02, 'rms_norm_eps': 1e-05, 'pretraining_tp': 1, 'use_cache': True, 'rope_theta': 10000.0, 'rope_scaling': None, 'attention_bias': False, 'attention_dropout': 0.0, 'return_dict': True, 'output_hidden_states': False, 'output_attentions': False, 'torchscript': False, 'torch_dtype': 'float16', 'use_bfloat16': False, 'tf_legacy_loss': False, 'pruned_heads': {}, 'tie_word_embeddings': False, 'chunk_size_feed_forward': 0, 'is_encoder_decoder': False, 'is_decoder': False, 'cross_attention_hidden_size': None, 'add_cross_attention': False, 'tie_encoder_decoder': False, 'max_length': 20, 'min_length': 0, 'do_sample': False, 'early_stopping': False, 'num_beams': 1, 'num_beam_groups': 1, 'diversity_penalty': 0.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0, 'typical_p': 1.0, 'repetition_penalty': 1.0, 'length_penalty': 1.0, 'no_repeat_ngram_size': 0, 'encoder_no_repeat_ngram_size': 0, 'bad_words_ids': None, 'num_return_sequences': 1, 'output_scores': False, 'return_dict_in_generate': False, 'forced_bos_token_id': None, 'forced_eos_token_id': None, 'remove_invalid_values': False, 'exponential_decay_length_penalty': None, 'suppress_tokens': None, 'begin_suppress_tokens': None, 'architectures': ['LlamaForCausalLM'], 'finetuning_task': None, 'id2label': {0: 'LABEL_0', 1: 'LABEL_1'}, 'label2id': {'LABEL_0': 0, 'LABEL_1': 1}, 'tokenizer_class': None, 'prefix': None, 'bos_token_id': 1, 'pad_token_id': 2, 'eos_token_id': 32000, 'sep_token_id': None, 'decoder_start_token_id': None, 'task_specific_params': None, 'problem_type': None, '_name_or_path': 'Edentns/DataVortexS-10.7B-dpo-v1.11', 'transformers_version': '4.38.2', 'model_type': 'llama', 'quantization_config': {'quant_method': 'QuantizationMethod.BITS_AND_BYTES', '_load_in_8bit': False, '_load_in_4bit': True, 'llm_int8_threshold': 6.0, 'llm_int8_skip_modules': None, 'llm_int8_enable_fp32_cpu_offload': False, 'llm_int8_has_fp16_weight': False, 'bnb_4bit_quant_type': 'nf4', 'bnb_4bit_use_double_quant': True, 'bnb_4bit_compute_dtype': 'float16', 'load_in_4bit': True, 'load_in_8bit': False}, 'output_dir': '/content/drive/MyDrive/best-one', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'evaluation_strategy': 'no', 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 8, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 1, 'eval_accumulation_steps': None, 'eval_delay': 0, 'learning_rate': 0.0002, 'weight_decay': 0.0, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 0.3, 'num_train_epochs': 4, 'max_steps': -1, 'lr_scheduler_type': 'constant', 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.03, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': '/content/drive/MyDrive/best-one/runs/Mar10_06-48-48_17daf5749447', 'logging_strategy': 'steps', 'logging_first_step': False, 'logging_steps': 500, 'logging_nan_inf_filter': True, 'save_strategy': 'steps', 'save_steps': 100, 'save_total_limit': None, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': False, 'fp16': True, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': '/content/drive/MyDrive/best-one', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True}, 'deepspeed': None, 'label_smoothing_factor': 0.0, 'optim': 'adamw_torch_fused', 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['tensorboard', 'wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': True, 'resume_from_checkpoint': '/content/drive/MyDrive/best-one/checkpoint-1000', 'hub_model_id': None, 'hub_strategy': 'every_save', 'hub_token': '<HUB_TOKEN>', 'hub_private_repo': False, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': '<PUSH_TO_HUB_TOKEN>', 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'dispatch_batches': None, 'split_batches': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None}
31
+ 2024-03-10 09:09:59,311 INFO MainThread:929 [jupyter.py:save_ipynb():373] not saving jupyter notebook
32
+ 2024-03-10 09:09:59,313 INFO MainThread:929 [wandb_init.py:_pause_backend():437] pausing backend
33
+ 2024-03-10 09:10:16,586 INFO MainThread:929 [wandb_init.py:_resume_backend():442] resuming backend