[2025-05-08 14:10:02] Created output directory: train_results_ar/meta-llama_Llama-2-7b-hf_full_upsample1000 [2025-05-08 14:10:02] Chat mode disabled [2025-05-08 14:10:02] Set MODEL_MAX_LENGTH to 4096 for Llama-2 model [2025-05-08 14:10:02] Model size is over 3B (7 B). Using LoRA training. [2025-05-08 14:10:02] Adjusted learning rate for LoRA: 2e-4 [2025-05-08 14:10:02] No QA format data will be used [2025-05-08 14:10:02] ======================================= [2025-05-08 14:10:02] Starting training for model: meta-llama/Llama-2-7b-hf [2025-05-08 14:10:02] ======================================= [2025-05-08 14:10:02] CUDA_VISIBLE_DEVICES: 0,1,2,3,4,5,6,7 [2025-05-08 14:10:02] WANDB_PROJECT: wikidyk-ar [2025-05-08 14:10:02] DATA_PATH: data/wikidyk2022-2025_01082025_gpt-4o_evalv2_pages_formatted_combined_v2.json [2025-05-08 14:10:02] Global Batch Size: 256 [2025-05-08 14:10:02] Data Size: -1 [2025-05-08 14:10:02] Executing command: torchrun --nproc_per_node "8" --master-port 29503 src/train.py --model_name_or_path "meta-llama/Llama-2-7b-hf" --data_path "data/wikidyk2022-2025_01082025_gpt-4o_evalv2_pages_formatted_combined_v2.json" --output_dir "train_results_ar/meta-llama_Llama-2-7b-hf_full_upsample1000" --num_upsample "1000" --per_device_train_batch_size "32" --gradient_accumulation_steps "1" --learning_rate "2e-4" --num_train_epochs "1" --model_max_length "4096" --report_to wandb --logging_steps 50 --save_strategy no --bf16 True --use_flash_attention_2 True --qa_data_ratio "-1" --predict_mask "false" --use_lora --lora_r 32 --lora_alpha 16 [2025-05-08 14:10:02] Training started at 2025年 05月 08日 星期四 14:10:02 CST W0508 14:10:03.166000 3286582 site-packages/torch/distributed/run.py:792] W0508 14:10:03.166000 3286582 site-packages/torch/distributed/run.py:792] ***************************************** W0508 14:10:03.166000 3286582 site-packages/torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W0508 14:10:03.166000 3286582 site-packages/torch/distributed/run.py:792] ***************************************** WARNING:root:Output directory: train_results_ar/meta-llama_Llama-2-7b-hf_full_upsample1000 WARNING:root:Output directory: train_results_ar/meta-llama_Llama-2-7b-hf_full_upsample1000 The model was loaded with use_flash_attention_2=True, which is deprecated and may be removed in a future release. Please use `attn_implementation="flash_attention_2"` instead. [rank5]: Traceback (most recent call last): [rank5]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/train.py", line 134, in [rank5]: train() [rank5]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/train.py", line 81, in train [rank5]: model = load_model( [rank5]: ^^^^^^^^^^^ [rank5]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/utils/tools.py", line 119, in load_model [rank5]: return AutoModelForCausalLM.from_pretrained( [rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank5]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained [rank5]: return model_class.from_pretrained( [rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank5]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 279, in _wrapper [rank5]: return func(*args, **kwargs) [rank5]: ^^^^^^^^^^^^^^^^^^^^^ [rank5]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4336, in from_pretrained [rank5]: config = cls._autoset_attn_implementation( [rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank5]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2109, in _autoset_attn_implementation [rank5]: cls._check_and_enable_flash_attn_2( [rank5]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2252, in _check_and_enable_flash_attn_2 [rank5]: raise ImportError(f"{preface} the package flash_attn seems to be not installed. {install_message}") [rank5]: ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2. WARNING:root:Output directory: train_results_ar/meta-llama_Llama-2-7b-hf_full_upsample1000 WARNING:root:Output directory: train_results_ar/meta-llama_Llama-2-7b-hf_full_upsample1000 WARNING:root:Output directory: train_results_ar/meta-llama_Llama-2-7b-hf_full_upsample1000 WARNING:root:Output directory: train_results_ar/meta-llama_Llama-2-7b-hf_full_upsample1000 WARNING:root:Output directory: train_results_ar/meta-llama_Llama-2-7b-hf_full_upsample1000 WARNING:root:Output directory: train_results_ar/meta-llama_Llama-2-7b-hf_full_upsample1000 The model was loaded with use_flash_attention_2=True, which is deprecated and may be removed in a future release. Please use `attn_implementation="flash_attention_2"` instead. [rank0]: Traceback (most recent call last): [rank0]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/train.py", line 134, in [rank0]: train() [rank0]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/train.py", line 81, in train [rank0]: model = load_model( [rank0]: ^^^^^^^^^^^ [rank0]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/utils/tools.py", line 119, in load_model [rank0]: return AutoModelForCausalLM.from_pretrained( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained [rank0]: return model_class.from_pretrained( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 279, in _wrapper [rank0]: return func(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4336, in from_pretrained [rank0]: config = cls._autoset_attn_implementation( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2109, in _autoset_attn_implementation [rank0]: cls._check_and_enable_flash_attn_2( [rank0]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2252, in _check_and_enable_flash_attn_2 [rank0]: raise ImportError(f"{preface} the package flash_attn seems to be not installed. {install_message}") [rank0]: ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2. The model was loaded with use_flash_attention_2=True, which is deprecated and may be removed in a future release. Please use `attn_implementation="flash_attention_2"` instead. [rank2]: Traceback (most recent call last): [rank2]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/train.py", line 134, in [rank2]: train() [rank2]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/train.py", line 81, in train [rank2]: model = load_model( [rank2]: ^^^^^^^^^^^ [rank2]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/utils/tools.py", line 119, in load_model [rank2]: return AutoModelForCausalLM.from_pretrained( [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained [rank2]: return model_class.from_pretrained( [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 279, in _wrapper [rank2]: return func(*args, **kwargs) [rank2]: ^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4336, in from_pretrained [rank2]: config = cls._autoset_attn_implementation( [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2109, in _autoset_attn_implementation [rank2]: cls._check_and_enable_flash_attn_2( [rank2]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2252, in _check_and_enable_flash_attn_2 [rank2]: raise ImportError(f"{preface} the package flash_attn seems to be not installed. {install_message}") [rank2]: ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2. The model was loaded with use_flash_attention_2=True, which is deprecated and may be removed in a future release. Please use `attn_implementation="flash_attention_2"` instead. [rank7]: Traceback (most recent call last): [rank7]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/train.py", line 134, in [rank7]: train() [rank7]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/train.py", line 81, in train [rank7]: model = load_model( [rank7]: ^^^^^^^^^^^ [rank7]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/utils/tools.py", line 119, in load_model [rank7]: return AutoModelForCausalLM.from_pretrained( [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained [rank7]: return model_class.from_pretrained( [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 279, in _wrapper [rank7]: return func(*args, **kwargs) [rank7]: ^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4336, in from_pretrained [rank7]: config = cls._autoset_attn_implementation( [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2109, in _autoset_attn_implementation [rank7]: cls._check_and_enable_flash_attn_2( [rank7]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2252, in _check_and_enable_flash_attn_2 [rank7]: raise ImportError(f"{preface} the package flash_attn seems to be not installed. {install_message}") [rank7]: ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2. [rank0]:[W508 14:10:17.329577027 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) The model was loaded with use_flash_attention_2=True, which is deprecated and may be removed in a future release. Please use `attn_implementation="flash_attention_2"` instead. [rank4]: Traceback (most recent call last): [rank4]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/train.py", line 134, in [rank4]: train() [rank4]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/train.py", line 81, in train [rank4]: model = load_model( [rank4]: ^^^^^^^^^^^ [rank4]: File "/cq_1/share_1603164/user/wenhaowyu/WikiDYKEvalV2/src/utils/tools.py", line 119, in load_model [rank4]: return AutoModelForCausalLM.from_pretrained( [rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank4]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained [rank4]: return model_class.from_pretrained( [rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank4]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 279, in _wrapper [rank4]: return func(*args, **kwargs) [rank4]: ^^^^^^^^^^^^^^^^^^^^^ [rank4]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4336, in from_pretrained [rank4]: config = cls._autoset_attn_implementation( [rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank4]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2109, in _autoset_attn_implementation [rank4]: cls._check_and_enable_flash_attn_2( [rank4]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2252, in _check_and_enable_flash_attn_2 [rank4]: raise ImportError(f"{preface} the package flash_attn seems to be not installed. {install_message}") [rank4]: ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2. W0508 14:10:17.120000 3286582 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 3286648 closing signal SIGTERM W0508 14:10:17.120000 3286582 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 3286649 closing signal SIGTERM W0508 14:10:17.121000 3286582 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 3286650 closing signal SIGTERM W0508 14:10:17.121000 3286582 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 3286651 closing signal SIGTERM W0508 14:10:17.123000 3286582 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 3286652 closing signal SIGTERM W0508 14:10:17.123000 3286582 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 3286654 closing signal SIGTERM W0508 14:10:17.123000 3286582 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 3286655 closing signal SIGTERM E0508 14:10:18.479000 3286582 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 5 (pid: 3286653) of binary: /root/miniconda3/bin/python Traceback (most recent call last): File "/root/miniconda3/bin/torchrun", line 8, in sys.exit(main()) ^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 918, in main run(args) File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 909, in run elastic_launch( File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 138, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ src/train.py FAILED ------------------------------------------------------------ Failures: ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2025-05-08_14:10:17 host : TENCENT64.site rank : 5 (local_rank: 5) exitcode : 1 (pid: 3286653) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ [2025-05-08 14:10:18] ERROR: Training failed for meta-llama/Llama-2-7b-hf with exit code 1 [2025-05-08 14:10:18] ERROR: Training failed for meta-llama/Llama-2-7b-hf with exit code 1 [2025-05-08 14:10:18] Check error log for details: train_results_ar/meta-llama_Llama-2-7b-hf_full_upsample1000/20250508_141001.log [2025-05-08 14:10:18] Resource usage after training meta-llama/Llama-2-7b-hf: [2025-05-08 14:10:18] GPU memory usage: 0 MiB, 97871 MiB 0 MiB, 97871 MiB 0 MiB, 97871 MiB 0 MiB, 97871 MiB 0 MiB, 97871 MiB 0 MiB, 97871 MiB 0 MiB, 97871 MiB 0 MiB, 97871 MiB [2025-05-08 14:10:18] Disk space usage for model outputs: 52K train_results_ar/meta-llama_Llama-2-7b-hf_full_upsample1000 [2025-05-08 14:10:18]