test-time-adaptation-self-certainty-ep1-lr001-llm-adamerge-mistral-7b-instrcut-math-code
/
logs
/save_merged_model_20250703_174232.log
| 2025-07-03 17:42:32 - experiment_save_merged_model - INFO - Starting merged model save process | |
| 2025-07-03 17:42:32 - experiment_save_merged_model - INFO - Arguments: {'lambdas_path': '/work/gj26/b20042/LLM-AdaMerge/outputs/mistral-7b/parameter-wise/test-time-adaptation-self_certainty-lr0.01-ep1/llm_adamerge_parameterwise_lambdas.json', 'model_config': '/work/gj26/b20042/LLM-AdaMerge/src/configs/model_config.yaml', 'output_dir': '/work/gj26/b20042/LLM-AdaMerge/mergekit/outputs/mistral-7b/llmadamerge/test-time-adaptation-self_certainty/ep1-lr0.01', 'model_name': 'merged-model', 'push_to_hub': False, 'hub_repo_id': 'lejelly/test-time-adaptation-self-certainty-ep1-lr001-llm-adamerge-mistral-7b-instrcut-math-code', 'private': False, 'device': 'cuda', 'debug': False} | |
| 2025-07-03 17:42:32 - experiment_save_merged_model - INFO - Loading lambdas from /work/gj26/b20042/LLM-AdaMerge/outputs/mistral-7b/parameter-wise/test-time-adaptation-self_certainty-lr0.01-ep1/llm_adamerge_parameterwise_lambdas.json | |
| 2025-07-03 17:42:32 - experiment_save_merged_model - INFO - Auto-detected parameter-wise merge from JSON structure | |
| 2025-07-03 17:42:32 - experiment_save_merged_model - INFO - Merge type: parameter_wise | |
| 2025-07-03 17:42:32 - experiment_save_merged_model - INFO - [Initial] Memory Usage: | |
| 2025-07-03 17:42:32 - experiment_save_merged_model - INFO - Process: 0.47 GB (0.2%) | |
| 2025-07-03 17:42:32 - experiment_save_merged_model - INFO - System: 7.89 GB / 212.49 GB (8.3%) | |
| 2025-07-03 17:42:32 - experiment_save_merged_model - INFO - Available: 194.78 GB | |
| 2025-07-03 17:42:32 - experiment_save_merged_model - INFO - GPU 0: Allocated: 0.00 GB, Reserved: 0.00 GB, Total: 94.50 GB | |
| 2025-07-03 17:42:32 - experiment_save_merged_model - INFO - Loading models | |
| 2025-07-03 17:42:50 - experiment_save_merged_model - INFO - [After loading models] Memory Usage: | |
| 2025-07-03 17:42:50 - experiment_save_merged_model - INFO - Process: 41.58 GB (19.6%) | |
| 2025-07-03 17:42:50 - experiment_save_merged_model - INFO - System: 49.11 GB / 212.49 GB (31.0%) | |
| 2025-07-03 17:42:50 - experiment_save_merged_model - INFO - Available: 146.55 GB | |
| 2025-07-03 17:42:50 - experiment_save_merged_model - INFO - GPU 0: Allocated: 0.00 GB, Reserved: 0.00 GB, Total: 94.50 GB | |
| 2025-07-03 17:42:50 - experiment_save_merged_model - INFO - Initializing parameter_wise AdaMerge | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Loading learned lambdas | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Deleting original models to free memory (task vectors already computed) | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - [Before deleting models] Memory Usage: | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Process: 95.77 GB (45.1%) | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - System: 90.34 GB / 212.49 GB (50.5%) | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Available: 105.25 GB | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - GPU 0: Allocated: 0.00 GB, Reserved: 0.00 GB, Total: 94.50 GB | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Clearing model_loader references | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Deleting model variables | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Running garbage collection | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - [After deleting models and GC] Memory Usage: | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Process: 56.38 GB (26.5%) | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - System: 64.91 GB / 212.49 GB (38.5%) | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Available: 130.69 GB | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - GPU 0: Allocated: 0.00 GB, Reserved: 0.00 GB, Total: 94.50 GB | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - [After loading lambdas] Memory Usage: | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Process: 56.38 GB (26.5%) | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - System: 64.91 GB / 212.49 GB (38.5%) | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Available: 130.69 GB | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - GPU 0: Allocated: 0.00 GB, Reserved: 0.00 GB, Total: 94.50 GB | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Creating merged model with learned lambdas | |
| 2025-07-03 17:44:05 - experiment_save_merged_model - INFO - Using merge_models_for_save() | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - [After merging models] Memory Usage: | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - Process: 58.07 GB (27.3%) | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - System: 93.91 GB / 212.49 GB (49.0%) | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - Available: 108.40 GB | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - GPU 0: Allocated: 13.49 GB, Reserved: 27.23 GB, Total: 94.50 GB | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - Freeing memory from AdaMerge object (task vectors and base params no longer needed) | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - Deleting task vectors | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - Deleting base params | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - Deleting functional model | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - [After freeing AdaMerge memory] Memory Usage: | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - Process: 6.08 GB (2.9%) | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - System: 27.73 GB / 212.49 GB (17.8%) | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - Available: 174.58 GB | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - GPU 0: Allocated: 13.49 GB, Reserved: 13.62 GB, Total: 94.50 GB | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - Saving merged model to /work/gj26/b20042/LLM-AdaMerge/mergekit/outputs/mistral-7b/llmadamerge/test-time-adaptation-self_certainty/ep1-lr0.01 | |
| 2025-07-03 17:46:06 - experiment_save_merged_model - INFO - Moving merged model to CPU for saving | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - Successfully saved 3 safetensors files: | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - - model-00001-of-00003.safetensors (4714.17 MB) | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - - model-00003-of-00003.safetensors (4330.17 MB) | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - - model-00002-of-00003.safetensors (4768.20 MB) | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - [After saving model] Memory Usage: | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - Process: 15.33 GB (7.2%) | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - System: 23.37 GB / 212.49 GB (18.9%) | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - Available: 172.24 GB | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - GPU 0: Allocated: 0.00 GB, Reserved: 0.00 GB, Total: 94.50 GB | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - Saving tokenizer | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - Copied lambdas file to /work/gj26/b20042/LLM-AdaMerge/mergekit/outputs/mistral-7b/llmadamerge/test-time-adaptation-self_certainty/ep1-lr0.01/learned_lambdas.json | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - Creating model card | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - Cleaning up models | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - [After cleanup] Memory Usage: | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - Process: 15.33 GB (7.2%) | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - System: 23.37 GB / 212.49 GB (18.9%) | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - Available: 172.24 GB | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - GPU 0: Allocated: 0.00 GB, Reserved: 0.00 GB, Total: 94.50 GB | |
| 2025-07-03 17:46:46 - experiment_save_merged_model - INFO - Model saved successfully to /work/gj26/b20042/LLM-AdaMerge/mergekit/outputs/mistral-7b/llmadamerge/test-time-adaptation-self_certainty/ep1-lr0.01 | |