Kommunarus's picture
Upload folder using huggingface_hub
345acbf verified
[2025-01-31 20:13:33,790][46432] Worker 3 uses CPU cores [6, 7]
[2025-01-31 20:13:33,868][46435] Worker 6 uses CPU cores [12, 13]
[2025-01-31 20:13:33,891][46433] Worker 2 uses CPU cores [4, 5]
[2025-01-31 20:13:33,909][46434] Worker 4 uses CPU cores [8, 9]
[2025-01-31 20:13:34,088][46429] Using GPUs [0] for process 0 (actually maps to GPUs [0])
[2025-01-31 20:13:34,088][46429] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for inference process 0
[2025-01-31 20:13:34,113][46429] Num visible devices: 1
[2025-01-31 20:13:34,119][46430] Worker 0 uses CPU cores [0, 1]
[2025-01-31 20:13:34,141][46431] Worker 1 uses CPU cores [2, 3]
[2025-01-31 20:13:34,147][46416] Using GPUs [0] for process 0 (actually maps to GPUs [0])
[2025-01-31 20:13:34,148][46416] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for learning process 0
[2025-01-31 20:13:34,172][46416] Num visible devices: 1
[2025-01-31 20:13:34,176][46416] Starting seed is not provided
[2025-01-31 20:13:34,176][46416] Using GPUs [0] for process 0 (actually maps to GPUs [0])
[2025-01-31 20:13:34,176][46416] Initializing actor-critic model on device cuda:0
[2025-01-31 20:13:34,177][46416] RunningMeanStd input shape: (3, 72, 128)
[2025-01-31 20:13:34,177][46416] RunningMeanStd input shape: (1,)
[2025-01-31 20:13:34,191][46416] ConvEncoder: input_channels=3
[2025-01-31 20:13:34,219][46437] Worker 5 uses CPU cores [10, 11]
[2025-01-31 20:13:34,288][46436] Worker 7 uses CPU cores [14, 15]
[2025-01-31 20:13:34,315][46416] Conv encoder output size: 512
[2025-01-31 20:13:34,315][46416] Policy head output size: 512
[2025-01-31 20:13:34,330][46416] Created Actor Critic model with architecture:
[2025-01-31 20:13:34,330][46416] ActorCriticSharedWeights(
(obs_normalizer): ObservationNormalizer(
(running_mean_std): RunningMeanStdDictInPlace(
(running_mean_std): ModuleDict(
(obs): RunningMeanStdInPlace()
)
)
)
(returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace)
(encoder): VizdoomEncoder(
(basic_encoder): ConvEncoder(
(enc): RecursiveScriptModule(
original_name=ConvEncoderImpl
(conv_head): RecursiveScriptModule(
original_name=Sequential
(0): RecursiveScriptModule(original_name=Conv2d)
(1): RecursiveScriptModule(original_name=ELU)
(2): RecursiveScriptModule(original_name=Conv2d)
(3): RecursiveScriptModule(original_name=ELU)
(4): RecursiveScriptModule(original_name=Conv2d)
(5): RecursiveScriptModule(original_name=ELU)
)
(mlp_layers): RecursiveScriptModule(
original_name=Sequential
(0): RecursiveScriptModule(original_name=Linear)
(1): RecursiveScriptModule(original_name=ELU)
)
)
)
)
(core): ModelCoreRNN(
(core): GRU(512, 512)
)
(decoder): MlpDecoder(
(mlp): Identity()
)
(critic_linear): Linear(in_features=512, out_features=1, bias=True)
(action_parameterization): ActionParameterizationDefault(
(distribution_linear): Linear(in_features=512, out_features=5, bias=True)
)
)
[2025-01-31 20:13:34,494][46416] Using optimizer <class 'torch.optim.adam.Adam'>
[2025-01-31 20:13:35,209][46416] No checkpoints found
[2025-01-31 20:13:35,209][46416] Did not load from checkpoint, starting from scratch!
[2025-01-31 20:13:35,209][46416] Initialized policy 0 weights for model version 0
[2025-01-31 20:13:35,213][46416] LearnerWorker_p0 finished initialization!
[2025-01-31 20:13:35,213][46416] Using GPUs [0] for process 0 (actually maps to GPUs [0])
[2025-01-31 20:13:35,337][46429] RunningMeanStd input shape: (3, 72, 128)
[2025-01-31 20:13:35,338][46429] RunningMeanStd input shape: (1,)
[2025-01-31 20:13:35,350][46429] ConvEncoder: input_channels=3
[2025-01-31 20:13:35,461][46429] Conv encoder output size: 512
[2025-01-31 20:13:35,461][46429] Policy head output size: 512
[2025-01-31 20:13:35,547][46437] Doom resolution: 160x120, resize resolution: (128, 72)
[2025-01-31 20:13:35,550][46430] Doom resolution: 160x120, resize resolution: (128, 72)
[2025-01-31 20:13:35,556][46433] Doom resolution: 160x120, resize resolution: (128, 72)
[2025-01-31 20:13:35,557][46435] Doom resolution: 160x120, resize resolution: (128, 72)
[2025-01-31 20:13:35,561][46434] Doom resolution: 160x120, resize resolution: (128, 72)
[2025-01-31 20:13:35,568][46431] Doom resolution: 160x120, resize resolution: (128, 72)
[2025-01-31 20:13:35,579][46436] Doom resolution: 160x120, resize resolution: (128, 72)
[2025-01-31 20:13:35,579][46432] Doom resolution: 160x120, resize resolution: (128, 72)
[2025-01-31 20:13:35,925][46433] Decorrelating experience for 0 frames...
[2025-01-31 20:13:35,945][46436] Decorrelating experience for 0 frames...
[2025-01-31 20:13:35,953][46430] Decorrelating experience for 0 frames...
[2025-01-31 20:13:35,953][46437] Decorrelating experience for 0 frames...
[2025-01-31 20:13:35,963][46434] Decorrelating experience for 0 frames...
[2025-01-31 20:13:35,971][46431] Decorrelating experience for 0 frames...
[2025-01-31 20:13:36,308][46433] Decorrelating experience for 32 frames...
[2025-01-31 20:13:36,323][46437] Decorrelating experience for 32 frames...
[2025-01-31 20:13:36,326][46434] Decorrelating experience for 32 frames...
[2025-01-31 20:13:36,332][46435] Decorrelating experience for 0 frames...
[2025-01-31 20:13:36,338][46431] Decorrelating experience for 32 frames...
[2025-01-31 20:13:36,356][46432] Decorrelating experience for 0 frames...
[2025-01-31 20:13:36,373][46436] Decorrelating experience for 32 frames...
[2025-01-31 20:13:36,392][46430] Decorrelating experience for 32 frames...
[2025-01-31 20:13:36,721][46432] Decorrelating experience for 32 frames...
[2025-01-31 20:13:36,746][46431] Decorrelating experience for 64 frames...
[2025-01-31 20:13:36,747][46433] Decorrelating experience for 64 frames...
[2025-01-31 20:13:36,755][46435] Decorrelating experience for 32 frames...
[2025-01-31 20:13:36,823][46436] Decorrelating experience for 64 frames...
[2025-01-31 20:13:37,106][46430] Decorrelating experience for 64 frames...
[2025-01-31 20:13:37,128][46433] Decorrelating experience for 96 frames...
[2025-01-31 20:13:37,164][46434] Decorrelating experience for 64 frames...
[2025-01-31 20:13:37,196][46436] Decorrelating experience for 96 frames...
[2025-01-31 20:13:37,200][46435] Decorrelating experience for 64 frames...
[2025-01-31 20:13:37,440][46437] Decorrelating experience for 64 frames...
[2025-01-31 20:13:37,533][46430] Decorrelating experience for 96 frames...
[2025-01-31 20:13:37,547][46431] Decorrelating experience for 96 frames...
[2025-01-31 20:13:37,586][46434] Decorrelating experience for 96 frames...
[2025-01-31 20:13:37,591][46435] Decorrelating experience for 96 frames...
[2025-01-31 20:13:37,864][46432] Decorrelating experience for 64 frames...
[2025-01-31 20:13:37,877][46437] Decorrelating experience for 96 frames...
[2025-01-31 20:13:38,289][46432] Decorrelating experience for 96 frames...
[2025-01-31 20:13:38,658][46416] Signal inference workers to stop experience collection...
[2025-01-31 20:13:38,665][46429] InferenceWorker_p0-w0: stopping experience collection
[2025-01-31 20:13:41,309][46416] Signal inference workers to resume experience collection...
[2025-01-31 20:13:41,310][46429] InferenceWorker_p0-w0: resuming experience collection
[2025-01-31 20:13:44,232][46429] Updated weights for policy 0, policy_version 10 (0.0219)
[2025-01-31 20:13:47,366][46429] Updated weights for policy 0, policy_version 20 (0.0014)
[2025-01-31 20:13:50,648][46429] Updated weights for policy 0, policy_version 30 (0.0013)
[2025-01-31 20:13:53,577][46416] Saving new best policy, reward=4.509!
[2025-01-31 20:13:53,890][46429] Updated weights for policy 0, policy_version 40 (0.0014)
[2025-01-31 20:13:57,165][46429] Updated weights for policy 0, policy_version 50 (0.0014)
[2025-01-31 20:14:00,354][46429] Updated weights for policy 0, policy_version 60 (0.0012)
[2025-01-31 20:14:03,499][46429] Updated weights for policy 0, policy_version 70 (0.0013)
[2025-01-31 20:14:06,678][46429] Updated weights for policy 0, policy_version 80 (0.0013)
[2025-01-31 20:14:09,805][46429] Updated weights for policy 0, policy_version 90 (0.0014)
[2025-01-31 20:14:12,954][46429] Updated weights for policy 0, policy_version 100 (0.0013)
[2025-01-31 20:14:13,634][46416] Saving new best policy, reward=4.791!
[2025-01-31 20:14:16,207][46429] Updated weights for policy 0, policy_version 110 (0.0015)
[2025-01-31 20:14:19,474][46429] Updated weights for policy 0, policy_version 120 (0.0013)
[2025-01-31 20:14:22,495][46429] Updated weights for policy 0, policy_version 130 (0.0013)
[2025-01-31 20:14:25,795][46429] Updated weights for policy 0, policy_version 140 (0.0013)
[2025-01-31 20:14:28,975][46429] Updated weights for policy 0, policy_version 150 (0.0014)
[2025-01-31 20:14:32,173][46429] Updated weights for policy 0, policy_version 160 (0.0013)
[2025-01-31 20:14:33,578][46416] Saving new best policy, reward=4.804!
[2025-01-31 20:14:35,393][46429] Updated weights for policy 0, policy_version 170 (0.0013)
[2025-01-31 20:14:38,482][46429] Updated weights for policy 0, policy_version 180 (0.0013)
[2025-01-31 20:14:38,583][46416] Saving new best policy, reward=5.109!
[2025-01-31 20:14:41,545][46429] Updated weights for policy 0, policy_version 190 (0.0012)
[2025-01-31 20:14:43,578][46416] Saving new best policy, reward=5.574!
[2025-01-31 20:14:44,782][46429] Updated weights for policy 0, policy_version 200 (0.0014)
[2025-01-31 20:14:47,936][46429] Updated weights for policy 0, policy_version 210 (0.0013)
[2025-01-31 20:14:48,613][46416] Saving new best policy, reward=6.122!
[2025-01-31 20:14:51,163][46429] Updated weights for policy 0, policy_version 220 (0.0013)
[2025-01-31 20:14:54,369][46429] Updated weights for policy 0, policy_version 230 (0.0012)
[2025-01-31 20:14:57,579][46429] Updated weights for policy 0, policy_version 240 (0.0015)
[2025-01-31 20:15:00,685][46429] Updated weights for policy 0, policy_version 250 (0.0012)
[2025-01-31 20:15:03,578][46416] Saving new best policy, reward=6.260!
[2025-01-31 20:15:03,812][46429] Updated weights for policy 0, policy_version 260 (0.0012)
[2025-01-31 20:15:06,989][46429] Updated weights for policy 0, policy_version 270 (0.0013)
[2025-01-31 20:15:08,586][46416] Saving new best policy, reward=6.538!
[2025-01-31 20:15:10,133][46429] Updated weights for policy 0, policy_version 280 (0.0013)
[2025-01-31 20:15:13,201][46429] Updated weights for policy 0, policy_version 290 (0.0013)
[2025-01-31 20:15:13,578][46416] Saving new best policy, reward=7.909!
[2025-01-31 20:15:16,388][46429] Updated weights for policy 0, policy_version 300 (0.0014)
[2025-01-31 20:15:18,606][46416] Saving new best policy, reward=8.551!
[2025-01-31 20:15:19,573][46429] Updated weights for policy 0, policy_version 310 (0.0013)
[2025-01-31 20:15:22,678][46429] Updated weights for policy 0, policy_version 320 (0.0012)
[2025-01-31 20:15:23,578][46416] Saving new best policy, reward=10.038!
[2025-01-31 20:15:25,952][46429] Updated weights for policy 0, policy_version 330 (0.0014)
[2025-01-31 20:15:28,587][46416] Saving /home/neptun/PycharmProjects/RL_course/train_dir/default_experiment/checkpoint_p0/checkpoint_000000338_1384448.pth...
[2025-01-31 20:15:29,205][46429] Updated weights for policy 0, policy_version 340 (0.0014)
[2025-01-31 20:15:32,504][46429] Updated weights for policy 0, policy_version 350 (0.0014)
[2025-01-31 20:15:33,578][46416] Saving new best policy, reward=10.544!
[2025-01-31 20:15:35,771][46429] Updated weights for policy 0, policy_version 360 (0.0013)
[2025-01-31 20:15:38,583][46416] Saving new best policy, reward=10.780!
[2025-01-31 20:15:39,001][46429] Updated weights for policy 0, policy_version 370 (0.0014)
[2025-01-31 20:15:42,201][46429] Updated weights for policy 0, policy_version 380 (0.0014)
[2025-01-31 20:15:43,578][46416] Saving new best policy, reward=11.704!
[2025-01-31 20:15:45,508][46429] Updated weights for policy 0, policy_version 390 (0.0013)
[2025-01-31 20:15:48,582][46416] Saving new best policy, reward=13.792!
[2025-01-31 20:15:48,734][46429] Updated weights for policy 0, policy_version 400 (0.0014)
[2025-01-31 20:15:51,771][46429] Updated weights for policy 0, policy_version 410 (0.0013)
[2025-01-31 20:15:53,578][46416] Saving new best policy, reward=14.734!
[2025-01-31 20:15:55,029][46429] Updated weights for policy 0, policy_version 420 (0.0013)
[2025-01-31 20:15:58,172][46429] Updated weights for policy 0, policy_version 430 (0.0014)
[2025-01-31 20:15:58,582][46416] Saving new best policy, reward=15.724!
[2025-01-31 20:16:01,354][46429] Updated weights for policy 0, policy_version 440 (0.0013)
[2025-01-31 20:16:04,536][46429] Updated weights for policy 0, policy_version 450 (0.0012)
[2025-01-31 20:16:07,723][46429] Updated weights for policy 0, policy_version 460 (0.0014)
[2025-01-31 20:16:08,640][46416] Saving new best policy, reward=15.740!
[2025-01-31 20:16:10,822][46429] Updated weights for policy 0, policy_version 470 (0.0013)
[2025-01-31 20:16:13,579][46416] Saving new best policy, reward=17.087!
[2025-01-31 20:16:14,126][46429] Updated weights for policy 0, policy_version 480 (0.0013)
[2025-01-31 20:16:17,260][46429] Updated weights for policy 0, policy_version 490 (0.0012)
[2025-01-31 20:16:18,585][46416] Saving new best policy, reward=18.021!
[2025-01-31 20:16:20,422][46429] Updated weights for policy 0, policy_version 500 (0.0014)
[2025-01-31 20:16:23,465][46429] Updated weights for policy 0, policy_version 510 (0.0013)
[2025-01-31 20:16:23,579][46416] Saving new best policy, reward=19.312!
[2025-01-31 20:16:26,748][46429] Updated weights for policy 0, policy_version 520 (0.0013)
[2025-01-31 20:16:29,934][46429] Updated weights for policy 0, policy_version 530 (0.0014)
[2025-01-31 20:16:33,082][46429] Updated weights for policy 0, policy_version 540 (0.0013)
[2025-01-31 20:16:33,578][46416] Saving new best policy, reward=19.714!
[2025-01-31 20:16:36,231][46429] Updated weights for policy 0, policy_version 550 (0.0013)
[2025-01-31 20:16:39,448][46429] Updated weights for policy 0, policy_version 560 (0.0015)
[2025-01-31 20:16:42,603][46429] Updated weights for policy 0, policy_version 570 (0.0012)
[2025-01-31 20:16:43,578][46416] Saving new best policy, reward=20.811!
[2025-01-31 20:16:45,765][46429] Updated weights for policy 0, policy_version 580 (0.0014)
[2025-01-31 20:16:48,582][46416] Saving new best policy, reward=22.924!
[2025-01-31 20:16:48,888][46429] Updated weights for policy 0, policy_version 590 (0.0013)
[2025-01-31 20:16:52,060][46429] Updated weights for policy 0, policy_version 600 (0.0013)
[2025-01-31 20:16:55,236][46429] Updated weights for policy 0, policy_version 610 (0.0013)
[2025-01-31 20:16:58,599][46416] Saving new best policy, reward=24.094!
[2025-01-31 20:16:58,602][46429] Updated weights for policy 0, policy_version 620 (0.0014)
[2025-01-31 20:17:01,769][46429] Updated weights for policy 0, policy_version 630 (0.0013)
[2025-01-31 20:17:03,578][46416] Saving new best policy, reward=25.401!
[2025-01-31 20:17:04,949][46429] Updated weights for policy 0, policy_version 640 (0.0014)
[2025-01-31 20:17:08,060][46429] Updated weights for policy 0, policy_version 650 (0.0014)
[2025-01-31 20:17:11,268][46429] Updated weights for policy 0, policy_version 660 (0.0013)
[2025-01-31 20:17:14,343][46429] Updated weights for policy 0, policy_version 670 (0.0014)
[2025-01-31 20:17:17,552][46429] Updated weights for policy 0, policy_version 680 (0.0013)
[2025-01-31 20:17:18,591][46416] Saving new best policy, reward=26.343!
[2025-01-31 20:17:20,770][46429] Updated weights for policy 0, policy_version 690 (0.0013)
[2025-01-31 20:17:23,579][46416] Saving new best policy, reward=26.357!
[2025-01-31 20:17:23,870][46429] Updated weights for policy 0, policy_version 700 (0.0013)
[2025-01-31 20:17:27,039][46429] Updated weights for policy 0, policy_version 710 (0.0012)
[2025-01-31 20:17:28,583][46416] Saving /home/neptun/PycharmProjects/RL_course/train_dir/default_experiment/checkpoint_p0/checkpoint_000000714_2924544.pth...
[2025-01-31 20:17:30,340][46429] Updated weights for policy 0, policy_version 720 (0.0013)
[2025-01-31 20:17:33,487][46429] Updated weights for policy 0, policy_version 730 (0.0013)
[2025-01-31 20:17:36,627][46429] Updated weights for policy 0, policy_version 740 (0.0014)
[2025-01-31 20:17:39,812][46429] Updated weights for policy 0, policy_version 750 (0.0013)
[2025-01-31 20:17:42,885][46429] Updated weights for policy 0, policy_version 760 (0.0013)
[2025-01-31 20:17:46,008][46429] Updated weights for policy 0, policy_version 770 (0.0014)
[2025-01-31 20:17:49,232][46429] Updated weights for policy 0, policy_version 780 (0.0013)
[2025-01-31 20:17:52,321][46429] Updated weights for policy 0, policy_version 790 (0.0013)
[2025-01-31 20:17:55,575][46429] Updated weights for policy 0, policy_version 800 (0.0014)
[2025-01-31 20:17:58,730][46429] Updated weights for policy 0, policy_version 810 (0.0014)
[2025-01-31 20:18:01,923][46429] Updated weights for policy 0, policy_version 820 (0.0013)
[2025-01-31 20:18:05,043][46429] Updated weights for policy 0, policy_version 830 (0.0014)
[2025-01-31 20:18:08,157][46429] Updated weights for policy 0, policy_version 840 (0.0013)
[2025-01-31 20:18:11,258][46429] Updated weights for policy 0, policy_version 850 (0.0013)
[2025-01-31 20:18:14,438][46429] Updated weights for policy 0, policy_version 860 (0.0014)
[2025-01-31 20:18:17,488][46429] Updated weights for policy 0, policy_version 870 (0.0013)
[2025-01-31 20:18:20,750][46429] Updated weights for policy 0, policy_version 880 (0.0013)
[2025-01-31 20:18:23,921][46429] Updated weights for policy 0, policy_version 890 (0.0013)
[2025-01-31 20:18:27,184][46429] Updated weights for policy 0, policy_version 900 (0.0014)
[2025-01-31 20:18:30,373][46429] Updated weights for policy 0, policy_version 910 (0.0015)
[2025-01-31 20:18:33,523][46429] Updated weights for policy 0, policy_version 920 (0.0012)
[2025-01-31 20:18:36,624][46429] Updated weights for policy 0, policy_version 930 (0.0013)
[2025-01-31 20:18:39,829][46429] Updated weights for policy 0, policy_version 940 (0.0013)
[2025-01-31 20:18:43,000][46429] Updated weights for policy 0, policy_version 950 (0.0014)
[2025-01-31 20:18:46,093][46429] Updated weights for policy 0, policy_version 960 (0.0013)
[2025-01-31 20:18:48,585][46416] Saving new best policy, reward=26.837!
[2025-01-31 20:18:49,276][46429] Updated weights for policy 0, policy_version 970 (0.0013)
[2025-01-31 20:18:51,780][46416] Saving /home/neptun/PycharmProjects/RL_course/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth...
[2025-01-31 20:18:51,780][46416] Stopping Batcher_0...
[2025-01-31 20:18:51,788][46416] Loop batcher_evt_loop terminating...
[2025-01-31 20:18:51,808][46429] Weights refcount: 2 0
[2025-01-31 20:18:51,810][46429] Stopping InferenceWorker_p0-w0...
[2025-01-31 20:18:51,811][46429] Loop inference_proc0-0_evt_loop terminating...
[2025-01-31 20:18:51,839][46434] Stopping RolloutWorker_w4...
[2025-01-31 20:18:51,839][46434] Loop rollout_proc4_evt_loop terminating...
[2025-01-31 20:18:51,841][46433] Stopping RolloutWorker_w2...
[2025-01-31 20:18:51,842][46433] Loop rollout_proc2_evt_loop terminating...
[2025-01-31 20:18:51,847][46437] Stopping RolloutWorker_w5...
[2025-01-31 20:18:51,848][46437] Loop rollout_proc5_evt_loop terminating...
[2025-01-31 20:18:51,848][46435] Stopping RolloutWorker_w6...
[2025-01-31 20:18:51,849][46435] Loop rollout_proc6_evt_loop terminating...
[2025-01-31 20:18:51,850][46430] Stopping RolloutWorker_w0...
[2025-01-31 20:18:51,851][46430] Loop rollout_proc0_evt_loop terminating...
[2025-01-31 20:18:51,855][46431] Stopping RolloutWorker_w1...
[2025-01-31 20:18:51,856][46436] Stopping RolloutWorker_w7...
[2025-01-31 20:18:51,856][46431] Loop rollout_proc1_evt_loop terminating...
[2025-01-31 20:18:51,857][46436] Loop rollout_proc7_evt_loop terminating...
[2025-01-31 20:18:51,858][46432] Stopping RolloutWorker_w3...
[2025-01-31 20:18:51,858][46432] Loop rollout_proc3_evt_loop terminating...
[2025-01-31 20:18:51,891][46416] Removing /home/neptun/PycharmProjects/RL_course/train_dir/default_experiment/checkpoint_p0/checkpoint_000000338_1384448.pth
[2025-01-31 20:18:51,908][46416] Saving new best policy, reward=27.484!
[2025-01-31 20:18:52,110][46416] Saving /home/neptun/PycharmProjects/RL_course/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth...
[2025-01-31 20:18:52,383][46416] Stopping LearnerWorker_p0...
[2025-01-31 20:18:52,383][46416] Loop learner_proc0_evt_loop terminating...