Joosep Pata
added CLIC 2nd and 3rd retraining
d35efbc
[2025-03-09 17:37:56,979] INFO: Will use single-gpu: NVIDIA A100 80GB PCIe
[2025-03-09 17:37:56,979] INFO: configured dtype=torch.bfloat16 for autocast
[2025-03-09 17:37:56,979] INFO: configured dtype=torch.bfloat16 for autocast
[2025-03-09 17:37:57,065] INFO: layer conv_id_0 using attention_type=flash
[2025-03-09 17:37:57,065] INFO: layer conv_id_0 using attention_type=flash
[2025-03-09 17:37:57,096] INFO: layer conv_reg_0 using attention_type=flash
[2025-03-09 17:37:57,096] INFO: layer conv_reg_0 using attention_type=flash
[2025-03-09 17:37:57,128] INFO: layer conv_id_1 using attention_type=flash
[2025-03-09 17:37:57,128] INFO: layer conv_id_1 using attention_type=flash
[2025-03-09 17:37:57,160] INFO: layer conv_reg_1 using attention_type=flash
[2025-03-09 17:37:57,160] INFO: layer conv_reg_1 using attention_type=flash
[2025-03-09 17:37:57,192] INFO: layer conv_id_2 using attention_type=flash
[2025-03-09 17:37:57,192] INFO: layer conv_id_2 using attention_type=flash
[2025-03-09 17:37:57,223] INFO: layer conv_reg_2 using attention_type=flash
[2025-03-09 17:37:57,223] INFO: layer conv_reg_2 using attention_type=flash
[2025-03-09 17:37:57,522] INFO: MLPF(
(nn0_id): ModuleList(
(0-1): 2 x Sequential(
(0): Linear(in_features=17, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=1024, bias=True)
)
)
(nn0_reg): ModuleList(
(0-1): 2 x Sequential(
(0): Linear(in_features=17, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=1024, bias=True)
)
)
(conv_id): ModuleList(
(0-2): 3 x PreLnSelfAttentionLayer(
(mha): MultiheadAttention(
(out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
)
(norm0): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(norm1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(seq): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): Linear(in_features=1024, out_features=1024, bias=True)
(3): ReLU()
)
(dropout): Dropout(p=0.0, inplace=False)
)
)
(conv_reg): ModuleList(
(0-2): 3 x PreLnSelfAttentionLayer(
(mha): MultiheadAttention(
(out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
)
(norm0): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(norm1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(seq): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): Linear(in_features=1024, out_features=1024, bias=True)
(3): ReLU()
)
(dropout): Dropout(p=0.0, inplace=False)
)
)
(nn_binary_particle): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=2, bias=True)
)
(nn_pid): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=6, bias=True)
)
(nn_pu): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=1, bias=True)
)
(nn_pt): RegressionOutput(
(nn): ModuleList(
(0-1): 2 x Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=1, bias=True)
)
)
)
(nn_eta): RegressionOutput(
(nn): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=2, bias=True)
)
)
(nn_sin_phi): RegressionOutput(
(nn): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=2, bias=True)
)
)
(nn_cos_phi): RegressionOutput(
(nn): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=2, bias=True)
)
)
(nn_energy): RegressionOutput(
(nn): ModuleList(
(0-1): 2 x Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=1, bias=True)
)
)
)
(final_norm_id): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(final_norm_reg): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
)
[2025-03-09 17:37:57,522] INFO: MLPF(
(nn0_id): ModuleList(
(0-1): 2 x Sequential(
(0): Linear(in_features=17, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=1024, bias=True)
)
)
(nn0_reg): ModuleList(
(0-1): 2 x Sequential(
(0): Linear(in_features=17, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=1024, bias=True)
)
)
(conv_id): ModuleList(
(0-2): 3 x PreLnSelfAttentionLayer(
(mha): MultiheadAttention(
(out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
)
(norm0): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(norm1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(seq): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): Linear(in_features=1024, out_features=1024, bias=True)
(3): ReLU()
)
(dropout): Dropout(p=0.0, inplace=False)
)
)
(conv_reg): ModuleList(
(0-2): 3 x PreLnSelfAttentionLayer(
(mha): MultiheadAttention(
(out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
)
(norm0): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(norm1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(seq): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): Linear(in_features=1024, out_features=1024, bias=True)
(3): ReLU()
)
(dropout): Dropout(p=0.0, inplace=False)
)
)
(nn_binary_particle): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=2, bias=True)
)
(nn_pid): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=6, bias=True)
)
(nn_pu): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=1, bias=True)
)
(nn_pt): RegressionOutput(
(nn): ModuleList(
(0-1): 2 x Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=1, bias=True)
)
)
)
(nn_eta): RegressionOutput(
(nn): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=2, bias=True)
)
)
(nn_sin_phi): RegressionOutput(
(nn): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=2, bias=True)
)
)
(nn_cos_phi): RegressionOutput(
(nn): Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=2, bias=True)
)
)
(nn_energy): RegressionOutput(
(nn): ModuleList(
(0-1): 2 x Sequential(
(0): Linear(in_features=1024, out_features=1024, bias=True)
(1): ReLU()
(2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(3): Dropout(p=0.0, inplace=False)
(4): Linear(in_features=1024, out_features=1, bias=True)
)
)
)
(final_norm_id): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(final_norm_reg): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
)
[2025-03-09 17:37:57,523] INFO: Trainable parameters: 52630547
[2025-03-09 17:37:57,523] INFO: Trainable parameters: 52630547
[2025-03-09 17:37:57,523] INFO: Non-trainable parameters: 0
[2025-03-09 17:37:57,523] INFO: Non-trainable parameters: 0
[2025-03-09 17:37:57,523] INFO: Total parameters: 52630547
[2025-03-09 17:37:57,523] INFO: Total parameters: 52630547
[2025-03-09 17:37:57,525] INFO: Modules Trainable parameters Non-trainable parameters
nn0_id.0.0.weight 17408 0
nn0_id.0.0.bias 1024 0
nn0_id.0.2.weight 1024 0
nn0_id.0.2.bias 1024 0
nn0_id.0.4.weight 1048576 0
nn0_id.0.4.bias 1024 0
nn0_id.1.0.weight 17408 0
nn0_id.1.0.bias 1024 0
nn0_id.1.2.weight 1024 0
nn0_id.1.2.bias 1024 0
nn0_id.1.4.weight 1048576 0
nn0_id.1.4.bias 1024 0
nn0_reg.0.0.weight 17408 0
nn0_reg.0.0.bias 1024 0
nn0_reg.0.2.weight 1024 0
nn0_reg.0.2.bias 1024 0
nn0_reg.0.4.weight 1048576 0
nn0_reg.0.4.bias 1024 0
nn0_reg.1.0.weight 17408 0
nn0_reg.1.0.bias 1024 0
nn0_reg.1.2.weight 1024 0
nn0_reg.1.2.bias 1024 0
nn0_reg.1.4.weight 1048576 0
nn0_reg.1.4.bias 1024 0
conv_id.0.mha.in_proj_weight 3145728 0
conv_id.0.mha.in_proj_bias 3072 0
conv_id.0.mha.out_proj.weight 1048576 0
conv_id.0.mha.out_proj.bias 1024 0
conv_id.0.norm0.weight 1024 0
conv_id.0.norm0.bias 1024 0
conv_id.0.norm1.weight 1024 0
conv_id.0.norm1.bias 1024 0
conv_id.0.seq.0.weight 1048576 0
conv_id.0.seq.0.bias 1024 0
conv_id.0.seq.2.weight 1048576 0
conv_id.0.seq.2.bias 1024 0
conv_id.1.mha.in_proj_weight 3145728 0
conv_id.1.mha.in_proj_bias 3072 0
conv_id.1.mha.out_proj.weight 1048576 0
conv_id.1.mha.out_proj.bias 1024 0
conv_id.1.norm0.weight 1024 0
conv_id.1.norm0.bias 1024 0
conv_id.1.norm1.weight 1024 0
conv_id.1.norm1.bias 1024 0
conv_id.1.seq.0.weight 1048576 0
conv_id.1.seq.0.bias 1024 0
conv_id.1.seq.2.weight 1048576 0
conv_id.1.seq.2.bias 1024 0
conv_id.2.mha.in_proj_weight 3145728 0
conv_id.2.mha.in_proj_bias 3072 0
conv_id.2.mha.out_proj.weight 1048576 0
conv_id.2.mha.out_proj.bias 1024 0
conv_id.2.norm0.weight 1024 0
conv_id.2.norm0.bias 1024 0
conv_id.2.norm1.weight 1024 0
conv_id.2.norm1.bias 1024 0
conv_id.2.seq.0.weight 1048576 0
conv_id.2.seq.0.bias 1024 0
conv_id.2.seq.2.weight 1048576 0
conv_id.2.seq.2.bias 1024 0
conv_reg.0.mha.in_proj_weight 3145728 0
conv_reg.0.mha.in_proj_bias 3072 0
conv_reg.0.mha.out_proj.weight 1048576 0
conv_reg.0.mha.out_proj.bias 1024 0
conv_reg.0.norm0.weight 1024 0
conv_reg.0.norm0.bias 1024 0
conv_reg.0.norm1.weight 1024 0
conv_reg.0.norm1.bias 1024 0
conv_reg.0.seq.0.weight 1048576 0
conv_reg.0.seq.0.bias 1024 0
conv_reg.0.seq.2.weight 1048576 0
conv_reg.0.seq.2.bias 1024 0
conv_reg.1.mha.in_proj_weight 3145728 0
conv_reg.1.mha.in_proj_bias 3072 0
conv_reg.1.mha.out_proj.weight 1048576 0
conv_reg.1.mha.out_proj.bias 1024 0
conv_reg.1.norm0.weight 1024 0
conv_reg.1.norm0.bias 1024 0
conv_reg.1.norm1.weight 1024 0
conv_reg.1.norm1.bias 1024 0
conv_reg.1.seq.0.weight 1048576 0
conv_reg.1.seq.0.bias 1024 0
conv_reg.1.seq.2.weight 1048576 0
conv_reg.1.seq.2.bias 1024 0
conv_reg.2.mha.in_proj_weight 3145728 0
conv_reg.2.mha.in_proj_bias 3072 0
conv_reg.2.mha.out_proj.weight 1048576 0
conv_reg.2.mha.out_proj.bias 1024 0
conv_reg.2.norm0.weight 1024 0
conv_reg.2.norm0.bias 1024 0
conv_reg.2.norm1.weight 1024 0
conv_reg.2.norm1.bias 1024 0
conv_reg.2.seq.0.weight 1048576 0
conv_reg.2.seq.0.bias 1024 0
conv_reg.2.seq.2.weight 1048576 0
conv_reg.2.seq.2.bias 1024 0
nn_binary_particle.0.weight 1048576 0
nn_binary_particle.0.bias 1024 0
nn_binary_particle.2.weight 1024 0
nn_binary_particle.2.bias 1024 0
nn_binary_particle.4.weight 2048 0
nn_binary_particle.4.bias 2 0
nn_pid.0.weight 1048576 0
nn_pid.0.bias 1024 0
nn_pid.2.weight 1024 0
nn_pid.2.bias 1024 0
nn_pid.4.weight 6144 0
nn_pid.4.bias 6 0
nn_pu.0.weight 1048576 0
nn_pu.0.bias 1024 0
nn_pu.2.weight 1024 0
nn_pu.2.bias 1024 0
nn_pu.4.weight 1024 0
nn_pu.4.bias 1 0
nn_pt.nn.0.0.weight 1048576 0
nn_pt.nn.0.0.bias 1024 0
nn_pt.nn.0.2.weight 1024 0
nn_pt.nn.0.2.bias 1024 0
nn_pt.nn.0.4.weight 1024 0
nn_pt.nn.0.4.bias 1 0
nn_pt.nn.1.0.weight 1048576 0
nn_pt.nn.1.0.bias 1024 0
nn_pt.nn.1.2.weight 1024 0
nn_pt.nn.1.2.bias 1024 0
nn_pt.nn.1.4.weight 1024 0
nn_pt.nn.1.4.bias 1 0
nn_eta.nn.0.weight 1048576 0
nn_eta.nn.0.bias 1024 0
nn_eta.nn.2.weight 1024 0
nn_eta.nn.2.bias 1024 0
nn_eta.nn.4.weight 2048 0
nn_eta.nn.4.bias 2 0
nn_sin_phi.nn.0.weight 1048576 0
nn_sin_phi.nn.0.bias 1024 0
nn_sin_phi.nn.2.weight 1024 0
nn_sin_phi.nn.2.bias 1024 0
nn_sin_phi.nn.4.weight 2048 0
nn_sin_phi.nn.4.bias 2 0
nn_cos_phi.nn.0.weight 1048576 0
nn_cos_phi.nn.0.bias 1024 0
nn_cos_phi.nn.2.weight 1024 0
nn_cos_phi.nn.2.bias 1024 0
nn_cos_phi.nn.4.weight 2048 0
nn_cos_phi.nn.4.bias 2 0
nn_energy.nn.0.0.weight 1048576 0
nn_energy.nn.0.0.bias 1024 0
nn_energy.nn.0.2.weight 1024 0
nn_energy.nn.0.2.bias 1024 0
nn_energy.nn.0.4.weight 1024 0
nn_energy.nn.0.4.bias 1 0
nn_energy.nn.1.0.weight 1048576 0
nn_energy.nn.1.0.bias 1024 0
nn_energy.nn.1.2.weight 1024 0
nn_energy.nn.1.2.bias 1024 0
nn_energy.nn.1.4.weight 1024 0
nn_energy.nn.1.4.bias 1 0
final_norm_id.weight 1024 0
final_norm_id.bias 1024 0
final_norm_reg.weight 1024 0
final_norm_reg.bias 1024 0
[2025-03-09 17:37:57,525] INFO: Modules Trainable parameters Non-trainable parameters
nn0_id.0.0.weight 17408 0
nn0_id.0.0.bias 1024 0
nn0_id.0.2.weight 1024 0
nn0_id.0.2.bias 1024 0
nn0_id.0.4.weight 1048576 0
nn0_id.0.4.bias 1024 0
nn0_id.1.0.weight 17408 0
nn0_id.1.0.bias 1024 0
nn0_id.1.2.weight 1024 0
nn0_id.1.2.bias 1024 0
nn0_id.1.4.weight 1048576 0
nn0_id.1.4.bias 1024 0
nn0_reg.0.0.weight 17408 0
nn0_reg.0.0.bias 1024 0
nn0_reg.0.2.weight 1024 0
nn0_reg.0.2.bias 1024 0
nn0_reg.0.4.weight 1048576 0
nn0_reg.0.4.bias 1024 0
nn0_reg.1.0.weight 17408 0
nn0_reg.1.0.bias 1024 0
nn0_reg.1.2.weight 1024 0
nn0_reg.1.2.bias 1024 0
nn0_reg.1.4.weight 1048576 0
nn0_reg.1.4.bias 1024 0
conv_id.0.mha.in_proj_weight 3145728 0
conv_id.0.mha.in_proj_bias 3072 0
conv_id.0.mha.out_proj.weight 1048576 0
conv_id.0.mha.out_proj.bias 1024 0
conv_id.0.norm0.weight 1024 0
conv_id.0.norm0.bias 1024 0
conv_id.0.norm1.weight 1024 0
conv_id.0.norm1.bias 1024 0
conv_id.0.seq.0.weight 1048576 0
conv_id.0.seq.0.bias 1024 0
conv_id.0.seq.2.weight 1048576 0
conv_id.0.seq.2.bias 1024 0
conv_id.1.mha.in_proj_weight 3145728 0
conv_id.1.mha.in_proj_bias 3072 0
conv_id.1.mha.out_proj.weight 1048576 0
conv_id.1.mha.out_proj.bias 1024 0
conv_id.1.norm0.weight 1024 0
conv_id.1.norm0.bias 1024 0
conv_id.1.norm1.weight 1024 0
conv_id.1.norm1.bias 1024 0
conv_id.1.seq.0.weight 1048576 0
conv_id.1.seq.0.bias 1024 0
conv_id.1.seq.2.weight 1048576 0
conv_id.1.seq.2.bias 1024 0
conv_id.2.mha.in_proj_weight 3145728 0
conv_id.2.mha.in_proj_bias 3072 0
conv_id.2.mha.out_proj.weight 1048576 0
conv_id.2.mha.out_proj.bias 1024 0
conv_id.2.norm0.weight 1024 0
conv_id.2.norm0.bias 1024 0
conv_id.2.norm1.weight 1024 0
conv_id.2.norm1.bias 1024 0
conv_id.2.seq.0.weight 1048576 0
conv_id.2.seq.0.bias 1024 0
conv_id.2.seq.2.weight 1048576 0
conv_id.2.seq.2.bias 1024 0
conv_reg.0.mha.in_proj_weight 3145728 0
conv_reg.0.mha.in_proj_bias 3072 0
conv_reg.0.mha.out_proj.weight 1048576 0
conv_reg.0.mha.out_proj.bias 1024 0
conv_reg.0.norm0.weight 1024 0
conv_reg.0.norm0.bias 1024 0
conv_reg.0.norm1.weight 1024 0
conv_reg.0.norm1.bias 1024 0
conv_reg.0.seq.0.weight 1048576 0
conv_reg.0.seq.0.bias 1024 0
conv_reg.0.seq.2.weight 1048576 0
conv_reg.0.seq.2.bias 1024 0
conv_reg.1.mha.in_proj_weight 3145728 0
conv_reg.1.mha.in_proj_bias 3072 0
conv_reg.1.mha.out_proj.weight 1048576 0
conv_reg.1.mha.out_proj.bias 1024 0
conv_reg.1.norm0.weight 1024 0
conv_reg.1.norm0.bias 1024 0
conv_reg.1.norm1.weight 1024 0
conv_reg.1.norm1.bias 1024 0
conv_reg.1.seq.0.weight 1048576 0
conv_reg.1.seq.0.bias 1024 0
conv_reg.1.seq.2.weight 1048576 0
conv_reg.1.seq.2.bias 1024 0
conv_reg.2.mha.in_proj_weight 3145728 0
conv_reg.2.mha.in_proj_bias 3072 0
conv_reg.2.mha.out_proj.weight 1048576 0
conv_reg.2.mha.out_proj.bias 1024 0
conv_reg.2.norm0.weight 1024 0
conv_reg.2.norm0.bias 1024 0
conv_reg.2.norm1.weight 1024 0
conv_reg.2.norm1.bias 1024 0
conv_reg.2.seq.0.weight 1048576 0
conv_reg.2.seq.0.bias 1024 0
conv_reg.2.seq.2.weight 1048576 0
conv_reg.2.seq.2.bias 1024 0
nn_binary_particle.0.weight 1048576 0
nn_binary_particle.0.bias 1024 0
nn_binary_particle.2.weight 1024 0
nn_binary_particle.2.bias 1024 0
nn_binary_particle.4.weight 2048 0
nn_binary_particle.4.bias 2 0
nn_pid.0.weight 1048576 0
nn_pid.0.bias 1024 0
nn_pid.2.weight 1024 0
nn_pid.2.bias 1024 0
nn_pid.4.weight 6144 0
nn_pid.4.bias 6 0
nn_pu.0.weight 1048576 0
nn_pu.0.bias 1024 0
nn_pu.2.weight 1024 0
nn_pu.2.bias 1024 0
nn_pu.4.weight 1024 0
nn_pu.4.bias 1 0
nn_pt.nn.0.0.weight 1048576 0
nn_pt.nn.0.0.bias 1024 0
nn_pt.nn.0.2.weight 1024 0
nn_pt.nn.0.2.bias 1024 0
nn_pt.nn.0.4.weight 1024 0
nn_pt.nn.0.4.bias 1 0
nn_pt.nn.1.0.weight 1048576 0
nn_pt.nn.1.0.bias 1024 0
nn_pt.nn.1.2.weight 1024 0
nn_pt.nn.1.2.bias 1024 0
nn_pt.nn.1.4.weight 1024 0
nn_pt.nn.1.4.bias 1 0
nn_eta.nn.0.weight 1048576 0
nn_eta.nn.0.bias 1024 0
nn_eta.nn.2.weight 1024 0
nn_eta.nn.2.bias 1024 0
nn_eta.nn.4.weight 2048 0
nn_eta.nn.4.bias 2 0
nn_sin_phi.nn.0.weight 1048576 0
nn_sin_phi.nn.0.bias 1024 0
nn_sin_phi.nn.2.weight 1024 0
nn_sin_phi.nn.2.bias 1024 0
nn_sin_phi.nn.4.weight 2048 0
nn_sin_phi.nn.4.bias 2 0
nn_cos_phi.nn.0.weight 1048576 0
nn_cos_phi.nn.0.bias 1024 0
nn_cos_phi.nn.2.weight 1024 0
nn_cos_phi.nn.2.bias 1024 0
nn_cos_phi.nn.4.weight 2048 0
nn_cos_phi.nn.4.bias 2 0
nn_energy.nn.0.0.weight 1048576 0
nn_energy.nn.0.0.bias 1024 0
nn_energy.nn.0.2.weight 1024 0
nn_energy.nn.0.2.bias 1024 0
nn_energy.nn.0.4.weight 1024 0
nn_energy.nn.0.4.bias 1 0
nn_energy.nn.1.0.weight 1048576 0
nn_energy.nn.1.0.bias 1024 0
nn_energy.nn.1.2.weight 1024 0
nn_energy.nn.1.2.bias 1024 0
nn_energy.nn.1.4.weight 1024 0
nn_energy.nn.1.4.bias 1 0
final_norm_id.weight 1024 0
final_norm_id.bias 1024 0
final_norm_reg.weight 1024 0
final_norm_reg.bias 1024 0
[2025-03-09 17:37:57,526] INFO: Creating experiment dir experiments/pyg-clic_20250309_173756_957486
[2025-03-09 17:37:57,526] INFO: Creating experiment dir experiments/pyg-clic_20250309_173756_957486
[2025-03-09 17:37:57,526] INFO: Model directory experiments/pyg-clic_20250309_173756_957486
[2025-03-09 17:37:57,526] INFO: Model directory experiments/pyg-clic_20250309_173756_957486
[2025-03-09 17:38:02,881] INFO: train_dataset: clic_edm_qq_pf, 719492
[2025-03-09 17:38:02,881] INFO: train_dataset: clic_edm_qq_pf, 719492
[2025-03-09 17:38:02,899] INFO: train_dataset: clic_edm_qq_pf, 719490
[2025-03-09 17:38:02,899] INFO: train_dataset: clic_edm_qq_pf, 719490
[2025-03-09 17:38:02,918] INFO: train_dataset: clic_edm_qq_pf, 719489
[2025-03-09 17:38:02,918] INFO: train_dataset: clic_edm_qq_pf, 719489
[2025-03-09 17:38:02,937] INFO: train_dataset: clic_edm_qq_pf, 719515
[2025-03-09 17:38:02,937] INFO: train_dataset: clic_edm_qq_pf, 719515
[2025-03-09 17:38:02,957] INFO: train_dataset: clic_edm_qq_pf, 719510
[2025-03-09 17:38:02,957] INFO: train_dataset: clic_edm_qq_pf, 719510
[2025-03-09 17:38:02,976] INFO: train_dataset: clic_edm_qq_pf, 719503
[2025-03-09 17:38:02,976] INFO: train_dataset: clic_edm_qq_pf, 719503
[2025-03-09 17:38:02,994] INFO: train_dataset: clic_edm_qq_pf, 719509
[2025-03-09 17:38:02,994] INFO: train_dataset: clic_edm_qq_pf, 719509
[2025-03-09 17:38:03,010] INFO: train_dataset: clic_edm_qq_pf, 719484
[2025-03-09 17:38:03,010] INFO: train_dataset: clic_edm_qq_pf, 719484
[2025-03-09 17:38:03,023] INFO: train_dataset: clic_edm_qq_pf, 719474
[2025-03-09 17:38:03,023] INFO: train_dataset: clic_edm_qq_pf, 719474
[2025-03-09 17:38:03,038] INFO: train_dataset: clic_edm_qq_pf, 720386
[2025-03-09 17:38:03,038] INFO: train_dataset: clic_edm_qq_pf, 720386
[2025-03-09 17:38:03,066] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,066] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,097] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,097] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,124] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,124] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,152] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,152] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,184] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,184] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,213] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,213] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,242] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,242] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,266] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,266] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,298] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,298] INFO: train_dataset: clic_edm_ttbar_pf, 713900
[2025-03-09 17:38:03,327] INFO: train_dataset: clic_edm_ttbar_pf, 714700
[2025-03-09 17:38:03,327] INFO: train_dataset: clic_edm_ttbar_pf, 714700
[2025-03-09 17:38:03,345] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,345] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,361] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,361] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,379] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,379] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,398] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,398] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,417] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,417] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,442] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,442] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,473] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,473] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,491] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,491] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,509] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,509] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720000
[2025-03-09 17:38:03,526] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720700
[2025-03-09 17:38:03,526] INFO: train_dataset: clic_edm_ww_fullhad_pf, 720700
[2025-03-09 17:38:04,840] INFO: valid_dataset: clic_edm_qq_pf, 79948
[2025-03-09 17:38:04,840] INFO: valid_dataset: clic_edm_qq_pf, 79948
[2025-03-09 17:38:04,845] INFO: valid_dataset: clic_edm_qq_pf, 79950
[2025-03-09 17:38:04,845] INFO: valid_dataset: clic_edm_qq_pf, 79950
[2025-03-09 17:38:04,850] INFO: valid_dataset: clic_edm_qq_pf, 79939
[2025-03-09 17:38:04,850] INFO: valid_dataset: clic_edm_qq_pf, 79939
[2025-03-09 17:38:04,855] INFO: valid_dataset: clic_edm_qq_pf, 79939
[2025-03-09 17:38:04,855] INFO: valid_dataset: clic_edm_qq_pf, 79939
[2025-03-09 17:38:04,994] INFO: valid_dataset: clic_edm_qq_pf, 79950
[2025-03-09 17:38:04,994] INFO: valid_dataset: clic_edm_qq_pf, 79950
[2025-03-09 17:38:04,999] INFO: valid_dataset: clic_edm_qq_pf, 79950
[2025-03-09 17:38:04,999] INFO: valid_dataset: clic_edm_qq_pf, 79950
[2025-03-09 17:38:05,006] INFO: valid_dataset: clic_edm_qq_pf, 79938
[2025-03-09 17:38:05,006] INFO: valid_dataset: clic_edm_qq_pf, 79938
[2025-03-09 17:38:05,011] INFO: valid_dataset: clic_edm_qq_pf, 79957
[2025-03-09 17:38:05,011] INFO: valid_dataset: clic_edm_qq_pf, 79957
[2025-03-09 17:38:05,016] INFO: valid_dataset: clic_edm_qq_pf, 79955
[2025-03-09 17:38:05,016] INFO: valid_dataset: clic_edm_qq_pf, 79955
[2025-03-09 17:38:05,021] INFO: valid_dataset: clic_edm_qq_pf, 80035
[2025-03-09 17:38:05,021] INFO: valid_dataset: clic_edm_qq_pf, 80035
[2025-03-09 17:38:05,028] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,028] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,034] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,034] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,041] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,041] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,048] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,048] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,055] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,055] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,061] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,061] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,068] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,068] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,076] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,076] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,083] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,083] INFO: valid_dataset: clic_edm_ttbar_pf, 79300
[2025-03-09 17:38:05,090] INFO: valid_dataset: clic_edm_ttbar_pf, 79700
[2025-03-09 17:38:05,090] INFO: valid_dataset: clic_edm_ttbar_pf, 79700
[2025-03-09 17:38:05,096] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,096] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,102] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,102] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,108] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,108] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,114] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,114] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,120] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,120] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,125] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,125] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,131] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,131] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,136] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,136] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,142] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,142] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80000
[2025-03-09 17:38:05,147] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80100
[2025-03-09 17:38:05,147] INFO: valid_dataset: clic_edm_ww_fullhad_pf, 80100
[2025-03-10 01:15:53,881] INFO: Rank 0: epoch=1/10 train_loss=2.6711 valid_loss=2.4053 stale=0 epoch_train_time=440.60m epoch_valid_time=17.18m epoch_total_time=457.78m eta=4120.2m
[2025-03-10 01:15:53,881] INFO: Rank 0: epoch=1/10 train_loss=2.6711 valid_loss=2.4053 stale=0 epoch_train_time=440.60m epoch_valid_time=17.18m epoch_total_time=457.78m eta=4120.2m
[2025-03-10 01:15:53,885] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-10 01:15:53,885] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-10 01:15:53,995] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-10 01:15:53,995] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-10 01:15:54,006] INFO: Running predictions on clic_edm_qq_pf
[2025-03-10 01:15:54,006] INFO: Running predictions on clic_edm_qq_pf
[2025-03-10 01:15:57,237] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-10 01:15:57,237] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-10 01:15:57,678] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-10 01:15:57,678] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-10 01:15:58,140] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-10 01:15:58,140] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-10 01:15:58,608] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-10 01:15:58,608] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-10 01:15:59,042] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-10 01:15:59,042] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-10 01:15:59,488] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-10 01:15:59,488] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-10 01:15:59,918] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-10 01:15:59,918] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-10 01:16:00,323] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-10 01:16:00,323] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_1/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-10 01:16:00,404] INFO: Time taken to make predictions on device 0 is: 0.10 min
[2025-03-10 01:16:00,404] INFO: Time taken to make predictions on device 0 is: 0.10 min
[2025-03-10 08:53:43,258] INFO: Rank 0: epoch=2/10 train_loss=2.2809 valid_loss=2.2379 stale=0 epoch_train_time=439.84m epoch_valid_time=17.18m epoch_total_time=457.02m eta=3662.5m
[2025-03-10 08:53:43,258] INFO: Rank 0: epoch=2/10 train_loss=2.2809 valid_loss=2.2379 stale=0 epoch_train_time=439.84m epoch_valid_time=17.18m epoch_total_time=457.02m eta=3662.5m
[2025-03-10 08:53:43,262] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-10 08:53:43,262] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-10 08:53:43,369] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-10 08:53:43,369] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-10 08:53:43,372] INFO: Running predictions on clic_edm_qq_pf
[2025-03-10 08:53:43,372] INFO: Running predictions on clic_edm_qq_pf
[2025-03-10 08:53:44,621] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-10 08:53:44,621] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-10 08:53:45,064] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-10 08:53:45,064] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-10 08:53:45,510] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-10 08:53:45,510] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-10 08:53:45,950] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-10 08:53:45,950] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-10 08:53:46,384] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-10 08:53:46,384] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-10 08:53:46,808] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-10 08:53:46,808] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-10 08:53:47,272] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-10 08:53:47,272] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-10 08:53:47,678] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-10 08:53:47,678] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_2/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-10 08:53:47,760] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-10 08:53:47,760] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-10 16:34:36,889] INFO: Rank 0: epoch=3/10 train_loss=2.1644 valid_loss=2.1493 stale=0 epoch_train_time=442.37m epoch_valid_time=17.74m epoch_total_time=460.11m eta=3211.9m
[2025-03-10 16:34:36,889] INFO: Rank 0: epoch=3/10 train_loss=2.1644 valid_loss=2.1493 stale=0 epoch_train_time=442.37m epoch_valid_time=17.74m epoch_total_time=460.11m eta=3211.9m
[2025-03-10 16:34:36,895] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-10 16:34:36,895] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-10 16:34:37,045] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-10 16:34:37,045] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-10 16:34:37,050] INFO: Running predictions on clic_edm_qq_pf
[2025-03-10 16:34:37,050] INFO: Running predictions on clic_edm_qq_pf
[2025-03-10 16:34:38,302] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-10 16:34:38,302] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-10 16:34:38,795] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-10 16:34:38,795] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-10 16:34:39,223] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-10 16:34:39,223] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-10 16:34:39,662] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-10 16:34:39,662] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-10 16:34:40,163] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-10 16:34:40,163] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-10 16:34:40,595] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-10 16:34:40,595] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-10 16:34:41,038] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-10 16:34:41,038] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-10 16:34:41,436] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-10 16:34:41,436] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_3/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-10 16:34:41,536] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-10 16:34:41,536] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-11 00:17:28,553] INFO: Rank 0: epoch=4/10 train_loss=2.0819 valid_loss=2.0866 stale=0 epoch_train_time=444.75m epoch_valid_time=17.30m epoch_total_time=462.05m eta=2759.1m
[2025-03-11 00:17:28,553] INFO: Rank 0: epoch=4/10 train_loss=2.0819 valid_loss=2.0866 stale=0 epoch_train_time=444.75m epoch_valid_time=17.30m epoch_total_time=462.05m eta=2759.1m
[2025-03-11 00:17:28,558] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-11 00:17:28,558] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-11 00:17:28,668] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-11 00:17:28,668] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-11 00:17:28,672] INFO: Running predictions on clic_edm_qq_pf
[2025-03-11 00:17:28,672] INFO: Running predictions on clic_edm_qq_pf
[2025-03-11 00:17:29,767] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-11 00:17:29,767] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-11 00:17:30,209] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-11 00:17:30,209] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-11 00:17:30,655] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-11 00:17:30,655] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-11 00:17:31,104] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-11 00:17:31,104] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-11 00:17:31,585] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-11 00:17:31,585] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-11 00:17:32,035] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-11 00:17:32,035] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-11 00:17:32,487] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-11 00:17:32,487] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-11 00:17:32,871] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-11 00:17:32,871] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_4/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-11 00:17:32,990] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-11 00:17:32,990] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-11 07:57:23,268] INFO: Rank 0: epoch=5/10 train_loss=2.0258 valid_loss=2.0461 stale=0 epoch_train_time=441.99m epoch_valid_time=17.14m epoch_total_time=459.13m eta=2299.3m
[2025-03-11 07:57:23,268] INFO: Rank 0: epoch=5/10 train_loss=2.0258 valid_loss=2.0461 stale=0 epoch_train_time=441.99m epoch_valid_time=17.14m epoch_total_time=459.13m eta=2299.3m
[2025-03-11 07:57:23,272] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-11 07:57:23,272] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-11 07:57:23,380] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-11 07:57:23,380] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-11 07:57:23,384] INFO: Running predictions on clic_edm_qq_pf
[2025-03-11 07:57:23,384] INFO: Running predictions on clic_edm_qq_pf
[2025-03-11 07:57:24,747] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-11 07:57:24,747] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-11 07:57:25,238] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-11 07:57:25,238] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-11 07:57:25,680] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-11 07:57:25,680] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-11 07:57:26,148] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-11 07:57:26,148] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-11 07:57:26,602] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-11 07:57:26,602] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-11 07:57:27,039] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-11 07:57:27,039] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-11 07:57:27,488] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-11 07:57:27,488] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-11 07:57:27,892] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-11 07:57:27,892] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_5/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-11 07:57:28,020] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-11 07:57:28,020] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-11 15:33:56,886] INFO: Rank 0: epoch=6/10 train_loss=1.9880 valid_loss=2.0138 stale=0 epoch_train_time=438.69m epoch_valid_time=17.08m epoch_total_time=455.77m eta=1837.2m
[2025-03-11 15:33:56,886] INFO: Rank 0: epoch=6/10 train_loss=1.9880 valid_loss=2.0138 stale=0 epoch_train_time=438.69m epoch_valid_time=17.08m epoch_total_time=455.77m eta=1837.2m
[2025-03-11 15:33:56,891] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-11 15:33:56,891] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-11 15:33:57,002] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-11 15:33:57,002] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-11 15:33:57,006] INFO: Running predictions on clic_edm_qq_pf
[2025-03-11 15:33:57,006] INFO: Running predictions on clic_edm_qq_pf
[2025-03-11 15:33:58,269] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-11 15:33:58,269] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-11 15:33:58,718] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-11 15:33:58,718] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-11 15:33:59,164] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-11 15:33:59,164] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-11 15:33:59,623] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-11 15:33:59,623] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-11 15:34:00,064] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-11 15:34:00,064] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-11 15:34:00,515] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-11 15:34:00,515] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-11 15:34:00,952] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-11 15:34:00,952] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-11 15:34:01,388] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-11 15:34:01,388] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_6/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-11 15:34:01,492] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-11 15:34:01,492] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-11 23:11:22,938] INFO: Rank 0: epoch=7/10 train_loss=1.9555 valid_loss=1.9800 stale=0 epoch_train_time=439.45m epoch_valid_time=17.20m epoch_total_time=456.65m eta=1377.1m
[2025-03-11 23:11:22,938] INFO: Rank 0: epoch=7/10 train_loss=1.9555 valid_loss=1.9800 stale=0 epoch_train_time=439.45m epoch_valid_time=17.20m epoch_total_time=456.65m eta=1377.1m
[2025-03-11 23:11:22,942] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-11 23:11:22,942] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-11 23:11:23,051] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-11 23:11:23,051] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-11 23:11:23,055] INFO: Running predictions on clic_edm_qq_pf
[2025-03-11 23:11:23,055] INFO: Running predictions on clic_edm_qq_pf
[2025-03-11 23:11:24,370] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-11 23:11:24,370] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-11 23:11:24,843] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-11 23:11:24,843] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-11 23:11:25,279] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-11 23:11:25,279] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-11 23:11:25,720] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-11 23:11:25,720] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-11 23:11:26,187] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-11 23:11:26,187] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-11 23:11:26,649] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-11 23:11:26,649] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-11 23:11:27,095] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-11 23:11:27,095] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-11 23:11:27,529] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-11 23:11:27,529] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_7/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-11 23:11:27,634] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-11 23:11:27,634] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-12 06:49:11,362] INFO: Rank 0: epoch=8/10 train_loss=1.9260 valid_loss=1.9599 stale=0 epoch_train_time=439.86m epoch_valid_time=17.15m epoch_total_time=457.02m eta=917.8m
[2025-03-12 06:49:11,362] INFO: Rank 0: epoch=8/10 train_loss=1.9260 valid_loss=1.9599 stale=0 epoch_train_time=439.86m epoch_valid_time=17.15m epoch_total_time=457.02m eta=917.8m
[2025-03-12 06:49:11,366] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-12 06:49:11,366] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-12 06:49:11,469] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-12 06:49:11,469] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-12 06:49:11,473] INFO: Running predictions on clic_edm_qq_pf
[2025-03-12 06:49:11,473] INFO: Running predictions on clic_edm_qq_pf
[2025-03-12 06:49:12,749] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-12 06:49:12,749] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-12 06:49:13,263] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-12 06:49:13,263] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-12 06:49:13,699] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-12 06:49:13,699] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-12 06:49:14,141] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-12 06:49:14,141] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-12 06:49:14,562] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-12 06:49:14,562] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-12 06:49:15,016] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-12 06:49:15,016] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-12 06:49:15,440] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-12 06:49:15,440] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-12 06:49:15,837] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-12 06:49:15,837] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_8/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-12 06:49:15,939] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-12 06:49:15,939] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-12 14:26:48,174] INFO: Rank 0: epoch=9/10 train_loss=1.9043 valid_loss=1.9430 stale=0 epoch_train_time=439.68m epoch_valid_time=17.15m epoch_total_time=456.83m eta=458.7m
[2025-03-12 14:26:48,174] INFO: Rank 0: epoch=9/10 train_loss=1.9043 valid_loss=1.9430 stale=0 epoch_train_time=439.68m epoch_valid_time=17.15m epoch_total_time=456.83m eta=458.7m
[2025-03-12 14:26:48,178] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-12 14:26:48,178] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-12 14:26:48,277] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-12 14:26:48,277] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-12 14:26:48,280] INFO: Running predictions on clic_edm_qq_pf
[2025-03-12 14:26:48,280] INFO: Running predictions on clic_edm_qq_pf
[2025-03-12 14:26:49,640] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-12 14:26:49,640] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-12 14:26:50,159] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-12 14:26:50,159] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-12 14:26:50,591] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-12 14:26:50,591] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-12 14:26:51,054] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-12 14:26:51,054] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-12 14:26:51,514] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-12 14:26:51,514] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-12 14:26:51,975] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-12 14:26:51,975] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-12 14:26:52,435] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-12 14:26:52,435] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-12 14:26:52,876] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-12 14:26:52,876] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_9/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-12 14:26:52,999] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-12 14:26:52,999] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-12 22:04:52,920] INFO: Rank 0: epoch=10/10 train_loss=1.8910 valid_loss=1.9393 stale=0 epoch_train_time=440.08m epoch_valid_time=17.20m epoch_total_time=457.29m eta=0.0m
[2025-03-12 22:04:52,920] INFO: Rank 0: epoch=10/10 train_loss=1.8910 valid_loss=1.9393 stale=0 epoch_train_time=440.08m epoch_valid_time=17.20m epoch_total_time=457.29m eta=0.0m
[2025-03-12 22:04:52,924] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-12 22:04:52,924] INFO: split_configs=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[2025-03-12 22:04:53,034] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-12 22:04:53,034] INFO: test_dataset: clic_edm_qq_pf, 2000
[2025-03-12 22:04:53,037] INFO: Running predictions on clic_edm_qq_pf
[2025-03-12 22:04:53,037] INFO: Running predictions on clic_edm_qq_pf
[2025-03-12 22:04:54,393] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-12 22:04:54,393] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_0.parquet
[2025-03-12 22:04:54,889] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-12 22:04:54,889] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_1.parquet
[2025-03-12 22:04:55,366] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-12 22:04:55,366] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_2.parquet
[2025-03-12 22:04:55,820] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-12 22:04:55,820] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_3.parquet
[2025-03-12 22:04:56,259] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-12 22:04:56,259] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_4.parquet
[2025-03-12 22:04:56,694] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-12 22:04:56,694] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_5.parquet
[2025-03-12 22:04:57,128] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-12 22:04:57,128] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_6.parquet
[2025-03-12 22:04:57,537] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-12 22:04:57,537] INFO: Saved predictions at experiments/pyg-clic_20250309_173756_957486/preds_epoch_10/clic_edm_qq_pf/pred_0_7.parquet
[2025-03-12 22:04:57,681] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-12 22:04:57,681] INFO: Time taken to make predictions on device 0 is: 0.07 min
[2025-03-12 22:05:38,872] INFO: Training completed. Total time on device 0: 4587.548min
[2025-03-12 22:05:38,872] INFO: Training completed. Total time on device 0: 4587.548min