codet5_base_train_e5

README.md

@@ -1,7 +1,7 @@
 ---
 library_name: transformers
 license: apache-2.0
-base_model: Salesforce/codet5-small
+base_model: Salesforce/codet5-base
 tags:
 - generated_from_trainer
 datasets:
@@ -23,7 +23,7 @@ model-index:
     metrics:
     - name: Bleu
       type: bleu
-      value: 0.009912737763560728
+      value: 0.013233021060148939
 ---
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
@@ -31,10 +31,10 @@ should probably proofread and complete it, then remove this comment. -->
 
 # code_docstring_model
 
-This model is a fine-tuned version of [Salesforce/codet5-small](https://huggingface.co/Salesforce/codet5-small) on the code_search_net dataset.
+This model is a fine-tuned version of [Salesforce/codet5-base](https://huggingface.co/Salesforce/codet5-base) on the code_search_net dataset.
 It achieves the following results on the evaluation set:
-- Loss: 1.1194
-- Bleu: 0.0099
+- Loss: 0.9120
+- Bleu: 0.0132
 
 ## Model description
 
@@ -53,26 +53,27 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 1e-05
-- train_batch_size: 8
-- eval_batch_size: 8
+- learning_rate: 5e-05
+- train_batch_size: 16
+- eval_batch_size: 16
 - seed: 42
 - gradient_accumulation_steps: 4
-- total_train_batch_size: 32
+- total_train_batch_size: 64
 - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.1
 - num_epochs: 5
 - mixed_precision_training: Native AMP
 
 ### Training results
 
-| Training Loss | Epoch  | Step  | Validation Loss | Bleu   |
-|:-------------:|:------:|:-----:|:---------------:|:------:|
-| 1.4113        | 1.0    | 2004  | 1.2082          | 0.0095 |
-| 1.3283        | 2.0    | 4008  | 1.1537          | 0.0097 |
-| 1.3036        | 3.0    | 6012  | 1.1331          | 0.0098 |
-| 1.2585        | 4.0    | 8016  | 1.1226          | 0.0098 |
-| 1.2613        | 4.9978 | 10015 | 1.1194          | 0.0099 |
+| Training Loss | Epoch  | Step | Validation Loss | Bleu   |
+|:-------------:|:------:|:----:|:---------------:|:------:|
+| 1.0906        | 1.0    | 1002 | 0.9715          | 0.0107 |
+| 0.9922        | 2.0    | 2004 | 0.9390          | 0.0108 |
+| 0.9325        | 3.0    | 3006 | 0.9233          | 0.0113 |
+| 0.8936        | 4.0    | 4008 | 0.9134          | 0.0124 |
+| 0.8769        | 4.9953 | 5005 | 0.9120          | 0.0132 |
 
 
 ### Framework versions

config.json

@@ -1,13 +1,13 @@
 {
-  "_name_or_path": "Salesforce/codet5-small",
+  "_name_or_path": "Salesforce/codet5-base",
   "architectures": [
     "T5ForConditionalGeneration"
   ],
   "bos_token_id": 1,
   "classifier_dropout": 0.0,
-  "d_ff": 2048,
+  "d_ff": 3072,
   "d_kv": 64,
-  "d_model": 512,
+  "d_model": 768,
   "decoder_start_token_id": 0,
   "dense_act_fn": "relu",
   "dropout_rate": 0.1,
@@ -26,9 +26,9 @@
   "layer_norm_epsilon": 1e-06,
   "model_type": "t5",
   "n_positions": 512,
-  "num_decoder_layers": 6,
-  "num_heads": 8,
-  "num_layers": 6,
+  "num_decoder_layers": 12,
+  "num_heads": 12,
+  "num_layers": 12,
   "output_past": true,
   "pad_token_id": 0,
   "relative_attention_max_distance": 128,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:96e6e7f07f24e59be74c6d559a79325413ed1d8aed5a419fa41906e70a5bf66d
-size 242037800
+oid sha256:d74d7f8df141d0f53996700750693771b6f8aedeed3981d0a9e4c922ce32460f
+size 891638568

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3b37b21c03d518303a92294b07f5807779171f7cb7ed6a230c85043641d48951
+oid sha256:353600229a52d1bc40f4e2190c68b4e1e545a3595edbe90c69597e2cc3abf322
 size 5496