diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -1,3 +1,10867 @@ --- +model-index: +- name: no_model_name_available + results: + - dataset: + config: default + name: MTEB ArguAna (default) + revision: c22ab2a51041ffd869aaddef7af8d8215647e41a + split: test + type: mteb/arguana + metrics: + - type: main_score + value: 62.666 + - type: map_at_1 + value: 38.193 + - type: map_at_10 + value: 54.196 + - type: map_at_100 + value: 54.790000000000006 + - type: map_at_1000 + value: 54.791000000000004 + - type: map_at_20 + value: 54.691 + - type: map_at_3 + value: 49.644 + - type: map_at_5 + value: 52.47500000000001 + - type: mrr_at_1 + value: 38.762446657183496 + - type: mrr_at_10 + value: 54.39228702386607 + - type: mrr_at_100 + value: 54.98571942987527 + - type: mrr_at_1000 + value: 54.98684265760785 + - type: mrr_at_20 + value: 54.88631259457816 + - type: mrr_at_3 + value: 49.82219061166436 + - type: mrr_at_5 + value: 52.68492176386928 + - type: nauc_map_at_1000_diff1 + value: 12.947602614819145 + - type: nauc_map_at_1000_max + value: -9.987850231314727 + - type: nauc_map_at_1000_std + value: -17.515475948907223 + - type: nauc_map_at_100_diff1 + value: 12.948715892672372 + - type: nauc_map_at_100_max + value: -9.984257476131498 + - type: nauc_map_at_100_std + value: -17.512184326552735 + - type: nauc_map_at_10_diff1 + value: 12.67631091463424 + - type: nauc_map_at_10_max + value: -9.893868710127792 + - type: nauc_map_at_10_std + value: -17.394961603516936 + - type: nauc_map_at_1_diff1 + value: 15.841810182641536 + - type: nauc_map_at_1_max + value: -12.300180964189106 + - type: nauc_map_at_1_std + value: -19.269251850255216 + - type: nauc_map_at_20_diff1 + value: 13.00808878471498 + - type: nauc_map_at_20_max + value: -9.850137316681932 + - type: nauc_map_at_20_std + value: -17.447259697851912 + - type: nauc_map_at_3_diff1 + value: 12.69239436295672 + - type: nauc_map_at_3_max + value: -10.368307463632874 + - type: nauc_map_at_3_std + value: -18.199985684348547 + - type: nauc_map_at_5_diff1 + value: 12.422311022795533 + - type: nauc_map_at_5_max + value: -9.914304305052736 + - type: nauc_map_at_5_std + value: -17.66096347572028 + - type: nauc_mrr_at_1000_diff1 + value: 11.180245273621827 + - type: nauc_mrr_at_1000_max + value: -10.894158507712113 + - type: nauc_mrr_at_1000_std + value: -17.49976153775426 + - type: nauc_mrr_at_100_diff1 + value: 11.18142277871486 + - type: nauc_mrr_at_100_max + value: -10.890541252974147 + - type: nauc_mrr_at_100_std + value: -17.496479703672595 + - type: nauc_mrr_at_10_diff1 + value: 10.921714446140344 + - type: nauc_mrr_at_10_max + value: -10.792048413190232 + - type: nauc_mrr_at_10_std + value: -17.33190957361021 + - type: nauc_mrr_at_1_diff1 + value: 14.221022720755656 + - type: nauc_mrr_at_1_max + value: -12.52748081868306 + - type: nauc_mrr_at_1_std + value: -18.886835343065435 + - type: nauc_mrr_at_20_diff1 + value: 11.246975158571583 + - type: nauc_mrr_at_20_max + value: -10.753425023987633 + - type: nauc_mrr_at_20_std + value: -17.431704030640145 + - type: nauc_mrr_at_3_diff1 + value: 10.960249308252628 + - type: nauc_mrr_at_3_max + value: -11.264464166330386 + - type: nauc_mrr_at_3_std + value: -18.11489563671473 + - type: nauc_mrr_at_5_diff1 + value: 10.756134394305295 + - type: nauc_mrr_at_5_max + value: -10.761804905982734 + - type: nauc_mrr_at_5_std + value: -17.667529106760977 + - type: nauc_ndcg_at_1000_diff1 + value: 12.6390391884592 + - type: nauc_ndcg_at_1000_max + value: -9.41361560340564 + - type: nauc_ndcg_at_1000_std + value: -16.87810909581786 + - type: nauc_ndcg_at_100_diff1 + value: 12.67778264270421 + - type: nauc_ndcg_at_100_max + value: -9.298676143355197 + - type: nauc_ndcg_at_100_std + value: -16.772693001294982 + - type: nauc_ndcg_at_10_diff1 + value: 11.633673431342958 + - type: nauc_ndcg_at_10_max + value: -8.5910148745569 + - type: nauc_ndcg_at_10_std + value: -16.140320975304455 + - type: nauc_ndcg_at_1_diff1 + value: 15.841810182641536 + - type: nauc_ndcg_at_1_max + value: -12.300180964189106 + - type: nauc_ndcg_at_1_std + value: -19.269251850255216 + - type: nauc_ndcg_at_20_diff1 + value: 13.058036889736742 + - type: nauc_ndcg_at_20_max + value: -8.336575527683479 + - type: nauc_ndcg_at_20_std + value: -16.315062866812053 + - type: nauc_ndcg_at_3_diff1 + value: 11.582148410127758 + - type: nauc_ndcg_at_3_max + value: -9.82963620948141 + - type: nauc_ndcg_at_3_std + value: -17.99199628781833 + - type: nauc_ndcg_at_5_diff1 + value: 11.058463561410752 + - type: nauc_ndcg_at_5_max + value: -8.7961062162745 + - type: nauc_ndcg_at_5_std + value: -16.905819717115595 + - type: nauc_precision_at_1000_diff1 + value: 9.55460212276602 + - type: nauc_precision_at_1000_max + value: 24.775170554257024 + - type: nauc_precision_at_1000_std + value: 70.07345061131439 + - type: nauc_precision_at_100_diff1 + value: 22.84894979867915 + - type: nauc_precision_at_100_max + value: 47.4108021596634 + - type: nauc_precision_at_100_std + value: 70.00340119762156 + - type: nauc_precision_at_10_diff1 + value: 3.791683701992363 + - type: nauc_precision_at_10_max + value: 2.9195191931534183 + - type: nauc_precision_at_10_std + value: -4.791878871375184 + - type: nauc_precision_at_1_diff1 + value: 15.841810182641536 + - type: nauc_precision_at_1_max + value: -12.300180964189106 + - type: nauc_precision_at_1_std + value: -19.269251850255216 + - type: nauc_precision_at_20_diff1 + value: 25.111346924709686 + - type: nauc_precision_at_20_max + value: 28.580648802218356 + - type: nauc_precision_at_20_std + value: 10.1739189341263 + - type: nauc_precision_at_3_diff1 + value: 7.907993267968324 + - type: nauc_precision_at_3_max + value: -8.096948169326348 + - type: nauc_precision_at_3_std + value: -17.362968493669033 + - type: nauc_precision_at_5_diff1 + value: 5.181990539586378 + - type: nauc_precision_at_5_max + value: -3.72943379775899 + - type: nauc_precision_at_5_std + value: -13.48054977591215 + - type: nauc_recall_at_1000_diff1 + value: 9.55460212276554 + - type: nauc_recall_at_1000_max + value: 24.77517055425506 + - type: nauc_recall_at_1000_std + value: 70.07345061131205 + - type: nauc_recall_at_100_diff1 + value: 22.84894979867997 + - type: nauc_recall_at_100_max + value: 47.41080215966243 + - type: nauc_recall_at_100_std + value: 70.0034011976204 + - type: nauc_recall_at_10_diff1 + value: 3.791683701992332 + - type: nauc_recall_at_10_max + value: 2.919519193153294 + - type: nauc_recall_at_10_std + value: -4.791878871375061 + - type: nauc_recall_at_1_diff1 + value: 15.841810182641536 + - type: nauc_recall_at_1_max + value: -12.300180964189106 + - type: nauc_recall_at_1_std + value: -19.269251850255216 + - type: nauc_recall_at_20_diff1 + value: 25.111346924709625 + - type: nauc_recall_at_20_max + value: 28.580648802218118 + - type: nauc_recall_at_20_std + value: 10.173918934126617 + - type: nauc_recall_at_3_diff1 + value: 7.907993267968336 + - type: nauc_recall_at_3_max + value: -8.096948169326346 + - type: nauc_recall_at_3_std + value: -17.362968493669033 + - type: nauc_recall_at_5_diff1 + value: 5.181990539586512 + - type: nauc_recall_at_5_max + value: -3.7294337977588468 + - type: nauc_recall_at_5_std + value: -13.480549775911967 + - type: ndcg_at_1 + value: 38.193 + - type: ndcg_at_10 + value: 62.666 + - type: ndcg_at_100 + value: 65.031 + - type: ndcg_at_1000 + value: 65.059 + - type: ndcg_at_20 + value: 64.39399999999999 + - type: ndcg_at_3 + value: 53.456 + - type: ndcg_at_5 + value: 58.540000000000006 + - type: precision_at_1 + value: 38.193 + - type: precision_at_10 + value: 8.947 + - type: precision_at_100 + value: 0.9939999999999999 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.808 + - type: precision_at_3 + value: 21.503 + - type: precision_at_5 + value: 15.363 + - type: recall_at_1 + value: 38.193 + - type: recall_at_10 + value: 89.474 + - type: recall_at_100 + value: 99.431 + - type: recall_at_1000 + value: 99.644 + - type: recall_at_20 + value: 96.15899999999999 + - type: recall_at_3 + value: 64.509 + - type: recall_at_5 + value: 76.81400000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackAndroidRetrieval (default) + revision: f46a197baaae43b4f621051089b82a364682dfeb + split: test + type: mteb/cqadupstack-android + metrics: + - type: main_score + value: 54.846 + - type: map_at_1 + value: 36.111 + - type: map_at_10 + value: 48.448 + - type: map_at_100 + value: 49.94 + - type: map_at_1000 + value: 50.05 + - type: map_at_20 + value: 49.241 + - type: map_at_3 + value: 44.629000000000005 + - type: map_at_5 + value: 46.934 + - type: mrr_at_1 + value: 43.776824034334766 + - type: mrr_at_10 + value: 53.94719213388739 + - type: mrr_at_100 + value: 54.531422307626244 + - type: mrr_at_1000 + value: 54.56839458978485 + - type: mrr_at_20 + value: 54.25921925852162 + - type: mrr_at_3 + value: 51.19217930376727 + - type: mrr_at_5 + value: 53.00190748688598 + - type: nauc_map_at_1000_diff1 + value: 53.17441767032215 + - type: nauc_map_at_1000_max + value: 38.95908882595929 + - type: nauc_map_at_1000_std + value: -2.1420694508117175 + - type: nauc_map_at_100_diff1 + value: 53.16194042285659 + - type: nauc_map_at_100_max + value: 38.966089740038754 + - type: nauc_map_at_100_std + value: -2.1214852009755183 + - type: nauc_map_at_10_diff1 + value: 53.20405470513967 + - type: nauc_map_at_10_max + value: 38.515786134267124 + - type: nauc_map_at_10_std + value: -3.0520185798271813 + - type: nauc_map_at_1_diff1 + value: 59.126435885543636 + - type: nauc_map_at_1_max + value: 34.385415653648046 + - type: nauc_map_at_1_std + value: -4.766310174116546 + - type: nauc_map_at_20_diff1 + value: 53.231550803006925 + - type: nauc_map_at_20_max + value: 38.70471444895874 + - type: nauc_map_at_20_std + value: -2.50164393146215 + - type: nauc_map_at_3_diff1 + value: 54.53310358576739 + - type: nauc_map_at_3_max + value: 38.08930056367847 + - type: nauc_map_at_3_std + value: -3.8283271850099254 + - type: nauc_map_at_5_diff1 + value: 53.35970802180032 + - type: nauc_map_at_5_max + value: 38.408935826687454 + - type: nauc_map_at_5_std + value: -3.1732562397893114 + - type: nauc_mrr_at_1000_diff1 + value: 52.831405883799576 + - type: nauc_mrr_at_1000_max + value: 41.050112222663046 + - type: nauc_mrr_at_1000_std + value: -2.1687323979650333 + - type: nauc_mrr_at_100_diff1 + value: 52.81238331526759 + - type: nauc_mrr_at_100_max + value: 41.05011629980655 + - type: nauc_mrr_at_100_std + value: -2.1658586354799327 + - type: nauc_mrr_at_10_diff1 + value: 52.828498477917464 + - type: nauc_mrr_at_10_max + value: 41.08270816155139 + - type: nauc_mrr_at_10_std + value: -2.3047516303383233 + - type: nauc_mrr_at_1_diff1 + value: 57.22994388721246 + - type: nauc_mrr_at_1_max + value: 39.77208041523064 + - type: nauc_mrr_at_1_std + value: -4.892401212851535 + - type: nauc_mrr_at_20_diff1 + value: 52.820389171191486 + - type: nauc_mrr_at_20_max + value: 41.01552772770224 + - type: nauc_mrr_at_20_std + value: -2.1647853891627844 + - type: nauc_mrr_at_3_diff1 + value: 52.803347328106135 + - type: nauc_mrr_at_3_max + value: 40.644737785024674 + - type: nauc_mrr_at_3_std + value: -3.158251307960102 + - type: nauc_mrr_at_5_diff1 + value: 52.590402661333535 + - type: nauc_mrr_at_5_max + value: 40.85807379760336 + - type: nauc_mrr_at_5_std + value: -2.4051222145052966 + - type: nauc_ndcg_at_1000_diff1 + value: 51.66344148361651 + - type: nauc_ndcg_at_1000_max + value: 40.29657461365782 + - type: nauc_ndcg_at_1000_std + value: -0.19915825919029412 + - type: nauc_ndcg_at_100_diff1 + value: 50.8854928029126 + - type: nauc_ndcg_at_100_max + value: 40.37615388766142 + - type: nauc_ndcg_at_100_std + value: 0.32767687009655566 + - type: nauc_ndcg_at_10_diff1 + value: 50.62637141454391 + - type: nauc_ndcg_at_10_max + value: 39.24511745556687 + - type: nauc_ndcg_at_10_std + value: -2.083556882081868 + - type: nauc_ndcg_at_1_diff1 + value: 57.22994388721246 + - type: nauc_ndcg_at_1_max + value: 39.77208041523064 + - type: nauc_ndcg_at_1_std + value: -4.892401212851535 + - type: nauc_ndcg_at_20_diff1 + value: 50.81415751469669 + - type: nauc_ndcg_at_20_max + value: 39.24364922375507 + - type: nauc_ndcg_at_20_std + value: -1.214457557106301 + - type: nauc_ndcg_at_3_diff1 + value: 52.02290866353769 + - type: nauc_ndcg_at_3_max + value: 39.35380416960457 + - type: nauc_ndcg_at_3_std + value: -2.7477400632000153 + - type: nauc_ndcg_at_5_diff1 + value: 50.63588698360948 + - type: nauc_ndcg_at_5_max + value: 39.145893391510725 + - type: nauc_ndcg_at_5_std + value: -2.0181440458445152 + - type: nauc_precision_at_1000_diff1 + value: -24.251750773406442 + - type: nauc_precision_at_1000_max + value: -10.590893152082419 + - type: nauc_precision_at_1000_std + value: -3.9877455972266294 + - type: nauc_precision_at_100_diff1 + value: -16.84401603591616 + - type: nauc_precision_at_100_max + value: 0.743127411576026 + - type: nauc_precision_at_100_std + value: 6.260820358988766 + - type: nauc_precision_at_10_diff1 + value: 5.428947053187325 + - type: nauc_precision_at_10_max + value: 18.827264174531997 + - type: nauc_precision_at_10_std + value: 3.555865794585147 + - type: nauc_precision_at_1_diff1 + value: 57.22994388721246 + - type: nauc_precision_at_1_max + value: 39.77208041523064 + - type: nauc_precision_at_1_std + value: -4.892401212851535 + - type: nauc_precision_at_20_diff1 + value: -2.794125936925247 + - type: nauc_precision_at_20_max + value: 11.198388078993144 + - type: nauc_precision_at_20_std + value: 5.9460738597807525 + - type: nauc_precision_at_3_diff1 + value: 26.95987488764992 + - type: nauc_precision_at_3_max + value: 32.91043306531649 + - type: nauc_precision_at_3_std + value: 0.15771291919908714 + - type: nauc_precision_at_5_diff1 + value: 15.193687395003655 + - type: nauc_precision_at_5_max + value: 26.640232573772494 + - type: nauc_precision_at_5_std + value: 2.3332720112865526 + - type: nauc_recall_at_1000_diff1 + value: 33.86420489805354 + - type: nauc_recall_at_1000_max + value: 52.80386737090323 + - type: nauc_recall_at_1000_std + value: 57.14825831673145 + - type: nauc_recall_at_100_diff1 + value: 32.577933083129366 + - type: nauc_recall_at_100_max + value: 43.09256175270545 + - type: nauc_recall_at_100_std + value: 20.428339510048225 + - type: nauc_recall_at_10_diff1 + value: 40.87858857153924 + - type: nauc_recall_at_10_max + value: 35.86123623977615 + - type: nauc_recall_at_10_std + value: -0.4779603506635357 + - type: nauc_recall_at_1_diff1 + value: 59.126435885543636 + - type: nauc_recall_at_1_max + value: 34.385415653648046 + - type: nauc_recall_at_1_std + value: -4.766310174116546 + - type: nauc_recall_at_20_diff1 + value: 39.26228101729742 + - type: nauc_recall_at_20_max + value: 34.824136650321634 + - type: nauc_recall_at_20_std + value: 2.8795388295600026 + - type: nauc_recall_at_3_diff1 + value: 47.860494124794144 + - type: nauc_recall_at_3_max + value: 36.93328202943318 + - type: nauc_recall_at_3_std + value: -3.5506384393326464 + - type: nauc_recall_at_5_diff1 + value: 42.530969807369665 + - type: nauc_recall_at_5_max + value: 36.39847750684429 + - type: nauc_recall_at_5_std + value: -0.5382019811651719 + - type: ndcg_at_1 + value: 43.777 + - type: ndcg_at_10 + value: 54.846 + - type: ndcg_at_100 + value: 59.760000000000005 + - type: ndcg_at_1000 + value: 61.345000000000006 + - type: ndcg_at_20 + value: 56.647000000000006 + - type: ndcg_at_3 + value: 49.667 + - type: ndcg_at_5 + value: 52.43 + - type: precision_at_1 + value: 43.777 + - type: precision_at_10 + value: 10.415000000000001 + - type: precision_at_100 + value: 1.624 + - type: precision_at_1000 + value: 0.20400000000000001 + - type: precision_at_20 + value: 6.094 + - type: precision_at_3 + value: 23.796 + - type: precision_at_5 + value: 17.253 + - type: recall_at_1 + value: 36.111 + - type: recall_at_10 + value: 66.90700000000001 + - type: recall_at_100 + value: 87.246 + - type: recall_at_1000 + value: 97.08099999999999 + - type: recall_at_20 + value: 73.063 + - type: recall_at_3 + value: 52.010999999999996 + - type: recall_at_5 + value: 59.75 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackEnglishRetrieval (default) + revision: ad9991cb51e31e31e430383c75ffb2885547b5f0 + split: test + type: mteb/cqadupstack-english + metrics: + - type: main_score + value: 53.196 + - type: map_at_1 + value: 36.26 + - type: map_at_10 + value: 47.335 + - type: map_at_100 + value: 48.730000000000004 + - type: map_at_1000 + value: 48.861 + - type: map_at_20 + value: 48.129 + - type: map_at_3 + value: 44.238 + - type: map_at_5 + value: 46.034000000000006 + - type: mrr_at_1 + value: 44.968152866242036 + - type: mrr_at_10 + value: 53.73892932969372 + - type: mrr_at_100 + value: 54.333468364982906 + - type: mrr_at_1000 + value: 54.370836026869505 + - type: mrr_at_20 + value: 54.123794088345726 + - type: mrr_at_3 + value: 51.73036093418263 + - type: mrr_at_5 + value: 52.99150743099798 + - type: nauc_map_at_1000_diff1 + value: 57.295287468233326 + - type: nauc_map_at_1000_max + value: 42.6149573215277 + - type: nauc_map_at_1000_std + value: -0.7686489421965506 + - type: nauc_map_at_100_diff1 + value: 57.29507462500336 + - type: nauc_map_at_100_max + value: 42.52833408586094 + - type: nauc_map_at_100_std + value: -0.881263562796828 + - type: nauc_map_at_10_diff1 + value: 57.61672768052849 + - type: nauc_map_at_10_max + value: 41.31671828332081 + - type: nauc_map_at_10_std + value: -2.8891635216535634 + - type: nauc_map_at_1_diff1 + value: 63.46209537521443 + - type: nauc_map_at_1_max + value: 35.119400032152384 + - type: nauc_map_at_1_std + value: -8.992917127322993 + - type: nauc_map_at_20_diff1 + value: 57.49917967887862 + - type: nauc_map_at_20_max + value: 42.02781525990085 + - type: nauc_map_at_20_std + value: -1.7651407976632814 + - type: nauc_map_at_3_diff1 + value: 58.92506593587894 + - type: nauc_map_at_3_max + value: 39.01971384516871 + - type: nauc_map_at_3_std + value: -5.698455736708154 + - type: nauc_map_at_5_diff1 + value: 58.15791264144069 + - type: nauc_map_at_5_max + value: 40.31321601551874 + - type: nauc_map_at_5_std + value: -4.405539864562691 + - type: nauc_mrr_at_1000_diff1 + value: 57.079620549112285 + - type: nauc_mrr_at_1000_max + value: 46.317163525725405 + - type: nauc_mrr_at_1000_std + value: 4.096363731383523 + - type: nauc_mrr_at_100_diff1 + value: 57.058762579897646 + - type: nauc_mrr_at_100_max + value: 46.317519602828064 + - type: nauc_mrr_at_100_std + value: 4.111843329849721 + - type: nauc_mrr_at_10_diff1 + value: 57.01353195027898 + - type: nauc_mrr_at_10_max + value: 46.247087927027096 + - type: nauc_mrr_at_10_std + value: 3.855178674237452 + - type: nauc_mrr_at_1_diff1 + value: 61.506327445953616 + - type: nauc_mrr_at_1_max + value: 45.19424350033609 + - type: nauc_mrr_at_1_std + value: 0.7987910550079926 + - type: nauc_mrr_at_20_diff1 + value: 57.07223516281667 + - type: nauc_mrr_at_20_max + value: 46.342882408112764 + - type: nauc_mrr_at_20_std + value: 4.045367784789082 + - type: nauc_mrr_at_3_diff1 + value: 57.225233158442215 + - type: nauc_mrr_at_3_max + value: 45.57614183549352 + - type: nauc_mrr_at_3_std + value: 3.189358028856844 + - type: nauc_mrr_at_5_diff1 + value: 57.234506328279586 + - type: nauc_mrr_at_5_max + value: 46.30351706317368 + - type: nauc_mrr_at_5_std + value: 3.46299613868041 + - type: nauc_ndcg_at_1000_diff1 + value: 55.13374824013285 + - type: nauc_ndcg_at_1000_max + value: 45.39974682871558 + - type: nauc_ndcg_at_1000_std + value: 5.039985316744238 + - type: nauc_ndcg_at_100_diff1 + value: 54.85507279232029 + - type: nauc_ndcg_at_100_max + value: 45.038625735601116 + - type: nauc_ndcg_at_100_std + value: 4.818120273357255 + - type: nauc_ndcg_at_10_diff1 + value: 55.17633770981898 + - type: nauc_ndcg_at_10_max + value: 44.0302162545744 + - type: nauc_ndcg_at_10_std + value: 1.5547730106285536 + - type: nauc_ndcg_at_1_diff1 + value: 61.506327445953616 + - type: nauc_ndcg_at_1_max + value: 45.19424350033609 + - type: nauc_ndcg_at_1_std + value: 0.7987910550079926 + - type: nauc_ndcg_at_20_diff1 + value: 55.42817718592058 + - type: nauc_ndcg_at_20_max + value: 44.71135035931726 + - type: nauc_ndcg_at_20_std + value: 3.101599419662149 + - type: nauc_ndcg_at_3_diff1 + value: 55.79812891233724 + - type: nauc_ndcg_at_3_max + value: 42.97992291062368 + - type: nauc_ndcg_at_3_std + value: 0.1801063909376762 + - type: nauc_ndcg_at_5_diff1 + value: 55.560055634664685 + - type: nauc_ndcg_at_5_max + value: 43.646815509387274 + - type: nauc_ndcg_at_5_std + value: 0.2022491010696674 + - type: nauc_precision_at_1000_diff1 + value: -20.9404575085418 + - type: nauc_precision_at_1000_max + value: 16.391802822680553 + - type: nauc_precision_at_1000_std + value: 34.41741021409238 + - type: nauc_precision_at_100_diff1 + value: -14.838657165724722 + - type: nauc_precision_at_100_max + value: 25.698451793254833 + - type: nauc_precision_at_100_std + value: 38.71230837808965 + - type: nauc_precision_at_10_diff1 + value: 4.375206706211809 + - type: nauc_precision_at_10_max + value: 38.053470570590356 + - type: nauc_precision_at_10_std + value: 29.0386868568606 + - type: nauc_precision_at_1_diff1 + value: 61.506327445953616 + - type: nauc_precision_at_1_max + value: 45.19424350033609 + - type: nauc_precision_at_1_std + value: 0.7987910550079926 + - type: nauc_precision_at_20_diff1 + value: -4.102313008121591 + - type: nauc_precision_at_20_max + value: 34.049305847004455 + - type: nauc_precision_at_20_std + value: 35.041624123826296 + - type: nauc_precision_at_3_diff1 + value: 26.057824880524944 + - type: nauc_precision_at_3_max + value: 40.784901166011835 + - type: nauc_precision_at_3_std + value: 14.692002630812059 + - type: nauc_precision_at_5_diff1 + value: 15.813117607484017 + - type: nauc_precision_at_5_max + value: 40.09400286802362 + - type: nauc_precision_at_5_std + value: 20.615925554830874 + - type: nauc_recall_at_1000_diff1 + value: 39.47900750044286 + - type: nauc_recall_at_1000_max + value: 53.20678234501871 + - type: nauc_recall_at_1000_std + value: 43.78613858304841 + - type: nauc_recall_at_100_diff1 + value: 39.764211604314866 + - type: nauc_recall_at_100_max + value: 45.0047263455656 + - type: nauc_recall_at_100_std + value: 24.46451792418812 + - type: nauc_recall_at_10_diff1 + value: 47.511887567768454 + - type: nauc_recall_at_10_max + value: 40.477978340514724 + - type: nauc_recall_at_10_std + value: 3.466419712605683 + - type: nauc_recall_at_1_diff1 + value: 63.46209537521443 + - type: nauc_recall_at_1_max + value: 35.119400032152384 + - type: nauc_recall_at_1_std + value: -8.992917127322993 + - type: nauc_recall_at_20_diff1 + value: 46.47798165321177 + - type: nauc_recall_at_20_max + value: 43.19071448032017 + - type: nauc_recall_at_20_std + value: 10.516647508669879 + - type: nauc_recall_at_3_diff1 + value: 53.12605810797017 + - type: nauc_recall_at_3_max + value: 37.08090251596953 + - type: nauc_recall_at_3_std + value: -4.2126538180213595 + - type: nauc_recall_at_5_diff1 + value: 50.38486742193533 + - type: nauc_recall_at_5_max + value: 39.281215574838164 + - type: nauc_recall_at_5_std + value: -1.8374428832701868 + - type: ndcg_at_1 + value: 44.968 + - type: ndcg_at_10 + value: 53.196 + - type: ndcg_at_100 + value: 57.513000000000005 + - type: ndcg_at_1000 + value: 59.288 + - type: ndcg_at_20 + value: 55.010999999999996 + - type: ndcg_at_3 + value: 49.314 + - type: ndcg_at_5 + value: 51.256 + - type: precision_at_1 + value: 44.968 + - type: precision_at_10 + value: 9.911 + - type: precision_at_100 + value: 1.564 + - type: precision_at_1000 + value: 0.202 + - type: precision_at_20 + value: 5.847 + - type: precision_at_3 + value: 23.949 + - type: precision_at_5 + value: 16.713 + - type: recall_at_1 + value: 36.26 + - type: recall_at_10 + value: 62.681 + - type: recall_at_100 + value: 80.701 + - type: recall_at_1000 + value: 91.58500000000001 + - type: recall_at_20 + value: 69.292 + - type: recall_at_3 + value: 50.646 + - type: recall_at_5 + value: 56.431 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackGamingRetrieval (default) + revision: 4885aa143210c98657558c04aaf3dc47cfb54340 + split: test + type: mteb/cqadupstack-gaming + metrics: + - type: main_score + value: 63.605000000000004 + - type: map_at_1 + value: 44.778 + - type: map_at_10 + value: 57.824 + - type: map_at_100 + value: 58.760999999999996 + - type: map_at_1000 + value: 58.806000000000004 + - type: map_at_20 + value: 58.394 + - type: map_at_3 + value: 54.547000000000004 + - type: map_at_5 + value: 56.457 + - type: mrr_at_1 + value: 51.285266457680244 + - type: mrr_at_10 + value: 61.24976364631546 + - type: mrr_at_100 + value: 61.81759160268465 + - type: mrr_at_1000 + value: 61.83867821618564 + - type: mrr_at_20 + value: 61.62420765129267 + - type: mrr_at_3 + value: 59.22675026123311 + - type: mrr_at_5 + value: 60.32706374085697 + - type: nauc_map_at_1000_diff1 + value: 60.368503415762234 + - type: nauc_map_at_1000_max + value: 44.96514108052982 + - type: nauc_map_at_1000_std + value: 3.4866290173840144 + - type: nauc_map_at_100_diff1 + value: 60.34998450918084 + - type: nauc_map_at_100_max + value: 44.94833189060464 + - type: nauc_map_at_100_std + value: 3.4939332021956413 + - type: nauc_map_at_10_diff1 + value: 60.32860594945097 + - type: nauc_map_at_10_max + value: 44.34580225711415 + - type: nauc_map_at_10_std + value: 2.609385098766796 + - type: nauc_map_at_1_diff1 + value: 63.29885856726938 + - type: nauc_map_at_1_max + value: 38.19351589152847 + - type: nauc_map_at_1_std + value: -0.23987289074118623 + - type: nauc_map_at_20_diff1 + value: 60.34976037843081 + - type: nauc_map_at_20_max + value: 44.72198491325629 + - type: nauc_map_at_20_std + value: 3.188622507579321 + - type: nauc_map_at_3_diff1 + value: 61.05821691143568 + - type: nauc_map_at_3_max + value: 42.114968451893226 + - type: nauc_map_at_3_std + value: 0.13242006589544147 + - type: nauc_map_at_5_diff1 + value: 60.6767648942907 + - type: nauc_map_at_5_max + value: 43.370520039538015 + - type: nauc_map_at_5_std + value: 1.4732371506495183 + - type: nauc_mrr_at_1000_diff1 + value: 60.33670030244939 + - type: nauc_mrr_at_1000_max + value: 46.26644440327484 + - type: nauc_mrr_at_1000_std + value: 4.0753678374963 + - type: nauc_mrr_at_100_diff1 + value: 60.33040763350716 + - type: nauc_mrr_at_100_max + value: 46.28095206465612 + - type: nauc_mrr_at_100_std + value: 4.097304112073528 + - type: nauc_mrr_at_10_diff1 + value: 60.221974328711156 + - type: nauc_mrr_at_10_max + value: 46.24510499418471 + - type: nauc_mrr_at_10_std + value: 3.822618648984892 + - type: nauc_mrr_at_1_diff1 + value: 63.39729495739813 + - type: nauc_mrr_at_1_max + value: 44.758696800866346 + - type: nauc_mrr_at_1_std + value: 2.574147496045342 + - type: nauc_mrr_at_20_diff1 + value: 60.285510023221214 + - type: nauc_mrr_at_20_max + value: 46.294365162588214 + - type: nauc_mrr_at_20_std + value: 4.111435754789516 + - type: nauc_mrr_at_3_diff1 + value: 60.38919050437198 + - type: nauc_mrr_at_3_max + value: 45.79114913690645 + - type: nauc_mrr_at_3_std + value: 3.197372036067399 + - type: nauc_mrr_at_5_diff1 + value: 60.456285995649104 + - type: nauc_mrr_at_5_max + value: 46.0777953302136 + - type: nauc_mrr_at_5_std + value: 3.609115141016006 + - type: nauc_ndcg_at_1000_diff1 + value: 59.70545642624211 + - type: nauc_ndcg_at_1000_max + value: 47.1115207880212 + - type: nauc_ndcg_at_1000_std + value: 6.189995928512801 + - type: nauc_ndcg_at_100_diff1 + value: 59.43304412306364 + - type: nauc_ndcg_at_100_max + value: 47.395181885438205 + - type: nauc_ndcg_at_100_std + value: 6.866092962605067 + - type: nauc_ndcg_at_10_diff1 + value: 58.85235502885112 + - type: nauc_ndcg_at_10_max + value: 46.48904034902361 + - type: nauc_ndcg_at_10_std + value: 4.961601579653053 + - type: nauc_ndcg_at_1_diff1 + value: 63.39729495739813 + - type: nauc_ndcg_at_1_max + value: 44.758696800866346 + - type: nauc_ndcg_at_1_std + value: 2.574147496045342 + - type: nauc_ndcg_at_20_diff1 + value: 59.07286332574637 + - type: nauc_ndcg_at_20_max + value: 46.956257681706255 + - type: nauc_ndcg_at_20_std + value: 6.204842712266714 + - type: nauc_ndcg_at_3_diff1 + value: 59.681604757964024 + - type: nauc_ndcg_at_3_max + value: 43.90998875056703 + - type: nauc_ndcg_at_3_std + value: 2.0078194423874485 + - type: nauc_ndcg_at_5_diff1 + value: 59.55490127342846 + - type: nauc_ndcg_at_5_max + value: 45.33113431677653 + - type: nauc_ndcg_at_5_std + value: 3.378002916809058 + - type: nauc_precision_at_1000_diff1 + value: -14.884956709081617 + - type: nauc_precision_at_1000_max + value: 18.98095971115716 + - type: nauc_precision_at_1000_std + value: 23.789090645369622 + - type: nauc_precision_at_100_diff1 + value: -10.816559367799492 + - type: nauc_precision_at_100_max + value: 24.897619006149043 + - type: nauc_precision_at_100_std + value: 29.061470361467258 + - type: nauc_precision_at_10_diff1 + value: 7.787651610995103 + - type: nauc_precision_at_10_max + value: 33.906978421485725 + - type: nauc_precision_at_10_std + value: 21.393195923303523 + - type: nauc_precision_at_1_diff1 + value: 63.39729495739813 + - type: nauc_precision_at_1_max + value: 44.758696800866346 + - type: nauc_precision_at_1_std + value: 2.574147496045342 + - type: nauc_precision_at_20_diff1 + value: 0.2668276229984366 + - type: nauc_precision_at_20_max + value: 31.875194290987903 + - type: nauc_precision_at_20_std + value: 27.058324844498895 + - type: nauc_precision_at_3_diff1 + value: 32.666541380773666 + - type: nauc_precision_at_3_max + value: 39.98649542781057 + - type: nauc_precision_at_3_std + value: 8.490073623416471 + - type: nauc_precision_at_5_diff1 + value: 20.711670056819955 + - type: nauc_precision_at_5_max + value: 35.71939539778521 + - type: nauc_precision_at_5_std + value: 13.471668992214287 + - type: nauc_recall_at_1000_diff1 + value: 62.892298858056826 + - type: nauc_recall_at_1000_max + value: 75.77988246906914 + - type: nauc_recall_at_1000_std + value: 63.301158848319986 + - type: nauc_recall_at_100_diff1 + value: 50.719790043186165 + - type: nauc_recall_at_100_max + value: 61.12299499874101 + - type: nauc_recall_at_100_std + value: 38.38009197782247 + - type: nauc_recall_at_10_diff1 + value: 50.684877427596774 + - type: nauc_recall_at_10_max + value: 48.10578874723287 + - type: nauc_recall_at_10_std + value: 10.277736318694416 + - type: nauc_recall_at_1_diff1 + value: 63.29885856726938 + - type: nauc_recall_at_1_max + value: 38.19351589152847 + - type: nauc_recall_at_1_std + value: -0.23987289074118623 + - type: nauc_recall_at_20_diff1 + value: 49.256639082753146 + - type: nauc_recall_at_20_max + value: 51.552559594046635 + - type: nauc_recall_at_20_std + value: 20.05675162623505 + - type: nauc_recall_at_3_diff1 + value: 55.478536684903254 + - type: nauc_recall_at_3_max + value: 40.817342729367454 + - type: nauc_recall_at_3_std + value: 0.17901094170760476 + - type: nauc_recall_at_5_diff1 + value: 54.58027041274638 + - type: nauc_recall_at_5_max + value: 44.316981753662745 + - type: nauc_recall_at_5_std + value: 4.033411279046144 + - type: ndcg_at_1 + value: 51.285000000000004 + - type: ndcg_at_10 + value: 63.605000000000004 + - type: ndcg_at_100 + value: 67.11200000000001 + - type: ndcg_at_1000 + value: 67.948 + - type: ndcg_at_20 + value: 65.22099999999999 + - type: ndcg_at_3 + value: 58.506 + - type: ndcg_at_5 + value: 60.973 + - type: precision_at_1 + value: 51.285000000000004 + - type: precision_at_10 + value: 10.062999999999999 + - type: precision_at_100 + value: 1.2710000000000001 + - type: precision_at_1000 + value: 0.13799999999999998 + - type: precision_at_20 + value: 5.555000000000001 + - type: precision_at_3 + value: 25.977 + - type: precision_at_5 + value: 17.605 + - type: recall_at_1 + value: 44.778 + - type: recall_at_10 + value: 77.238 + - type: recall_at_100 + value: 92.113 + - type: recall_at_1000 + value: 97.89099999999999 + - type: recall_at_20 + value: 83.202 + - type: recall_at_3 + value: 63.532 + - type: recall_at_5 + value: 69.571 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackGisRetrieval (default) + revision: 5003b3064772da1887988e05400cf3806fe491f2 + split: test + type: mteb/cqadupstack-gis + metrics: + - type: main_score + value: 44.948 + - type: map_at_1 + value: 31.04 + - type: map_at_10 + value: 39.848 + - type: map_at_100 + value: 40.896 + - type: map_at_1000 + value: 40.967 + - type: map_at_20 + value: 40.438 + - type: map_at_3 + value: 37.063 + - type: map_at_5 + value: 38.757000000000005 + - type: mrr_at_1 + value: 33.44632768361582 + - type: mrr_at_10 + value: 42.21011568469193 + - type: mrr_at_100 + value: 43.1069719291059 + - type: mrr_at_1000 + value: 43.155829177829894 + - type: mrr_at_20 + value: 42.72957678143432 + - type: mrr_at_3 + value: 39.64218455743879 + - type: mrr_at_5 + value: 41.274952919020684 + - type: nauc_map_at_1000_diff1 + value: 48.00849889614142 + - type: nauc_map_at_1000_max + value: 31.97006878564366 + - type: nauc_map_at_1000_std + value: 1.1696545540432803 + - type: nauc_map_at_100_diff1 + value: 48.003071640167846 + - type: nauc_map_at_100_max + value: 31.979308717491595 + - type: nauc_map_at_100_std + value: 1.1891534903322152 + - type: nauc_map_at_10_diff1 + value: 48.15386552887778 + - type: nauc_map_at_10_max + value: 31.76230068741604 + - type: nauc_map_at_10_std + value: 0.817902137679551 + - type: nauc_map_at_1_diff1 + value: 53.07565456846688 + - type: nauc_map_at_1_max + value: 29.9195663940298 + - type: nauc_map_at_1_std + value: -2.3576346178097927 + - type: nauc_map_at_20_diff1 + value: 48.08024890348257 + - type: nauc_map_at_20_max + value: 31.933158846818 + - type: nauc_map_at_20_std + value: 1.0243217944649639 + - type: nauc_map_at_3_diff1 + value: 48.78680143848761 + - type: nauc_map_at_3_max + value: 30.564584864919848 + - type: nauc_map_at_3_std + value: -1.0076685799927312 + - type: nauc_map_at_5_diff1 + value: 48.52947693708406 + - type: nauc_map_at_5_max + value: 31.453309890862247 + - type: nauc_map_at_5_std + value: 0.16728890552183778 + - type: nauc_mrr_at_1000_diff1 + value: 46.88318707147919 + - type: nauc_mrr_at_1000_max + value: 33.61920633928203 + - type: nauc_mrr_at_1000_std + value: 3.687572823577428 + - type: nauc_mrr_at_100_diff1 + value: 46.86863389715309 + - type: nauc_mrr_at_100_max + value: 33.638956800639114 + - type: nauc_mrr_at_100_std + value: 3.7193070910704784 + - type: nauc_mrr_at_10_diff1 + value: 47.040515098594845 + - type: nauc_mrr_at_10_max + value: 33.644894798310574 + - type: nauc_mrr_at_10_std + value: 3.5999062398229107 + - type: nauc_mrr_at_1_diff1 + value: 51.154492674437925 + - type: nauc_mrr_at_1_max + value: 32.73966183856691 + - type: nauc_mrr_at_1_std + value: 1.0119408661264722 + - type: nauc_mrr_at_20_diff1 + value: 46.88926670209024 + - type: nauc_mrr_at_20_max + value: 33.577570615335524 + - type: nauc_mrr_at_20_std + value: 3.6250024194666204 + - type: nauc_mrr_at_3_diff1 + value: 47.311477370268854 + - type: nauc_mrr_at_3_max + value: 32.692914296795216 + - type: nauc_mrr_at_3_std + value: 2.2626862819143856 + - type: nauc_mrr_at_5_diff1 + value: 47.18744555316655 + - type: nauc_mrr_at_5_max + value: 33.45591954848659 + - type: nauc_mrr_at_5_std + value: 3.229394882887573 + - type: nauc_ndcg_at_1000_diff1 + value: 45.951566512917516 + - type: nauc_ndcg_at_1000_max + value: 33.43012408537567 + - type: nauc_ndcg_at_1000_std + value: 4.31643781436298 + - type: nauc_ndcg_at_100_diff1 + value: 45.456007487383765 + - type: nauc_ndcg_at_100_max + value: 33.83179122623623 + - type: nauc_ndcg_at_100_std + value: 5.379900101535729 + - type: nauc_ndcg_at_10_diff1 + value: 46.000081854341865 + - type: nauc_ndcg_at_10_max + value: 33.03276312112147 + - type: nauc_ndcg_at_10_std + value: 3.506762476640791 + - type: nauc_ndcg_at_1_diff1 + value: 51.154492674437925 + - type: nauc_ndcg_at_1_max + value: 32.73966183856691 + - type: nauc_ndcg_at_1_std + value: 1.0119408661264722 + - type: nauc_ndcg_at_20_diff1 + value: 45.573888372579795 + - type: nauc_ndcg_at_20_max + value: 33.380340508347636 + - type: nauc_ndcg_at_20_std + value: 4.13712193339463 + - type: nauc_ndcg_at_3_diff1 + value: 47.219537672268345 + - type: nauc_ndcg_at_3_max + value: 31.25427095001378 + - type: nauc_ndcg_at_3_std + value: 0.19650122528013553 + - type: nauc_ndcg_at_5_diff1 + value: 46.824607904034984 + - type: nauc_ndcg_at_5_max + value: 32.57290649460352 + - type: nauc_ndcg_at_5_std + value: 2.2108841210375423 + - type: nauc_precision_at_1000_diff1 + value: -9.05766409125184 + - type: nauc_precision_at_1000_max + value: 10.263044699939536 + - type: nauc_precision_at_1000_std + value: 14.336533795297726 + - type: nauc_precision_at_100_diff1 + value: 5.312802430858622 + - type: nauc_precision_at_100_max + value: 25.236257571579397 + - type: nauc_precision_at_100_std + value: 23.90999849032771 + - type: nauc_precision_at_10_diff1 + value: 28.218764261414282 + - type: nauc_precision_at_10_max + value: 34.88412816136581 + - type: nauc_precision_at_10_std + value: 14.816816782229475 + - type: nauc_precision_at_1_diff1 + value: 51.154492674437925 + - type: nauc_precision_at_1_max + value: 32.73966183856691 + - type: nauc_precision_at_1_std + value: 1.0119408661264722 + - type: nauc_precision_at_20_diff1 + value: 21.987075117572203 + - type: nauc_precision_at_20_max + value: 34.00807588379176 + - type: nauc_precision_at_20_std + value: 17.952393737171324 + - type: nauc_precision_at_3_diff1 + value: 38.241632453674086 + - type: nauc_precision_at_3_max + value: 34.02447106776669 + - type: nauc_precision_at_3_std + value: 5.221908095891579 + - type: nauc_precision_at_5_diff1 + value: 34.959017294772075 + - type: nauc_precision_at_5_max + value: 35.786475752990945 + - type: nauc_precision_at_5_std + value: 10.037526622256422 + - type: nauc_recall_at_1000_diff1 + value: 30.939385299834992 + - type: nauc_recall_at_1000_max + value: 42.335514267683294 + - type: nauc_recall_at_1000_std + value: 29.62033644828526 + - type: nauc_recall_at_100_diff1 + value: 31.056126901046007 + - type: nauc_recall_at_100_max + value: 41.04565569591625 + - type: nauc_recall_at_100_std + value: 28.870201603175804 + - type: nauc_recall_at_10_diff1 + value: 38.698128777796995 + - type: nauc_recall_at_10_max + value: 34.1057229273387 + - type: nauc_recall_at_10_std + value: 9.303813133656739 + - type: nauc_recall_at_1_diff1 + value: 53.07565456846688 + - type: nauc_recall_at_1_max + value: 29.9195663940298 + - type: nauc_recall_at_1_std + value: -2.3576346178097927 + - type: nauc_recall_at_20_diff1 + value: 35.88498518647808 + - type: nauc_recall_at_20_max + value: 35.605146792803275 + - type: nauc_recall_at_20_std + value: 12.755207480142033 + - type: nauc_recall_at_3_diff1 + value: 43.447279197430376 + - type: nauc_recall_at_3_max + value: 30.100206519419526 + - type: nauc_recall_at_3_std + value: 0.6341561653291098 + - type: nauc_recall_at_5_diff1 + value: 41.828402463798966 + - type: nauc_recall_at_5_max + value: 33.18644866462676 + - type: nauc_recall_at_5_std + value: 5.551496303251337 + - type: ndcg_at_1 + value: 33.446 + - type: ndcg_at_10 + value: 44.948 + - type: ndcg_at_100 + value: 49.992 + - type: ndcg_at_1000 + value: 51.675000000000004 + - type: ndcg_at_20 + value: 46.936 + - type: ndcg_at_3 + value: 39.653 + - type: ndcg_at_5 + value: 42.542 + - type: precision_at_1 + value: 33.446 + - type: precision_at_10 + value: 6.701 + - type: precision_at_100 + value: 0.971 + - type: precision_at_1000 + value: 0.11499999999999999 + - type: precision_at_20 + value: 3.819 + - type: precision_at_3 + value: 16.422 + - type: precision_at_5 + value: 11.525 + - type: recall_at_1 + value: 31.04 + - type: recall_at_10 + value: 58.269000000000005 + - type: recall_at_100 + value: 81.21000000000001 + - type: recall_at_1000 + value: 93.69800000000001 + - type: recall_at_20 + value: 65.725 + - type: recall_at_3 + value: 44.282 + - type: recall_at_5 + value: 51.256 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackMathematicaRetrieval (default) + revision: 90fceea13679c63fe563ded68f3b6f06e50061de + split: test + type: mteb/cqadupstack-mathematica + metrics: + - type: main_score + value: 35.128 + - type: map_at_1 + value: 20.381 + - type: map_at_10 + value: 29.670999999999996 + - type: map_at_100 + value: 30.994 + - type: map_at_1000 + value: 31.098 + - type: map_at_20 + value: 30.339 + - type: map_at_3 + value: 27.184 + - type: map_at_5 + value: 28.499000000000002 + - type: mrr_at_1 + value: 24.875621890547265 + - type: mrr_at_10 + value: 34.26197386085446 + - type: mrr_at_100 + value: 35.32144040146426 + - type: mrr_at_1000 + value: 35.37132797649873 + - type: mrr_at_20 + value: 34.8861975809379 + - type: mrr_at_3 + value: 32.08955223880599 + - type: mrr_at_5 + value: 33.29601990049751 + - type: nauc_map_at_1000_diff1 + value: 36.994845642818866 + - type: nauc_map_at_1000_max + value: 26.400354178727248 + - type: nauc_map_at_1000_std + value: 6.952454580638288 + - type: nauc_map_at_100_diff1 + value: 37.0088886165848 + - type: nauc_map_at_100_max + value: 26.41718320702855 + - type: nauc_map_at_100_std + value: 6.966989714148378 + - type: nauc_map_at_10_diff1 + value: 37.03155907561462 + - type: nauc_map_at_10_max + value: 26.020108129019203 + - type: nauc_map_at_10_std + value: 6.470815384430856 + - type: nauc_map_at_1_diff1 + value: 42.8208312334657 + - type: nauc_map_at_1_max + value: 27.0899809589084 + - type: nauc_map_at_1_std + value: 6.664045775598297 + - type: nauc_map_at_20_diff1 + value: 37.0161129647804 + - type: nauc_map_at_20_max + value: 26.32681186624649 + - type: nauc_map_at_20_std + value: 6.834736376599413 + - type: nauc_map_at_3_diff1 + value: 37.9711466146252 + - type: nauc_map_at_3_max + value: 26.014944389898158 + - type: nauc_map_at_3_std + value: 6.1503277538171846 + - type: nauc_map_at_5_diff1 + value: 37.48153651379401 + - type: nauc_map_at_5_max + value: 25.991920954101783 + - type: nauc_map_at_5_std + value: 6.42580077869363 + - type: nauc_mrr_at_1000_diff1 + value: 37.48018847168505 + - type: nauc_mrr_at_1000_max + value: 28.512902855492445 + - type: nauc_mrr_at_1000_std + value: 7.020126810110982 + - type: nauc_mrr_at_100_diff1 + value: 37.474647802683 + - type: nauc_mrr_at_100_max + value: 28.520400645653215 + - type: nauc_mrr_at_100_std + value: 7.018371716140309 + - type: nauc_mrr_at_10_diff1 + value: 37.29076623664705 + - type: nauc_mrr_at_10_max + value: 28.350027307831148 + - type: nauc_mrr_at_10_std + value: 6.716677805638879 + - type: nauc_mrr_at_1_diff1 + value: 43.77099265761195 + - type: nauc_mrr_at_1_max + value: 29.844718152190307 + - type: nauc_mrr_at_1_std + value: 6.4443116319739495 + - type: nauc_mrr_at_20_diff1 + value: 37.42124667315709 + - type: nauc_mrr_at_20_max + value: 28.466855908632592 + - type: nauc_mrr_at_20_std + value: 6.950483276094682 + - type: nauc_mrr_at_3_diff1 + value: 38.21247480395254 + - type: nauc_mrr_at_3_max + value: 28.460132667433953 + - type: nauc_mrr_at_3_std + value: 6.367936411383797 + - type: nauc_mrr_at_5_diff1 + value: 37.55913850771685 + - type: nauc_mrr_at_5_max + value: 28.248143682520748 + - type: nauc_mrr_at_5_std + value: 6.515553418798696 + - type: nauc_ndcg_at_1000_diff1 + value: 35.02742076410184 + - type: nauc_ndcg_at_1000_max + value: 27.975850737369562 + - type: nauc_ndcg_at_1000_std + value: 8.768009682357782 + - type: nauc_ndcg_at_100_diff1 + value: 35.12205238430365 + - type: nauc_ndcg_at_100_max + value: 27.892922478092775 + - type: nauc_ndcg_at_100_std + value: 8.800068503606802 + - type: nauc_ndcg_at_10_diff1 + value: 34.81773514368173 + - type: nauc_ndcg_at_10_max + value: 26.383602189421662 + - type: nauc_ndcg_at_10_std + value: 6.585969947394481 + - type: nauc_ndcg_at_1_diff1 + value: 43.77099265761195 + - type: nauc_ndcg_at_1_max + value: 29.844718152190307 + - type: nauc_ndcg_at_1_std + value: 6.4443116319739495 + - type: nauc_ndcg_at_20_diff1 + value: 35.01274512761417 + - type: nauc_ndcg_at_20_max + value: 27.290850084382285 + - type: nauc_ndcg_at_20_std + value: 7.69668586721584 + - type: nauc_ndcg_at_3_diff1 + value: 36.291798885239125 + - type: nauc_ndcg_at_3_max + value: 26.513693615681362 + - type: nauc_ndcg_at_3_std + value: 5.850342970264751 + - type: nauc_ndcg_at_5_diff1 + value: 35.48277611349641 + - type: nauc_ndcg_at_5_max + value: 26.2342176431082 + - type: nauc_ndcg_at_5_std + value: 6.41209023583701 + - type: nauc_precision_at_1000_diff1 + value: -4.3060123036217135 + - type: nauc_precision_at_1000_max + value: 4.251658612344567 + - type: nauc_precision_at_1000_std + value: -0.922924990639695 + - type: nauc_precision_at_100_diff1 + value: 4.6793200358061595 + - type: nauc_precision_at_100_max + value: 14.352547031067807 + - type: nauc_precision_at_100_std + value: 5.450076012249245 + - type: nauc_precision_at_10_diff1 + value: 20.488643375814117 + - type: nauc_precision_at_10_max + value: 22.40679068588318 + - type: nauc_precision_at_10_std + value: 5.363901147786655 + - type: nauc_precision_at_1_diff1 + value: 43.77099265761195 + - type: nauc_precision_at_1_max + value: 29.844718152190307 + - type: nauc_precision_at_1_std + value: 6.4443116319739495 + - type: nauc_precision_at_20_diff1 + value: 15.638517626666676 + - type: nauc_precision_at_20_max + value: 21.59028385020512 + - type: nauc_precision_at_20_std + value: 6.675823799140996 + - type: nauc_precision_at_3_diff1 + value: 28.516519789499096 + - type: nauc_precision_at_3_max + value: 24.433121214677897 + - type: nauc_precision_at_3_std + value: 4.032595263152071 + - type: nauc_precision_at_5_diff1 + value: 24.43948976197699 + - type: nauc_precision_at_5_max + value: 23.82210390394073 + - type: nauc_precision_at_5_std + value: 5.3426742695941085 + - type: nauc_recall_at_1000_diff1 + value: 15.353663120760896 + - type: nauc_recall_at_1000_max + value: 43.17454170382031 + - type: nauc_recall_at_1000_std + value: 35.89085371815144 + - type: nauc_recall_at_100_diff1 + value: 26.542582837417832 + - type: nauc_recall_at_100_max + value: 30.9917572493189 + - type: nauc_recall_at_100_std + value: 19.053889738697897 + - type: nauc_recall_at_10_diff1 + value: 27.2060446840431 + - type: nauc_recall_at_10_max + value: 24.153735802255362 + - type: nauc_recall_at_10_std + value: 6.030839218050536 + - type: nauc_recall_at_1_diff1 + value: 42.8208312334657 + - type: nauc_recall_at_1_max + value: 27.0899809589084 + - type: nauc_recall_at_1_std + value: 6.664045775598297 + - type: nauc_recall_at_20_diff1 + value: 27.759504553176367 + - type: nauc_recall_at_20_max + value: 27.36449521348782 + - type: nauc_recall_at_20_std + value: 10.145560708522265 + - type: nauc_recall_at_3_diff1 + value: 31.71806454114658 + - type: nauc_recall_at_3_max + value: 24.534464900357715 + - type: nauc_recall_at_3_std + value: 5.509383867948696 + - type: nauc_recall_at_5_diff1 + value: 29.764358566664338 + - type: nauc_recall_at_5_max + value: 23.870445841212657 + - type: nauc_recall_at_5_std + value: 6.230268761479784 + - type: ndcg_at_1 + value: 24.876 + - type: ndcg_at_10 + value: 35.128 + - type: ndcg_at_100 + value: 41.160999999999994 + - type: ndcg_at_1000 + value: 43.558 + - type: ndcg_at_20 + value: 37.377 + - type: ndcg_at_3 + value: 30.833 + - type: ndcg_at_5 + value: 32.695 + - type: precision_at_1 + value: 24.876 + - type: precision_at_10 + value: 6.368 + - type: precision_at_100 + value: 1.082 + - type: precision_at_1000 + value: 0.13999999999999999 + - type: precision_at_20 + value: 3.8309999999999995 + - type: precision_at_3 + value: 15.174000000000001 + - type: precision_at_5 + value: 10.572 + - type: recall_at_1 + value: 20.381 + - type: recall_at_10 + value: 47.138999999999996 + - type: recall_at_100 + value: 72.752 + - type: recall_at_1000 + value: 89.97 + - type: recall_at_20 + value: 55.089 + - type: recall_at_3 + value: 34.887 + - type: recall_at_5 + value: 39.853 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackPhysicsRetrieval (default) + revision: 79531abbd1fb92d06c6d6315a0cbbbf5bb247ea4 + split: test + type: mteb/cqadupstack-physics + metrics: + - type: main_score + value: 50.629999999999995 + - type: map_at_1 + value: 33.497 + - type: map_at_10 + value: 44.545 + - type: map_at_100 + value: 45.861000000000004 + - type: map_at_1000 + value: 45.969 + - type: map_at_20 + value: 45.28 + - type: map_at_3 + value: 41.05 + - type: map_at_5 + value: 43.062 + - type: mrr_at_1 + value: 40.423484119345524 + - type: mrr_at_10 + value: 50.22106115465112 + - type: mrr_at_100 + value: 50.97400445844568 + - type: mrr_at_1000 + value: 51.01054117056053 + - type: mrr_at_20 + value: 50.63019986028454 + - type: mrr_at_3 + value: 47.70612768687839 + - type: mrr_at_5 + value: 49.255694578119936 + - type: nauc_map_at_1000_diff1 + value: 54.20092757686139 + - type: nauc_map_at_1000_max + value: 39.83173871512557 + - type: nauc_map_at_1000_std + value: 1.2235866643835682 + - type: nauc_map_at_100_diff1 + value: 54.183059354916516 + - type: nauc_map_at_100_max + value: 39.824429278367326 + - type: nauc_map_at_100_std + value: 1.1962361816341982 + - type: nauc_map_at_10_diff1 + value: 54.181220324498334 + - type: nauc_map_at_10_max + value: 39.37488214775074 + - type: nauc_map_at_10_std + value: 0.449166845027239 + - type: nauc_map_at_1_diff1 + value: 58.9295062000639 + - type: nauc_map_at_1_max + value: 36.31283188494571 + - type: nauc_map_at_1_std + value: -4.774568385823226 + - type: nauc_map_at_20_diff1 + value: 54.208450723508086 + - type: nauc_map_at_20_max + value: 39.5501483019714 + - type: nauc_map_at_20_std + value: 0.7503146754904455 + - type: nauc_map_at_3_diff1 + value: 55.11394188646683 + - type: nauc_map_at_3_max + value: 39.08380991583403 + - type: nauc_map_at_3_std + value: 0.04479084007353637 + - type: nauc_map_at_5_diff1 + value: 54.31395773703653 + - type: nauc_map_at_5_max + value: 39.17683595910297 + - type: nauc_map_at_5_std + value: -0.0044853748214373455 + - type: nauc_mrr_at_1000_diff1 + value: 53.70635191547364 + - type: nauc_mrr_at_1000_max + value: 41.147112546476265 + - type: nauc_mrr_at_1000_std + value: 2.2225275965250892 + - type: nauc_mrr_at_100_diff1 + value: 53.68723611702497 + - type: nauc_mrr_at_100_max + value: 41.14620122736365 + - type: nauc_mrr_at_100_std + value: 2.2209413391816732 + - type: nauc_mrr_at_10_diff1 + value: 53.596524939380274 + - type: nauc_mrr_at_10_max + value: 40.90108780230196 + - type: nauc_mrr_at_10_std + value: 1.8279923348193807 + - type: nauc_mrr_at_1_diff1 + value: 58.01850581846227 + - type: nauc_mrr_at_1_max + value: 40.64859507463277 + - type: nauc_mrr_at_1_std + value: -0.46331678451254144 + - type: nauc_mrr_at_20_diff1 + value: 53.60646492043754 + - type: nauc_mrr_at_20_max + value: 40.97670725696802 + - type: nauc_mrr_at_20_std + value: 1.9984004205114907 + - type: nauc_mrr_at_3_diff1 + value: 54.01504269498941 + - type: nauc_mrr_at_3_max + value: 41.553653128998036 + - type: nauc_mrr_at_3_std + value: 2.848139807968868 + - type: nauc_mrr_at_5_diff1 + value: 53.575182274960355 + - type: nauc_mrr_at_5_max + value: 41.14109670598855 + - type: nauc_mrr_at_5_std + value: 1.8672107703418455 + - type: nauc_ndcg_at_1000_diff1 + value: 52.70533657035467 + - type: nauc_ndcg_at_1000_max + value: 41.25627414408392 + - type: nauc_ndcg_at_1000_std + value: 4.263369832603922 + - type: nauc_ndcg_at_100_diff1 + value: 52.237181363030615 + - type: nauc_ndcg_at_100_max + value: 41.25394087259199 + - type: nauc_ndcg_at_100_std + value: 4.3455920391450915 + - type: nauc_ndcg_at_10_diff1 + value: 52.093168049256164 + - type: nauc_ndcg_at_10_max + value: 39.46594517979851 + - type: nauc_ndcg_at_10_std + value: 1.3023915669308508 + - type: nauc_ndcg_at_1_diff1 + value: 58.01850581846227 + - type: nauc_ndcg_at_1_max + value: 40.64859507463277 + - type: nauc_ndcg_at_1_std + value: -0.46331678451254144 + - type: nauc_ndcg_at_20_diff1 + value: 52.137725285176494 + - type: nauc_ndcg_at_20_max + value: 39.87516740071308 + - type: nauc_ndcg_at_20_std + value: 2.059559762129985 + - type: nauc_ndcg_at_3_diff1 + value: 53.11030340684372 + - type: nauc_ndcg_at_3_max + value: 40.48904161228481 + - type: nauc_ndcg_at_3_std + value: 2.587550031562688 + - type: nauc_ndcg_at_5_diff1 + value: 52.23131807447599 + - type: nauc_ndcg_at_5_max + value: 39.834305848624545 + - type: nauc_ndcg_at_5_std + value: 1.4058962563675212 + - type: nauc_precision_at_1000_diff1 + value: -16.161300936259565 + - type: nauc_precision_at_1000_max + value: -0.8115303572148305 + - type: nauc_precision_at_1000_std + value: 13.92396647704085 + - type: nauc_precision_at_100_diff1 + value: -7.686479593960298 + - type: nauc_precision_at_100_max + value: 11.167912165722198 + - type: nauc_precision_at_100_std + value: 19.69851848467226 + - type: nauc_precision_at_10_diff1 + value: 13.11274789048853 + - type: nauc_precision_at_10_max + value: 25.966889003874282 + - type: nauc_precision_at_10_std + value: 14.48757299965834 + - type: nauc_precision_at_1_diff1 + value: 58.01850581846227 + - type: nauc_precision_at_1_max + value: 40.64859507463277 + - type: nauc_precision_at_1_std + value: -0.46331678451254144 + - type: nauc_precision_at_20_diff1 + value: 6.417726564438533 + - type: nauc_precision_at_20_max + value: 20.748304258867293 + - type: nauc_precision_at_20_std + value: 16.26691325320734 + - type: nauc_precision_at_3_diff1 + value: 33.58615409368469 + - type: nauc_precision_at_3_max + value: 37.22470451801486 + - type: nauc_precision_at_3_std + value: 12.305903402160746 + - type: nauc_precision_at_5_diff1 + value: 22.241354445663152 + - type: nauc_precision_at_5_max + value: 31.19518669721679 + - type: nauc_precision_at_5_std + value: 12.236619256927176 + - type: nauc_recall_at_1000_diff1 + value: 37.37272596834472 + - type: nauc_recall_at_1000_max + value: 57.05674570554847 + - type: nauc_recall_at_1000_std + value: 55.18895561738797 + - type: nauc_recall_at_100_diff1 + value: 36.62084030918699 + - type: nauc_recall_at_100_max + value: 43.66289378465585 + - type: nauc_recall_at_100_std + value: 23.676638493814732 + - type: nauc_recall_at_10_diff1 + value: 42.636818915384524 + - type: nauc_recall_at_10_max + value: 34.24973591817277 + - type: nauc_recall_at_10_std + value: 0.7069961094408362 + - type: nauc_recall_at_1_diff1 + value: 58.9295062000639 + - type: nauc_recall_at_1_max + value: 36.31283188494571 + - type: nauc_recall_at_1_std + value: -4.774568385823226 + - type: nauc_recall_at_20_diff1 + value: 41.41011976350058 + - type: nauc_recall_at_20_max + value: 35.24092182522975 + - type: nauc_recall_at_20_std + value: 3.8409076588443765 + - type: nauc_recall_at_3_diff1 + value: 49.097968740476084 + - type: nauc_recall_at_3_max + value: 38.2237826032796 + - type: nauc_recall_at_3_std + value: 2.6696770347153063 + - type: nauc_recall_at_5_diff1 + value: 45.28725903950895 + - type: nauc_recall_at_5_max + value: 36.434173396667326 + - type: nauc_recall_at_5_std + value: -0.020592087753619558 + - type: ndcg_at_1 + value: 40.422999999999995 + - type: ndcg_at_10 + value: 50.629999999999995 + - type: ndcg_at_100 + value: 55.879 + - type: ndcg_at_1000 + value: 57.66 + - type: ndcg_at_20 + value: 52.658 + - type: ndcg_at_3 + value: 45.401 + - type: ndcg_at_5 + value: 48.048 + - type: precision_at_1 + value: 40.422999999999995 + - type: precision_at_10 + value: 9.115 + - type: precision_at_100 + value: 1.3599999999999999 + - type: precision_at_1000 + value: 0.16999999999999998 + - type: precision_at_20 + value: 5.244999999999999 + - type: precision_at_3 + value: 21.367 + - type: precision_at_5 + value: 15.187999999999999 + - type: recall_at_1 + value: 33.497 + - type: recall_at_10 + value: 62.717999999999996 + - type: recall_at_100 + value: 84.661 + - type: recall_at_1000 + value: 95.91300000000001 + - type: recall_at_20 + value: 69.71300000000001 + - type: recall_at_3 + value: 48.27 + - type: recall_at_5 + value: 55.088 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackProgrammersRetrieval (default) + revision: 6184bc1440d2dbc7612be22b50686b8826d22b32 + split: test + type: mteb/cqadupstack-programmers + metrics: + - type: main_score + value: 46.836 + - type: map_at_1 + value: 29.608 + - type: map_at_10 + value: 40.599000000000004 + - type: map_at_100 + value: 41.979 + - type: map_at_1000 + value: 42.074 + - type: map_at_20 + value: 41.384 + - type: map_at_3 + value: 37.112 + - type: map_at_5 + value: 39.293 + - type: mrr_at_1 + value: 36.986301369863014 + - type: mrr_at_10 + value: 46.747798434442245 + - type: mrr_at_100 + value: 47.577293043981726 + - type: mrr_at_1000 + value: 47.619243282521914 + - type: mrr_at_20 + value: 47.233194463131554 + - type: mrr_at_3 + value: 44.06392694063925 + - type: mrr_at_5 + value: 45.81050228310499 + - type: nauc_map_at_1000_diff1 + value: 49.10316795557811 + - type: nauc_map_at_1000_max + value: 41.671749246544564 + - type: nauc_map_at_1000_std + value: 10.038630192417667 + - type: nauc_map_at_100_diff1 + value: 49.07942830525222 + - type: nauc_map_at_100_max + value: 41.670180901024025 + - type: nauc_map_at_100_std + value: 10.077420063502295 + - type: nauc_map_at_10_diff1 + value: 49.38813383833756 + - type: nauc_map_at_10_max + value: 41.22459028327236 + - type: nauc_map_at_10_std + value: 9.174478184975762 + - type: nauc_map_at_1_diff1 + value: 55.71321955074087 + - type: nauc_map_at_1_max + value: 36.11731976033434 + - type: nauc_map_at_1_std + value: 1.8068953588006964 + - type: nauc_map_at_20_diff1 + value: 49.18360865264359 + - type: nauc_map_at_20_max + value: 41.475699446781725 + - type: nauc_map_at_20_std + value: 9.578778511492251 + - type: nauc_map_at_3_diff1 + value: 49.91276350979302 + - type: nauc_map_at_3_max + value: 39.96695976638339 + - type: nauc_map_at_3_std + value: 7.236097990907886 + - type: nauc_map_at_5_diff1 + value: 49.65208799219414 + - type: nauc_map_at_5_max + value: 41.04361693268516 + - type: nauc_map_at_5_std + value: 8.562411756471908 + - type: nauc_mrr_at_1000_diff1 + value: 47.09937253453218 + - type: nauc_mrr_at_1000_max + value: 44.08795270671583 + - type: nauc_mrr_at_1000_std + value: 13.617996848349536 + - type: nauc_mrr_at_100_diff1 + value: 47.07785704747128 + - type: nauc_mrr_at_100_max + value: 44.07997652095115 + - type: nauc_mrr_at_100_std + value: 13.64220828332083 + - type: nauc_mrr_at_10_diff1 + value: 47.11380729360969 + - type: nauc_mrr_at_10_max + value: 44.196799809377396 + - type: nauc_mrr_at_10_std + value: 13.52183257714539 + - type: nauc_mrr_at_1_diff1 + value: 52.99521852178884 + - type: nauc_mrr_at_1_max + value: 41.33946116723129 + - type: nauc_mrr_at_1_std + value: 7.638017536406832 + - type: nauc_mrr_at_20_diff1 + value: 47.08378847581141 + - type: nauc_mrr_at_20_max + value: 44.109343504693044 + - type: nauc_mrr_at_20_std + value: 13.545682520572061 + - type: nauc_mrr_at_3_diff1 + value: 47.66063968516477 + - type: nauc_mrr_at_3_max + value: 44.2825861106925 + - type: nauc_mrr_at_3_std + value: 12.813104863306343 + - type: nauc_mrr_at_5_diff1 + value: 46.92924998178392 + - type: nauc_mrr_at_5_max + value: 44.109084887738916 + - type: nauc_mrr_at_5_std + value: 13.22363129408731 + - type: nauc_ndcg_at_1000_diff1 + value: 46.78586829943697 + - type: nauc_ndcg_at_1000_max + value: 43.58472176957621 + - type: nauc_ndcg_at_1000_std + value: 14.328001434471885 + - type: nauc_ndcg_at_100_diff1 + value: 46.35960778900108 + - type: nauc_ndcg_at_100_max + value: 43.69339368576078 + - type: nauc_ndcg_at_100_std + value: 15.310177450346195 + - type: nauc_ndcg_at_10_diff1 + value: 46.874802616685955 + - type: nauc_ndcg_at_10_max + value: 42.736773969244695 + - type: nauc_ndcg_at_10_std + value: 12.450242742509857 + - type: nauc_ndcg_at_1_diff1 + value: 52.99521852178884 + - type: nauc_ndcg_at_1_max + value: 41.33946116723129 + - type: nauc_ndcg_at_1_std + value: 7.638017536406832 + - type: nauc_ndcg_at_20_diff1 + value: 46.48325214816742 + - type: nauc_ndcg_at_20_max + value: 43.10920053679779 + - type: nauc_ndcg_at_20_std + value: 13.317430379059624 + - type: nauc_ndcg_at_3_diff1 + value: 47.07113348577298 + - type: nauc_ndcg_at_3_max + value: 41.94339250849152 + - type: nauc_ndcg_at_3_std + value: 10.387978942340968 + - type: nauc_ndcg_at_5_diff1 + value: 46.93778481741133 + - type: nauc_ndcg_at_5_max + value: 42.76083479370739 + - type: nauc_ndcg_at_5_std + value: 11.51504601240572 + - type: nauc_precision_at_1000_diff1 + value: -17.7677637122134 + - type: nauc_precision_at_1000_max + value: -1.3716516229082598 + - type: nauc_precision_at_1000_std + value: 9.428121521222364 + - type: nauc_precision_at_100_diff1 + value: -7.722278929381356 + - type: nauc_precision_at_100_max + value: 14.776676764108688 + - type: nauc_precision_at_100_std + value: 22.600345035167997 + - type: nauc_precision_at_10_diff1 + value: 14.866760669726403 + - type: nauc_precision_at_10_max + value: 33.44108782418061 + - type: nauc_precision_at_10_std + value: 22.27018259763186 + - type: nauc_precision_at_1_diff1 + value: 52.99521852178884 + - type: nauc_precision_at_1_max + value: 41.33946116723129 + - type: nauc_precision_at_1_std + value: 7.638017536406832 + - type: nauc_precision_at_20_diff1 + value: 6.413001218613121 + - type: nauc_precision_at_20_max + value: 28.23517321155765 + - type: nauc_precision_at_20_std + value: 22.46064223000567 + - type: nauc_precision_at_3_diff1 + value: 30.88734764618873 + - type: nauc_precision_at_3_max + value: 41.70431164290239 + - type: nauc_precision_at_3_std + value: 19.542453078764293 + - type: nauc_precision_at_5_diff1 + value: 23.50018424879535 + - type: nauc_precision_at_5_max + value: 39.7524274353907 + - type: nauc_precision_at_5_std + value: 21.945120979815957 + - type: nauc_recall_at_1000_diff1 + value: 25.87337092030495 + - type: nauc_recall_at_1000_max + value: 55.68360675563021 + - type: nauc_recall_at_1000_std + value: 59.08352468869176 + - type: nauc_recall_at_100_diff1 + value: 31.370635254798003 + - type: nauc_recall_at_100_max + value: 44.95908343673895 + - type: nauc_recall_at_100_std + value: 38.686956561690096 + - type: nauc_recall_at_10_diff1 + value: 38.148937664198144 + - type: nauc_recall_at_10_max + value: 40.52122490192401 + - type: nauc_recall_at_10_std + value: 16.92176700918241 + - type: nauc_recall_at_1_diff1 + value: 55.71321955074087 + - type: nauc_recall_at_1_max + value: 36.11731976033434 + - type: nauc_recall_at_1_std + value: 1.8068953588006964 + - type: nauc_recall_at_20_diff1 + value: 35.792860017782495 + - type: nauc_recall_at_20_max + value: 41.279423083925884 + - type: nauc_recall_at_20_std + value: 20.254270911576334 + - type: nauc_recall_at_3_diff1 + value: 43.062160943349234 + - type: nauc_recall_at_3_max + value: 40.457218825619925 + - type: nauc_recall_at_3_std + value: 10.394818250106198 + - type: nauc_recall_at_5_diff1 + value: 40.24389764000816 + - type: nauc_recall_at_5_max + value: 41.255066201614035 + - type: nauc_recall_at_5_std + value: 13.655116018014871 + - type: ndcg_at_1 + value: 36.986000000000004 + - type: ndcg_at_10 + value: 46.836 + - type: ndcg_at_100 + value: 52.425999999999995 + - type: ndcg_at_1000 + value: 54.264 + - type: ndcg_at_20 + value: 49.091 + - type: ndcg_at_3 + value: 41.483 + - type: ndcg_at_5 + value: 44.407000000000004 + - type: precision_at_1 + value: 36.986000000000004 + - type: precision_at_10 + value: 8.459 + - type: precision_at_100 + value: 1.309 + - type: precision_at_1000 + value: 0.163 + - type: precision_at_20 + value: 4.966 + - type: precision_at_3 + value: 19.825 + - type: precision_at_5 + value: 14.247000000000002 + - type: recall_at_1 + value: 29.608 + - type: recall_at_10 + value: 59.428000000000004 + - type: recall_at_100 + value: 83.025 + - type: recall_at_1000 + value: 95.18900000000001 + - type: recall_at_20 + value: 67.363 + - type: recall_at_3 + value: 44.261 + - type: recall_at_5 + value: 52.17699999999999 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackRetrieval (default) + revision: CQADupstackRetrieval_is_a_combined_dataset + split: test + type: CQADupstackRetrieval_is_a_combined_dataset + metrics: + - type: main_score + value: 46.209166666666675 + - type: ndcg_at_10 + value: 46.209166666666675 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackStatsRetrieval (default) + revision: 65ac3a16b8e91f9cee4c9828cc7c335575432a2a + split: test + type: mteb/cqadupstack-stats + metrics: + - type: main_score + value: 40.095 + - type: map_at_1 + value: 28.631 + - type: map_at_10 + value: 35.842 + - type: map_at_100 + value: 36.778 + - type: map_at_1000 + value: 36.881 + - type: map_at_20 + value: 36.367 + - type: map_at_3 + value: 33.593 + - type: map_at_5 + value: 34.831 + - type: mrr_at_1 + value: 32.20858895705521 + - type: mrr_at_10 + value: 38.84762391664232 + - type: mrr_at_100 + value: 39.64050420326875 + - type: mrr_at_1000 + value: 39.710339127925664 + - type: mrr_at_20 + value: 39.301645076766626 + - type: mrr_at_3 + value: 36.78425357873213 + - type: mrr_at_5 + value: 38.00357873210634 + - type: nauc_map_at_1000_diff1 + value: 48.931140515321864 + - type: nauc_map_at_1000_max + value: 44.13502581863409 + - type: nauc_map_at_1000_std + value: 6.747452339686901 + - type: nauc_map_at_100_diff1 + value: 48.93547799674662 + - type: nauc_map_at_100_max + value: 44.10681928382314 + - type: nauc_map_at_100_std + value: 6.730810485140218 + - type: nauc_map_at_10_diff1 + value: 49.021844196409326 + - type: nauc_map_at_10_max + value: 43.63205455207613 + - type: nauc_map_at_10_std + value: 6.032946015596642 + - type: nauc_map_at_1_diff1 + value: 56.40670369173174 + - type: nauc_map_at_1_max + value: 44.28894153655779 + - type: nauc_map_at_1_std + value: 0.7031741175020302 + - type: nauc_map_at_20_diff1 + value: 48.86279230717127 + - type: nauc_map_at_20_max + value: 43.92199417637222 + - type: nauc_map_at_20_std + value: 6.514660332263528 + - type: nauc_map_at_3_diff1 + value: 50.310205836352075 + - type: nauc_map_at_3_max + value: 43.76635060413731 + - type: nauc_map_at_3_std + value: 4.274247771914374 + - type: nauc_map_at_5_diff1 + value: 49.446175642020904 + - type: nauc_map_at_5_max + value: 43.68157266381459 + - type: nauc_map_at_5_std + value: 5.348312742430713 + - type: nauc_mrr_at_1000_diff1 + value: 48.151329887992766 + - type: nauc_mrr_at_1000_max + value: 46.63531166207783 + - type: nauc_mrr_at_1000_std + value: 9.239338850165316 + - type: nauc_mrr_at_100_diff1 + value: 48.148654677932896 + - type: nauc_mrr_at_100_max + value: 46.6302325650526 + - type: nauc_mrr_at_100_std + value: 9.243643499574722 + - type: nauc_mrr_at_10_diff1 + value: 48.20122701114527 + - type: nauc_mrr_at_10_max + value: 46.2713720160932 + - type: nauc_mrr_at_10_std + value: 8.755965334950478 + - type: nauc_mrr_at_1_diff1 + value: 54.52754433723723 + - type: nauc_mrr_at_1_max + value: 48.35064288413123 + - type: nauc_mrr_at_1_std + value: 5.97988808735259 + - type: nauc_mrr_at_20_diff1 + value: 48.111463762473775 + - type: nauc_mrr_at_20_max + value: 46.60149550084685 + - type: nauc_mrr_at_20_std + value: 9.160057023317378 + - type: nauc_mrr_at_3_diff1 + value: 49.0657233493215 + - type: nauc_mrr_at_3_max + value: 46.50752741245066 + - type: nauc_mrr_at_3_std + value: 7.341376278131818 + - type: nauc_mrr_at_5_diff1 + value: 48.40148413751887 + - type: nauc_mrr_at_5_max + value: 46.229816249410014 + - type: nauc_mrr_at_5_std + value: 8.421303183153373 + - type: nauc_ndcg_at_1000_diff1 + value: 46.56384323294792 + - type: nauc_ndcg_at_1000_max + value: 45.79414140766788 + - type: nauc_ndcg_at_1000_std + value: 11.328687674007927 + - type: nauc_ndcg_at_100_diff1 + value: 46.401267258605785 + - type: nauc_ndcg_at_100_max + value: 45.35211233370126 + - type: nauc_ndcg_at_100_std + value: 11.660046947749429 + - type: nauc_ndcg_at_10_diff1 + value: 46.44173538470958 + - type: nauc_ndcg_at_10_max + value: 43.52231365563246 + - type: nauc_ndcg_at_10_std + value: 8.611338107905029 + - type: nauc_ndcg_at_1_diff1 + value: 54.52754433723723 + - type: nauc_ndcg_at_1_max + value: 48.35064288413123 + - type: nauc_ndcg_at_1_std + value: 5.97988808735259 + - type: nauc_ndcg_at_20_diff1 + value: 45.890961691776475 + - type: nauc_ndcg_at_20_max + value: 44.449459964630066 + - type: nauc_ndcg_at_20_std + value: 10.29521394739863 + - type: nauc_ndcg_at_3_diff1 + value: 48.01248081097938 + - type: nauc_ndcg_at_3_max + value: 44.33083632061056 + - type: nauc_ndcg_at_3_std + value: 6.2042365450405 + - type: nauc_ndcg_at_5_diff1 + value: 46.946751432979454 + - type: nauc_ndcg_at_5_max + value: 43.72026147379341 + - type: nauc_ndcg_at_5_std + value: 7.4639971143354185 + - type: nauc_precision_at_1000_diff1 + value: -1.9063726967785282 + - type: nauc_precision_at_1000_max + value: 20.624966922733133 + - type: nauc_precision_at_1000_std + value: 20.837587751658624 + - type: nauc_precision_at_100_diff1 + value: 11.908520999566871 + - type: nauc_precision_at_100_max + value: 32.421206826758365 + - type: nauc_precision_at_100_std + value: 29.92867366592684 + - type: nauc_precision_at_10_diff1 + value: 24.820171374698837 + - type: nauc_precision_at_10_max + value: 37.75937998223015 + - type: nauc_precision_at_10_std + value: 22.04338573970698 + - type: nauc_precision_at_1_diff1 + value: 54.52754433723723 + - type: nauc_precision_at_1_max + value: 48.35064288413123 + - type: nauc_precision_at_1_std + value: 5.97988808735259 + - type: nauc_precision_at_20_diff1 + value: 18.91530923896028 + - type: nauc_precision_at_20_max + value: 36.931514025254465 + - type: nauc_precision_at_20_std + value: 26.514626995206182 + - type: nauc_precision_at_3_diff1 + value: 36.55775057431161 + - type: nauc_precision_at_3_max + value: 43.24238763210348 + - type: nauc_precision_at_3_std + value: 14.190011413410447 + - type: nauc_precision_at_5_diff1 + value: 30.52018542983859 + - type: nauc_precision_at_5_max + value: 40.879480782186626 + - type: nauc_precision_at_5_std + value: 19.36115232020444 + - type: nauc_recall_at_1000_diff1 + value: 29.043901865986577 + - type: nauc_recall_at_1000_max + value: 52.47078655709399 + - type: nauc_recall_at_1000_std + value: 42.185290335696344 + - type: nauc_recall_at_100_diff1 + value: 35.15203120056371 + - type: nauc_recall_at_100_max + value: 44.86588046642376 + - type: nauc_recall_at_100_std + value: 30.684090987631656 + - type: nauc_recall_at_10_diff1 + value: 38.422479702756426 + - type: nauc_recall_at_10_max + value: 39.13217287228193 + - type: nauc_recall_at_10_std + value: 13.293751784394262 + - type: nauc_recall_at_1_diff1 + value: 56.40670369173174 + - type: nauc_recall_at_1_max + value: 44.28894153655779 + - type: nauc_recall_at_1_std + value: 0.7031741175020302 + - type: nauc_recall_at_20_diff1 + value: 35.27010714537033 + - type: nauc_recall_at_20_max + value: 41.661315656797484 + - type: nauc_recall_at_20_std + value: 19.76265026884951 + - type: nauc_recall_at_3_diff1 + value: 44.102359606408534 + - type: nauc_recall_at_3_max + value: 41.801680765603436 + - type: nauc_recall_at_3_std + value: 6.654901262827276 + - type: nauc_recall_at_5_diff1 + value: 40.859707573904366 + - type: nauc_recall_at_5_max + value: 40.50071358525545 + - type: nauc_recall_at_5_std + value: 10.01818369467164 + - type: ndcg_at_1 + value: 32.208999999999996 + - type: ndcg_at_10 + value: 40.095 + - type: ndcg_at_100 + value: 44.671 + - type: ndcg_at_1000 + value: 47.059 + - type: ndcg_at_20 + value: 41.876000000000005 + - type: ndcg_at_3 + value: 36.014 + - type: ndcg_at_5 + value: 37.922 + - type: precision_at_1 + value: 32.208999999999996 + - type: precision_at_10 + value: 6.135 + - type: precision_at_100 + value: 0.9159999999999999 + - type: precision_at_1000 + value: 0.121 + - type: precision_at_20 + value: 3.551 + - type: precision_at_3 + value: 15.235000000000001 + - type: precision_at_5 + value: 10.491 + - type: recall_at_1 + value: 28.631 + - type: recall_at_10 + value: 50.20400000000001 + - type: recall_at_100 + value: 71.073 + - type: recall_at_1000 + value: 88.31700000000001 + - type: recall_at_20 + value: 56.916999999999994 + - type: recall_at_3 + value: 38.653999999999996 + - type: recall_at_5 + value: 43.468 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackTexRetrieval (default) + revision: 46989137a86843e03a6195de44b09deda022eec7 + split: test + type: mteb/cqadupstack-tex + metrics: + - type: main_score + value: 34.747 + - type: map_at_1 + value: 21.834999999999997 + - type: map_at_10 + value: 29.935000000000002 + - type: map_at_100 + value: 31.133 + - type: map_at_1000 + value: 31.258999999999997 + - type: map_at_20 + value: 30.587999999999997 + - type: map_at_3 + value: 27.485 + - type: map_at_5 + value: 28.866999999999997 + - type: mrr_at_1 + value: 26.221610461114935 + - type: mrr_at_10 + value: 33.87804651569282 + - type: mrr_at_100 + value: 34.83678998242746 + - type: mrr_at_1000 + value: 34.905521389722146 + - type: mrr_at_20 + value: 34.42328285743433 + - type: mrr_at_3 + value: 31.710254645560966 + - type: mrr_at_5 + value: 32.91293874741923 + - type: nauc_map_at_1000_diff1 + value: 43.02122284102053 + - type: nauc_map_at_1000_max + value: 36.68422595257989 + - type: nauc_map_at_1000_std + value: 4.397348184630558 + - type: nauc_map_at_100_diff1 + value: 42.98532909587403 + - type: nauc_map_at_100_max + value: 36.67028447941515 + - type: nauc_map_at_100_std + value: 4.379139628887621 + - type: nauc_map_at_10_diff1 + value: 43.03544541360761 + - type: nauc_map_at_10_max + value: 36.41955934296037 + - type: nauc_map_at_10_std + value: 3.7935852850401623 + - type: nauc_map_at_1_diff1 + value: 49.69299904195074 + - type: nauc_map_at_1_max + value: 35.3185664028284 + - type: nauc_map_at_1_std + value: 1.4688246990256346 + - type: nauc_map_at_20_diff1 + value: 43.0116365499819 + - type: nauc_map_at_20_max + value: 36.612321960213485 + - type: nauc_map_at_20_std + value: 4.117491300921353 + - type: nauc_map_at_3_diff1 + value: 44.62079952930992 + - type: nauc_map_at_3_max + value: 36.54343315029047 + - type: nauc_map_at_3_std + value: 2.962991466072102 + - type: nauc_map_at_5_diff1 + value: 43.473302378849475 + - type: nauc_map_at_5_max + value: 36.192440219483565 + - type: nauc_map_at_5_std + value: 3.3382912941576794 + - type: nauc_mrr_at_1000_diff1 + value: 43.783512503833826 + - type: nauc_mrr_at_1000_max + value: 37.98147948760256 + - type: nauc_mrr_at_1000_std + value: 4.5030855590288255 + - type: nauc_mrr_at_100_diff1 + value: 43.76661944168357 + - type: nauc_mrr_at_100_max + value: 37.97066602825651 + - type: nauc_mrr_at_100_std + value: 4.502489421472255 + - type: nauc_mrr_at_10_diff1 + value: 43.74177797071971 + - type: nauc_mrr_at_10_max + value: 37.85024699162985 + - type: nauc_mrr_at_10_std + value: 4.0745169993200605 + - type: nauc_mrr_at_1_diff1 + value: 49.89628065603443 + - type: nauc_mrr_at_1_max + value: 37.996939557776095 + - type: nauc_mrr_at_1_std + value: 2.153267590949631 + - type: nauc_mrr_at_20_diff1 + value: 43.72637993670338 + - type: nauc_mrr_at_20_max + value: 37.93357612838035 + - type: nauc_mrr_at_20_std + value: 4.365878583210505 + - type: nauc_mrr_at_3_diff1 + value: 45.17454171018137 + - type: nauc_mrr_at_3_max + value: 38.356905779210166 + - type: nauc_mrr_at_3_std + value: 3.379623170680107 + - type: nauc_mrr_at_5_diff1 + value: 44.19688247145983 + - type: nauc_mrr_at_5_max + value: 37.910156470320466 + - type: nauc_mrr_at_5_std + value: 3.578601864819898 + - type: nauc_ndcg_at_1000_diff1 + value: 40.76776377226417 + - type: nauc_ndcg_at_1000_max + value: 37.38209218426787 + - type: nauc_ndcg_at_1000_std + value: 7.562606053871329 + - type: nauc_ndcg_at_100_diff1 + value: 40.118002901267545 + - type: nauc_ndcg_at_100_max + value: 37.08747189664714 + - type: nauc_ndcg_at_100_std + value: 7.5960070954791705 + - type: nauc_ndcg_at_10_diff1 + value: 40.50559704908023 + - type: nauc_ndcg_at_10_max + value: 36.44654941397671 + - type: nauc_ndcg_at_10_std + value: 4.965127544282767 + - type: nauc_ndcg_at_1_diff1 + value: 49.89628065603443 + - type: nauc_ndcg_at_1_max + value: 37.996939557776095 + - type: nauc_ndcg_at_1_std + value: 2.153267590949631 + - type: nauc_ndcg_at_20_diff1 + value: 40.39254920165008 + - type: nauc_ndcg_at_20_max + value: 36.916854469693575 + - type: nauc_ndcg_at_20_std + value: 6.086494699104603 + - type: nauc_ndcg_at_3_diff1 + value: 43.36359122099107 + - type: nauc_ndcg_at_3_max + value: 37.046121931570916 + - type: nauc_ndcg_at_3_std + value: 3.1574964976166333 + - type: nauc_ndcg_at_5_diff1 + value: 41.5582652711063 + - type: nauc_ndcg_at_5_max + value: 36.18558444731444 + - type: nauc_ndcg_at_5_std + value: 3.832396124704952 + - type: nauc_precision_at_1000_diff1 + value: 2.2864644386233586 + - type: nauc_precision_at_1000_max + value: 7.20908177927649 + - type: nauc_precision_at_1000_std + value: 5.555003810450184 + - type: nauc_precision_at_100_diff1 + value: 9.862474724647964 + - type: nauc_precision_at_100_max + value: 18.32968696035674 + - type: nauc_precision_at_100_std + value: 11.623605222066711 + - type: nauc_precision_at_10_diff1 + value: 22.463771538601492 + - type: nauc_precision_at_10_max + value: 30.978133789439376 + - type: nauc_precision_at_10_std + value: 8.286121544404391 + - type: nauc_precision_at_1_diff1 + value: 49.89628065603443 + - type: nauc_precision_at_1_max + value: 37.996939557776095 + - type: nauc_precision_at_1_std + value: 2.153267590949631 + - type: nauc_precision_at_20_diff1 + value: 18.220595875083966 + - type: nauc_precision_at_20_max + value: 28.346921854722034 + - type: nauc_precision_at_20_std + value: 10.273810100275899 + - type: nauc_precision_at_3_diff1 + value: 35.012302714762285 + - type: nauc_precision_at_3_max + value: 36.28875088265941 + - type: nauc_precision_at_3_std + value: 4.005105982482449 + - type: nauc_precision_at_5_diff1 + value: 28.396470823961412 + - type: nauc_precision_at_5_max + value: 32.737997207189615 + - type: nauc_precision_at_5_std + value: 4.784459732005448 + - type: nauc_recall_at_1000_diff1 + value: 20.327711283956255 + - type: nauc_recall_at_1000_max + value: 37.0704878438467 + - type: nauc_recall_at_1000_std + value: 35.75674454719592 + - type: nauc_recall_at_100_diff1 + value: 24.36552307372351 + - type: nauc_recall_at_100_max + value: 33.539816313523005 + - type: nauc_recall_at_100_std + value: 21.32369323408807 + - type: nauc_recall_at_10_diff1 + value: 30.955033013118697 + - type: nauc_recall_at_10_max + value: 33.015068845269205 + - type: nauc_recall_at_10_std + value: 7.6648954096823765 + - type: nauc_recall_at_1_diff1 + value: 49.69299904195074 + - type: nauc_recall_at_1_max + value: 35.3185664028284 + - type: nauc_recall_at_1_std + value: 1.4688246990256346 + - type: nauc_recall_at_20_diff1 + value: 29.543554170900038 + - type: nauc_recall_at_20_max + value: 33.8343255326902 + - type: nauc_recall_at_20_std + value: 11.674093317168884 + - type: nauc_recall_at_3_diff1 + value: 38.78889409839137 + - type: nauc_recall_at_3_max + value: 35.267542834931746 + - type: nauc_recall_at_3_std + value: 4.155285803806091 + - type: nauc_recall_at_5_diff1 + value: 34.5062081359784 + - type: nauc_recall_at_5_max + value: 33.06934231809503 + - type: nauc_recall_at_5_std + value: 5.308553270025992 + - type: ndcg_at_1 + value: 26.222 + - type: ndcg_at_10 + value: 34.747 + - type: ndcg_at_100 + value: 40.227000000000004 + - type: ndcg_at_1000 + value: 42.957 + - type: ndcg_at_20 + value: 36.84 + - type: ndcg_at_3 + value: 30.546 + - type: ndcg_at_5 + value: 32.54 + - type: precision_at_1 + value: 26.222 + - type: precision_at_10 + value: 6.125 + - type: precision_at_100 + value: 1.042 + - type: precision_at_1000 + value: 0.145 + - type: precision_at_20 + value: 3.689 + - type: precision_at_3 + value: 14.304 + - type: precision_at_5 + value: 10.165000000000001 + - type: recall_at_1 + value: 21.834999999999997 + - type: recall_at_10 + value: 45.086 + - type: recall_at_100 + value: 69.42399999999999 + - type: recall_at_1000 + value: 88.637 + - type: recall_at_20 + value: 52.813 + - type: recall_at_3 + value: 33.339 + - type: recall_at_5 + value: 38.568999999999996 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackUnixRetrieval (default) + revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53 + split: test + type: mteb/cqadupstack-unix + metrics: + - type: main_score + value: 46.217000000000006 + - type: map_at_1 + value: 30.694 + - type: map_at_10 + value: 40.663 + - type: map_at_100 + value: 41.92 + - type: map_at_1000 + value: 42.016 + - type: map_at_20 + value: 41.453 + - type: map_at_3 + value: 37.586999999999996 + - type: map_at_5 + value: 39.206 + - type: mrr_at_1 + value: 35.634328358208954 + - type: mrr_at_10 + value: 44.59355010660978 + - type: mrr_at_100 + value: 45.45809181295436 + - type: mrr_at_1000 + value: 45.511480349861 + - type: mrr_at_20 + value: 45.14851149688157 + - type: mrr_at_3 + value: 42.10199004975122 + - type: mrr_at_5 + value: 43.4546019900497 + - type: nauc_map_at_1000_diff1 + value: 54.49157296835768 + - type: nauc_map_at_1000_max + value: 44.571646063255734 + - type: nauc_map_at_1000_std + value: 5.88489486258929 + - type: nauc_map_at_100_diff1 + value: 54.4750886109242 + - type: nauc_map_at_100_max + value: 44.58745640948606 + - type: nauc_map_at_100_std + value: 5.921654270108923 + - type: nauc_map_at_10_diff1 + value: 54.77323759820101 + - type: nauc_map_at_10_max + value: 44.5308708337522 + - type: nauc_map_at_10_std + value: 5.556181862868676 + - type: nauc_map_at_1_diff1 + value: 61.199521619603324 + - type: nauc_map_at_1_max + value: 43.35152212139949 + - type: nauc_map_at_1_std + value: 1.8250136220533804 + - type: nauc_map_at_20_diff1 + value: 54.53992127623092 + - type: nauc_map_at_20_max + value: 44.562426294305766 + - type: nauc_map_at_20_std + value: 5.883167785481589 + - type: nauc_map_at_3_diff1 + value: 55.849077974593264 + - type: nauc_map_at_3_max + value: 43.63537595305222 + - type: nauc_map_at_3_std + value: 3.955405001774955 + - type: nauc_map_at_5_diff1 + value: 54.875360007892915 + - type: nauc_map_at_5_max + value: 44.122844145457314 + - type: nauc_map_at_5_std + value: 5.292241654718073 + - type: nauc_mrr_at_1000_diff1 + value: 53.79026001696958 + - type: nauc_mrr_at_1000_max + value: 45.003335504255716 + - type: nauc_mrr_at_1000_std + value: 5.7193959704884625 + - type: nauc_mrr_at_100_diff1 + value: 53.77099382934476 + - type: nauc_mrr_at_100_max + value: 45.001137520052474 + - type: nauc_mrr_at_100_std + value: 5.744675154739931 + - type: nauc_mrr_at_10_diff1 + value: 53.67068659921886 + - type: nauc_mrr_at_10_max + value: 45.031787546530865 + - type: nauc_mrr_at_10_std + value: 5.552485217606094 + - type: nauc_mrr_at_1_diff1 + value: 60.11868299569921 + - type: nauc_mrr_at_1_max + value: 44.43993196821902 + - type: nauc_mrr_at_1_std + value: 1.6193064176014789 + - type: nauc_mrr_at_20_diff1 + value: 53.7063379529791 + - type: nauc_mrr_at_20_max + value: 45.026408969690365 + - type: nauc_mrr_at_20_std + value: 5.784653137718281 + - type: nauc_mrr_at_3_diff1 + value: 54.32684070276712 + - type: nauc_mrr_at_3_max + value: 44.907538081980206 + - type: nauc_mrr_at_3_std + value: 4.445174178524578 + - type: nauc_mrr_at_5_diff1 + value: 53.569381129347974 + - type: nauc_mrr_at_5_max + value: 44.78946405058036 + - type: nauc_mrr_at_5_std + value: 5.256110400574076 + - type: nauc_ndcg_at_1000_diff1 + value: 52.19534836509613 + - type: nauc_ndcg_at_1000_max + value: 44.97967472797531 + - type: nauc_ndcg_at_1000_std + value: 8.054384323530511 + - type: nauc_ndcg_at_100_diff1 + value: 51.71703032379199 + - type: nauc_ndcg_at_100_max + value: 45.165486288760206 + - type: nauc_ndcg_at_100_std + value: 8.935210771224625 + - type: nauc_ndcg_at_10_diff1 + value: 52.2813212134141 + - type: nauc_ndcg_at_10_max + value: 45.09360720038123 + - type: nauc_ndcg_at_10_std + value: 7.333462651609881 + - type: nauc_ndcg_at_1_diff1 + value: 60.11868299569921 + - type: nauc_ndcg_at_1_max + value: 44.43993196821902 + - type: nauc_ndcg_at_1_std + value: 1.6193064176014789 + - type: nauc_ndcg_at_20_diff1 + value: 51.79260120012986 + - type: nauc_ndcg_at_20_max + value: 45.11310286101053 + - type: nauc_ndcg_at_20_std + value: 8.64544289254536 + - type: nauc_ndcg_at_3_diff1 + value: 53.48446499869877 + - type: nauc_ndcg_at_3_max + value: 44.065508335360235 + - type: nauc_ndcg_at_3_std + value: 4.72003163528987 + - type: nauc_ndcg_at_5_diff1 + value: 52.2665514246751 + - type: nauc_ndcg_at_5_max + value: 44.309202727356826 + - type: nauc_ndcg_at_5_std + value: 6.557954992070128 + - type: nauc_precision_at_1000_diff1 + value: -20.135608241014314 + - type: nauc_precision_at_1000_max + value: -8.072232641333796 + - type: nauc_precision_at_1000_std + value: -0.9854111061728026 + - type: nauc_precision_at_100_diff1 + value: -6.675936748503976 + - type: nauc_precision_at_100_max + value: 9.458096538337845 + - type: nauc_precision_at_100_std + value: 9.669047880836656 + - type: nauc_precision_at_10_diff1 + value: 18.99731503204749 + - type: nauc_precision_at_10_max + value: 30.060390720696006 + - type: nauc_precision_at_10_std + value: 8.947603217350805 + - type: nauc_precision_at_1_diff1 + value: 60.11868299569921 + - type: nauc_precision_at_1_max + value: 44.43993196821902 + - type: nauc_precision_at_1_std + value: 1.6193064176014789 + - type: nauc_precision_at_20_diff1 + value: 8.492991282537064 + - type: nauc_precision_at_20_max + value: 22.547742822445805 + - type: nauc_precision_at_20_std + value: 11.66851645034729 + - type: nauc_precision_at_3_diff1 + value: 37.096049548744084 + - type: nauc_precision_at_3_max + value: 38.32665656060843 + - type: nauc_precision_at_3_std + value: 6.056505021027511 + - type: nauc_precision_at_5_diff1 + value: 28.026702227231453 + - type: nauc_precision_at_5_max + value: 35.52718369957977 + - type: nauc_precision_at_5_std + value: 9.579893460752707 + - type: nauc_recall_at_1000_diff1 + value: 36.522293248630646 + - type: nauc_recall_at_1000_max + value: 50.12987838816737 + - type: nauc_recall_at_1000_std + value: 41.87667128189119 + - type: nauc_recall_at_100_diff1 + value: 38.28689929050365 + - type: nauc_recall_at_100_max + value: 45.12984308724397 + - type: nauc_recall_at_100_std + value: 27.651176460124955 + - type: nauc_recall_at_10_diff1 + value: 43.915333865635795 + - type: nauc_recall_at_10_max + value: 44.12102590410687 + - type: nauc_recall_at_10_std + value: 12.929151897559302 + - type: nauc_recall_at_1_diff1 + value: 61.199521619603324 + - type: nauc_recall_at_1_max + value: 43.35152212139949 + - type: nauc_recall_at_1_std + value: 1.8250136220533804 + - type: nauc_recall_at_20_diff1 + value: 41.1919417904227 + - type: nauc_recall_at_20_max + value: 43.93891446263423 + - type: nauc_recall_at_20_std + value: 19.360962694822103 + - type: nauc_recall_at_3_diff1 + value: 49.012044588636314 + - type: nauc_recall_at_3_max + value: 42.36403255197195 + - type: nauc_recall_at_3_std + value: 5.910392359087258 + - type: nauc_recall_at_5_diff1 + value: 45.01684121003052 + - type: nauc_recall_at_5_max + value: 42.50353450375004 + - type: nauc_recall_at_5_std + value: 10.20686161068941 + - type: ndcg_at_1 + value: 35.634 + - type: ndcg_at_10 + value: 46.217000000000006 + - type: ndcg_at_100 + value: 51.382000000000005 + - type: ndcg_at_1000 + value: 53.398 + - type: ndcg_at_20 + value: 48.601 + - type: ndcg_at_3 + value: 40.885 + - type: ndcg_at_5 + value: 43.21 + - type: precision_at_1 + value: 35.634 + - type: precision_at_10 + value: 7.733 + - type: precision_at_100 + value: 1.1520000000000001 + - type: precision_at_1000 + value: 0.14200000000000002 + - type: precision_at_20 + value: 4.557 + - type: precision_at_3 + value: 18.407999999999998 + - type: precision_at_5 + value: 12.799 + - type: recall_at_1 + value: 30.694 + - type: recall_at_10 + value: 59.016000000000005 + - type: recall_at_100 + value: 80.55799999999999 + - type: recall_at_1000 + value: 94.396 + - type: recall_at_20 + value: 67.448 + - type: recall_at_3 + value: 44.47 + - type: recall_at_5 + value: 50.422999999999995 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackWebmastersRetrieval (default) + revision: 160c094312a0e1facb97e55eeddb698c0abe3571 + split: test + type: mteb/cqadupstack-webmasters + metrics: + - type: main_score + value: 45.617000000000004 + - type: map_at_1 + value: 30.605 + - type: map_at_10 + value: 39.82 + - type: map_at_100 + value: 41.436 + - type: map_at_1000 + value: 41.650999999999996 + - type: map_at_20 + value: 40.632000000000005 + - type: map_at_3 + value: 36.773 + - type: map_at_5 + value: 38.262 + - type: mrr_at_1 + value: 36.56126482213439 + - type: mrr_at_10 + value: 44.396292113683415 + - type: mrr_at_100 + value: 45.355059573471564 + - type: mrr_at_1000 + value: 45.395592279144 + - type: mrr_at_20 + value: 44.95137796744523 + - type: mrr_at_3 + value: 41.96310935441372 + - type: mrr_at_5 + value: 43.18840579710146 + - type: nauc_map_at_1000_diff1 + value: 54.037905802630824 + - type: nauc_map_at_1000_max + value: 40.450308786251725 + - type: nauc_map_at_1000_std + value: 9.148698758674364 + - type: nauc_map_at_100_diff1 + value: 54.072292884161065 + - type: nauc_map_at_100_max + value: 40.645549885870764 + - type: nauc_map_at_100_std + value: 9.045471413369627 + - type: nauc_map_at_10_diff1 + value: 54.18871972357613 + - type: nauc_map_at_10_max + value: 40.48047320299438 + - type: nauc_map_at_10_std + value: 7.759433230608552 + - type: nauc_map_at_1_diff1 + value: 60.252738170851295 + - type: nauc_map_at_1_max + value: 37.820479858935904 + - type: nauc_map_at_1_std + value: 4.358462724453915 + - type: nauc_map_at_20_diff1 + value: 53.992902705647595 + - type: nauc_map_at_20_max + value: 40.61396716993049 + - type: nauc_map_at_20_std + value: 8.367068955202162 + - type: nauc_map_at_3_diff1 + value: 55.02002806249898 + - type: nauc_map_at_3_max + value: 40.153735394036985 + - type: nauc_map_at_3_std + value: 6.356789989044052 + - type: nauc_map_at_5_diff1 + value: 54.39644719640333 + - type: nauc_map_at_5_max + value: 39.783628855308855 + - type: nauc_map_at_5_std + value: 6.884993267179124 + - type: nauc_mrr_at_1000_diff1 + value: 51.51322583010865 + - type: nauc_mrr_at_1000_max + value: 39.61505600996298 + - type: nauc_mrr_at_1000_std + value: 10.062810078262705 + - type: nauc_mrr_at_100_diff1 + value: 51.485129593394454 + - type: nauc_mrr_at_100_max + value: 39.61068326780849 + - type: nauc_mrr_at_100_std + value: 10.104664788214674 + - type: nauc_mrr_at_10_diff1 + value: 51.453327413203944 + - type: nauc_mrr_at_10_max + value: 39.59859304483541 + - type: nauc_mrr_at_10_std + value: 9.806291191245174 + - type: nauc_mrr_at_1_diff1 + value: 56.46194397143132 + - type: nauc_mrr_at_1_max + value: 38.46984315267595 + - type: nauc_mrr_at_1_std + value: 7.901227094621062 + - type: nauc_mrr_at_20_diff1 + value: 51.39284512555125 + - type: nauc_mrr_at_20_max + value: 39.5141389589787 + - type: nauc_mrr_at_20_std + value: 9.955396483907231 + - type: nauc_mrr_at_3_diff1 + value: 51.92875388910452 + - type: nauc_mrr_at_3_max + value: 39.61356707024846 + - type: nauc_mrr_at_3_std + value: 9.221894323860608 + - type: nauc_mrr_at_5_diff1 + value: 51.76533052697717 + - type: nauc_mrr_at_5_max + value: 39.37224340525692 + - type: nauc_mrr_at_5_std + value: 9.591107649683261 + - type: nauc_ndcg_at_1000_diff1 + value: 51.51985677496593 + - type: nauc_ndcg_at_1000_max + value: 41.205946764922885 + - type: nauc_ndcg_at_1000_std + value: 12.235715267234813 + - type: nauc_ndcg_at_100_diff1 + value: 50.976885819253184 + - type: nauc_ndcg_at_100_max + value: 41.3698483798701 + - type: nauc_ndcg_at_100_std + value: 12.926691259793136 + - type: nauc_ndcg_at_10_diff1 + value: 51.175922991306244 + - type: nauc_ndcg_at_10_max + value: 40.583412345898886 + - type: nauc_ndcg_at_10_std + value: 10.5105692131039 + - type: nauc_ndcg_at_1_diff1 + value: 56.46194397143132 + - type: nauc_ndcg_at_1_max + value: 38.46984315267595 + - type: nauc_ndcg_at_1_std + value: 7.901227094621062 + - type: nauc_ndcg_at_20_diff1 + value: 50.65555101985068 + - type: nauc_ndcg_at_20_max + value: 40.83254051658996 + - type: nauc_ndcg_at_20_std + value: 11.10240595140561 + - type: nauc_ndcg_at_3_diff1 + value: 51.64713670047041 + - type: nauc_ndcg_at_3_max + value: 40.150832427957 + - type: nauc_ndcg_at_3_std + value: 8.308961635349586 + - type: nauc_ndcg_at_5_diff1 + value: 51.67780215717258 + - type: nauc_ndcg_at_5_max + value: 39.73135207048929 + - type: nauc_ndcg_at_5_std + value: 9.362001359100065 + - type: nauc_precision_at_1000_diff1 + value: -18.35413896999055 + - type: nauc_precision_at_1000_max + value: -22.440813066322367 + - type: nauc_precision_at_1000_std + value: 13.713989618569943 + - type: nauc_precision_at_100_diff1 + value: -6.460027590526615 + - type: nauc_precision_at_100_max + value: -5.289552609878046 + - type: nauc_precision_at_100_std + value: 19.945140974151077 + - type: nauc_precision_at_10_diff1 + value: 13.811111882813027 + - type: nauc_precision_at_10_max + value: 22.083890093474498 + - type: nauc_precision_at_10_std + value: 19.864592346780555 + - type: nauc_precision_at_1_diff1 + value: 56.46194397143132 + - type: nauc_precision_at_1_max + value: 38.46984315267595 + - type: nauc_precision_at_1_std + value: 7.901227094621062 + - type: nauc_precision_at_20_diff1 + value: 3.615806105136385 + - type: nauc_precision_at_20_max + value: 14.545739153541733 + - type: nauc_precision_at_20_std + value: 21.59687523481294 + - type: nauc_precision_at_3_diff1 + value: 31.353226335214952 + - type: nauc_precision_at_3_max + value: 33.98522590517617 + - type: nauc_precision_at_3_std + value: 12.24165215255015 + - type: nauc_precision_at_5_diff1 + value: 23.921742105135273 + - type: nauc_precision_at_5_max + value: 28.33578039622996 + - type: nauc_precision_at_5_std + value: 14.595218126936347 + - type: nauc_recall_at_1000_diff1 + value: 37.072169332454884 + - type: nauc_recall_at_1000_max + value: 65.74181547600575 + - type: nauc_recall_at_1000_std + value: 63.879841544531715 + - type: nauc_recall_at_100_diff1 + value: 32.41938614666549 + - type: nauc_recall_at_100_max + value: 42.19385838299294 + - type: nauc_recall_at_100_std + value: 36.70934004379387 + - type: nauc_recall_at_10_diff1 + value: 43.41881951105109 + - type: nauc_recall_at_10_max + value: 40.77999928072556 + - type: nauc_recall_at_10_std + value: 13.08409072311216 + - type: nauc_recall_at_1_diff1 + value: 60.252738170851295 + - type: nauc_recall_at_1_max + value: 37.820479858935904 + - type: nauc_recall_at_1_std + value: 4.358462724453915 + - type: nauc_recall_at_20_diff1 + value: 40.466068680939564 + - type: nauc_recall_at_20_max + value: 40.456898157268476 + - type: nauc_recall_at_20_std + value: 16.802947038702285 + - type: nauc_recall_at_3_diff1 + value: 48.90985904873113 + - type: nauc_recall_at_3_max + value: 40.355251875045134 + - type: nauc_recall_at_3_std + value: 7.6831281622651195 + - type: nauc_recall_at_5_diff1 + value: 46.52974428597555 + - type: nauc_recall_at_5_max + value: 38.686366219208566 + - type: nauc_recall_at_5_std + value: 9.746838299246388 + - type: ndcg_at_1 + value: 36.561 + - type: ndcg_at_10 + value: 45.617000000000004 + - type: ndcg_at_100 + value: 51.498 + - type: ndcg_at_1000 + value: 53.55 + - type: ndcg_at_20 + value: 47.69 + - type: ndcg_at_3 + value: 40.959 + - type: ndcg_at_5 + value: 42.713 + - type: precision_at_1 + value: 36.561 + - type: precision_at_10 + value: 8.518 + - type: precision_at_100 + value: 1.6230000000000002 + - type: precision_at_1000 + value: 0.248 + - type: precision_at_20 + value: 5.296 + - type: precision_at_3 + value: 18.972 + - type: precision_at_5 + value: 13.281 + - type: recall_at_1 + value: 30.605 + - type: recall_at_10 + value: 56.507 + - type: recall_at_100 + value: 83.021 + - type: recall_at_1000 + value: 95.15400000000001 + - type: recall_at_20 + value: 64.211 + - type: recall_at_3 + value: 42.905 + - type: recall_at_5 + value: 48.222 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackWordpressRetrieval (default) + revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4 + split: test + type: mteb/cqadupstack-wordpress + metrics: + - type: main_score + value: 38.645 + - type: map_at_1 + value: 25.678 + - type: map_at_10 + value: 33.723 + - type: map_at_100 + value: 34.709 + - type: map_at_1000 + value: 34.807 + - type: map_at_20 + value: 34.239000000000004 + - type: map_at_3 + value: 31.038 + - type: map_at_5 + value: 32.412 + - type: mrr_at_1 + value: 27.911275415896487 + - type: mrr_at_10 + value: 35.97790687439485 + - type: mrr_at_100 + value: 36.780172693739885 + - type: mrr_at_1000 + value: 36.853206678752635 + - type: mrr_at_20 + value: 36.39926435386768 + - type: mrr_at_3 + value: 33.54898336414048 + - type: mrr_at_5 + value: 34.741219963031426 + - type: nauc_map_at_1000_diff1 + value: 50.70376514935502 + - type: nauc_map_at_1000_max + value: 34.715574916274015 + - type: nauc_map_at_1000_std + value: 2.857145703771197 + - type: nauc_map_at_100_diff1 + value: 50.68803386146576 + - type: nauc_map_at_100_max + value: 34.72480905664743 + - type: nauc_map_at_100_std + value: 2.874097116107013 + - type: nauc_map_at_10_diff1 + value: 50.85144940330611 + - type: nauc_map_at_10_max + value: 34.86971741140245 + - type: nauc_map_at_10_std + value: 2.3960927913664456 + - type: nauc_map_at_1_diff1 + value: 58.39635634775997 + - type: nauc_map_at_1_max + value: 33.21539901517558 + - type: nauc_map_at_1_std + value: -1.3643570885414436 + - type: nauc_map_at_20_diff1 + value: 50.6576547451562 + - type: nauc_map_at_20_max + value: 34.64296947893069 + - type: nauc_map_at_20_std + value: 2.5872198719498267 + - type: nauc_map_at_3_diff1 + value: 52.034597942323224 + - type: nauc_map_at_3_max + value: 34.76014051294698 + - type: nauc_map_at_3_std + value: 1.6016973476618055 + - type: nauc_map_at_5_diff1 + value: 50.91185539428835 + - type: nauc_map_at_5_max + value: 34.89661763697396 + - type: nauc_map_at_5_std + value: 1.9166919513100547 + - type: nauc_mrr_at_1000_diff1 + value: 49.44709709546153 + - type: nauc_mrr_at_1000_max + value: 34.76818294423692 + - type: nauc_mrr_at_1000_std + value: 3.784298299856579 + - type: nauc_mrr_at_100_diff1 + value: 49.414102627661094 + - type: nauc_mrr_at_100_max + value: 34.78222989030245 + - type: nauc_mrr_at_100_std + value: 3.81206420835598 + - type: nauc_mrr_at_10_diff1 + value: 49.46764370594936 + - type: nauc_mrr_at_10_max + value: 34.8973046542979 + - type: nauc_mrr_at_10_std + value: 3.4143729715445743 + - type: nauc_mrr_at_1_diff1 + value: 56.94689727516319 + - type: nauc_mrr_at_1_max + value: 33.63815035299124 + - type: nauc_mrr_at_1_std + value: -0.3238386146577392 + - type: nauc_mrr_at_20_diff1 + value: 49.38284176502219 + - type: nauc_mrr_at_20_max + value: 34.779770996479414 + - type: nauc_mrr_at_20_std + value: 3.6393928012551777 + - type: nauc_mrr_at_3_diff1 + value: 50.413534933718076 + - type: nauc_mrr_at_3_max + value: 35.2355086746895 + - type: nauc_mrr_at_3_std + value: 3.2874514415001355 + - type: nauc_mrr_at_5_diff1 + value: 49.54229976377922 + - type: nauc_mrr_at_5_max + value: 35.2158327592191 + - type: nauc_mrr_at_5_std + value: 3.1725476512174158 + - type: nauc_ndcg_at_1000_diff1 + value: 48.120538617666114 + - type: nauc_ndcg_at_1000_max + value: 35.04705009841914 + - type: nauc_ndcg_at_1000_std + value: 6.4351693550919835 + - type: nauc_ndcg_at_100_diff1 + value: 47.34364138839563 + - type: nauc_ndcg_at_100_max + value: 34.66685645306898 + - type: nauc_ndcg_at_100_std + value: 6.878861943573172 + - type: nauc_ndcg_at_10_diff1 + value: 47.71681977063441 + - type: nauc_ndcg_at_10_max + value: 34.7357965852759 + - type: nauc_ndcg_at_10_std + value: 4.221940362317388 + - type: nauc_ndcg_at_1_diff1 + value: 56.94689727516319 + - type: nauc_ndcg_at_1_max + value: 33.63815035299124 + - type: nauc_ndcg_at_1_std + value: -0.3238386146577392 + - type: nauc_ndcg_at_20_diff1 + value: 47.143175116342164 + - type: nauc_ndcg_at_20_max + value: 34.060564866094104 + - type: nauc_ndcg_at_20_std + value: 4.825115325641695 + - type: nauc_ndcg_at_3_diff1 + value: 49.344605024642604 + - type: nauc_ndcg_at_3_max + value: 35.130225181455025 + - type: nauc_ndcg_at_3_std + value: 3.2358596964463406 + - type: nauc_ndcg_at_5_diff1 + value: 47.78161337000044 + - type: nauc_ndcg_at_5_max + value: 35.21427173972542 + - type: nauc_ndcg_at_5_std + value: 3.3316671059413565 + - type: nauc_precision_at_1000_diff1 + value: -12.185229718736519 + - type: nauc_precision_at_1000_max + value: -5.143952938261702 + - type: nauc_precision_at_1000_std + value: 9.973950056060861 + - type: nauc_precision_at_100_diff1 + value: 9.58352444091056 + - type: nauc_precision_at_100_max + value: 21.382425938743122 + - type: nauc_precision_at_100_std + value: 24.16287723446871 + - type: nauc_precision_at_10_diff1 + value: 28.776525902554834 + - type: nauc_precision_at_10_max + value: 33.911171529963454 + - type: nauc_precision_at_10_std + value: 14.82005619941979 + - type: nauc_precision_at_1_diff1 + value: 56.94689727516319 + - type: nauc_precision_at_1_max + value: 33.63815035299124 + - type: nauc_precision_at_1_std + value: -0.3238386146577392 + - type: nauc_precision_at_20_diff1 + value: 23.158315288397596 + - type: nauc_precision_at_20_max + value: 27.80569911941973 + - type: nauc_precision_at_20_std + value: 16.04487961632009 + - type: nauc_precision_at_3_diff1 + value: 39.62535812885354 + - type: nauc_precision_at_3_max + value: 36.51213228287965 + - type: nauc_precision_at_3_std + value: 8.116280123866353 + - type: nauc_precision_at_5_diff1 + value: 31.712654781612365 + - type: nauc_precision_at_5_max + value: 35.816857659237215 + - type: nauc_precision_at_5_std + value: 9.268216959058009 + - type: nauc_recall_at_1000_diff1 + value: 38.391247252984925 + - type: nauc_recall_at_1000_max + value: 47.15753182456363 + - type: nauc_recall_at_1000_std + value: 48.58094810349862 + - type: nauc_recall_at_100_diff1 + value: 33.146385303867035 + - type: nauc_recall_at_100_max + value: 31.23397224541019 + - type: nauc_recall_at_100_std + value: 25.9640624687697 + - type: nauc_recall_at_10_diff1 + value: 38.48401335354199 + - type: nauc_recall_at_10_max + value: 32.224451286507225 + - type: nauc_recall_at_10_std + value: 8.123596700542349 + - type: nauc_recall_at_1_diff1 + value: 58.39635634775997 + - type: nauc_recall_at_1_max + value: 33.21539901517558 + - type: nauc_recall_at_1_std + value: -1.3643570885414436 + - type: nauc_recall_at_20_diff1 + value: 36.02226998004879 + - type: nauc_recall_at_20_max + value: 29.617461591260724 + - type: nauc_recall_at_20_std + value: 10.141289841509085 + - type: nauc_recall_at_3_diff1 + value: 43.8069577952235 + - type: nauc_recall_at_3_max + value: 34.79904316427403 + - type: nauc_recall_at_3_std + value: 5.155948130298837 + - type: nauc_recall_at_5_diff1 + value: 39.42033328271163 + - type: nauc_recall_at_5_max + value: 34.42928238899675 + - type: nauc_recall_at_5_std + value: 5.632638942940883 + - type: ndcg_at_1 + value: 27.911 + - type: ndcg_at_10 + value: 38.645 + - type: ndcg_at_100 + value: 43.478 + - type: ndcg_at_1000 + value: 45.895 + - type: ndcg_at_20 + value: 40.253 + - type: ndcg_at_3 + value: 33.521 + - type: ndcg_at_5 + value: 35.711999999999996 + - type: precision_at_1 + value: 27.911 + - type: precision_at_10 + value: 5.989 + - type: precision_at_100 + value: 0.915 + - type: precision_at_1000 + value: 0.123 + - type: precision_at_20 + value: 3.392 + - type: precision_at_3 + value: 14.11 + - type: precision_at_5 + value: 9.871 + - type: recall_at_1 + value: 25.678 + - type: recall_at_10 + value: 51.617000000000004 + - type: recall_at_100 + value: 74.117 + - type: recall_at_1000 + value: 92.06800000000001 + - type: recall_at_20 + value: 57.49999999999999 + - type: recall_at_3 + value: 37.705 + - type: recall_at_5 + value: 42.985 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ClimateFEVER (default) + revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380 + split: test + type: mteb/climate-fever + metrics: + - type: main_score + value: 39.893 + - type: map_at_1 + value: 17.015 + - type: map_at_10 + value: 29.774 + - type: map_at_100 + value: 31.85 + - type: map_at_1000 + value: 32.04 + - type: map_at_20 + value: 30.909 + - type: map_at_3 + value: 24.79 + - type: map_at_5 + value: 27.503 + - type: mrr_at_1 + value: 38.371335504886 + - type: mrr_at_10 + value: 51.698102476604134 + - type: mrr_at_100 + value: 52.37642053960552 + - type: mrr_at_1000 + value: 52.39929399165635 + - type: mrr_at_20 + value: 52.14169876214939 + - type: mrr_at_3 + value: 48.9467969598263 + - type: mrr_at_5 + value: 50.653637350705836 + - type: nauc_map_at_1000_diff1 + value: 27.944908093190573 + - type: nauc_map_at_1000_max + value: 36.8158438017512 + - type: nauc_map_at_1000_std + value: 21.049995457766265 + - type: nauc_map_at_100_diff1 + value: 27.89880319854821 + - type: nauc_map_at_100_max + value: 36.82081491300568 + - type: nauc_map_at_100_std + value: 21.033538531183574 + - type: nauc_map_at_10_diff1 + value: 28.04702868036437 + - type: nauc_map_at_10_max + value: 35.828396439615844 + - type: nauc_map_at_10_std + value: 19.386466847792313 + - type: nauc_map_at_1_diff1 + value: 35.990766700733666 + - type: nauc_map_at_1_max + value: 29.711074426265377 + - type: nauc_map_at_1_std + value: 8.80172791202737 + - type: nauc_map_at_20_diff1 + value: 27.778323786296617 + - type: nauc_map_at_20_max + value: 36.31833346940698 + - type: nauc_map_at_20_std + value: 20.291526594416357 + - type: nauc_map_at_3_diff1 + value: 28.94580764960388 + - type: nauc_map_at_3_max + value: 32.95647088190485 + - type: nauc_map_at_3_std + value: 14.111407635835146 + - type: nauc_map_at_5_diff1 + value: 27.945803526851424 + - type: nauc_map_at_5_max + value: 34.0185273762353 + - type: nauc_map_at_5_std + value: 16.215045945048686 + - type: nauc_mrr_at_1000_diff1 + value: 27.736625286063443 + - type: nauc_mrr_at_1000_max + value: 33.642204191814386 + - type: nauc_mrr_at_1000_std + value: 21.80704540877825 + - type: nauc_mrr_at_100_diff1 + value: 27.737166943875796 + - type: nauc_mrr_at_100_max + value: 33.664071472865594 + - type: nauc_mrr_at_100_std + value: 21.8379746015177 + - type: nauc_mrr_at_10_diff1 + value: 27.945163031104276 + - type: nauc_mrr_at_10_max + value: 33.859314923894594 + - type: nauc_mrr_at_10_std + value: 21.707287373363084 + - type: nauc_mrr_at_1_diff1 + value: 30.197548640710586 + - type: nauc_mrr_at_1_max + value: 28.699305487301448 + - type: nauc_mrr_at_1_std + value: 15.591542208606757 + - type: nauc_mrr_at_20_diff1 + value: 27.766720800081846 + - type: nauc_mrr_at_20_max + value: 33.720004243768756 + - type: nauc_mrr_at_20_std + value: 21.84490359079179 + - type: nauc_mrr_at_3_diff1 + value: 26.36316485516251 + - type: nauc_mrr_at_3_max + value: 32.162398379112986 + - type: nauc_mrr_at_3_std + value: 20.309889044335932 + - type: nauc_mrr_at_5_diff1 + value: 27.291015445042056 + - type: nauc_mrr_at_5_max + value: 33.40344599071943 + - type: nauc_mrr_at_5_std + value: 21.438249940391817 + - type: nauc_ndcg_at_1000_diff1 + value: 27.45379244203272 + - type: nauc_ndcg_at_1000_max + value: 40.090643062494216 + - type: nauc_ndcg_at_1000_std + value: 28.384199906296253 + - type: nauc_ndcg_at_100_diff1 + value: 26.837148341123285 + - type: nauc_ndcg_at_100_max + value: 40.33406358067434 + - type: nauc_ndcg_at_100_std + value: 28.574928808420076 + - type: nauc_ndcg_at_10_diff1 + value: 27.235940345262193 + - type: nauc_ndcg_at_10_max + value: 37.93372504090421 + - type: nauc_ndcg_at_10_std + value: 24.090106747060858 + - type: nauc_ndcg_at_1_diff1 + value: 30.197548640710586 + - type: nauc_ndcg_at_1_max + value: 28.699305487301448 + - type: nauc_ndcg_at_1_std + value: 15.591542208606757 + - type: nauc_ndcg_at_20_diff1 + value: 26.311542451666746 + - type: nauc_ndcg_at_20_max + value: 38.60977365910181 + - type: nauc_ndcg_at_20_std + value: 25.829059643447135 + - type: nauc_ndcg_at_3_diff1 + value: 26.313160792251562 + - type: nauc_ndcg_at_3_max + value: 32.87657878239498 + - type: nauc_ndcg_at_3_std + value: 17.40543281835367 + - type: nauc_ndcg_at_5_diff1 + value: 26.49173735975078 + - type: nauc_ndcg_at_5_max + value: 35.393781071988776 + - type: nauc_ndcg_at_5_std + value: 19.897433706259992 + - type: nauc_precision_at_1000_diff1 + value: -5.892707218823614 + - type: nauc_precision_at_1000_max + value: 8.008545637681289 + - type: nauc_precision_at_1000_std + value: 22.791171062328512 + - type: nauc_precision_at_100_diff1 + value: 1.1764994072550392 + - type: nauc_precision_at_100_max + value: 23.439657001775203 + - type: nauc_precision_at_100_std + value: 33.85346958354402 + - type: nauc_precision_at_10_diff1 + value: 10.599629887072993 + - type: nauc_precision_at_10_max + value: 30.796658344272643 + - type: nauc_precision_at_10_std + value: 31.592727804961047 + - type: nauc_precision_at_1_diff1 + value: 30.197548640710586 + - type: nauc_precision_at_1_max + value: 28.699305487301448 + - type: nauc_precision_at_1_std + value: 15.591542208606757 + - type: nauc_precision_at_20_diff1 + value: 5.169964235321207 + - type: nauc_precision_at_20_max + value: 27.84113856153298 + - type: nauc_precision_at_20_std + value: 32.687599811383315 + - type: nauc_precision_at_3_diff1 + value: 14.636299697383318 + - type: nauc_precision_at_3_max + value: 30.487599693224425 + - type: nauc_precision_at_3_std + value: 22.854868444529902 + - type: nauc_precision_at_5_diff1 + value: 11.483987469094416 + - type: nauc_precision_at_5_max + value: 30.416780863513328 + - type: nauc_precision_at_5_std + value: 25.885556187061187 + - type: nauc_recall_at_1000_diff1 + value: 21.348510039325454 + - type: nauc_recall_at_1000_max + value: 49.882181645332054 + - type: nauc_recall_at_1000_std + value: 56.01741566070508 + - type: nauc_recall_at_100_diff1 + value: 18.145639935832786 + - type: nauc_recall_at_100_max + value: 43.71208607215468 + - type: nauc_recall_at_100_std + value: 42.776960296168035 + - type: nauc_recall_at_10_diff1 + value: 21.508085827272694 + - type: nauc_recall_at_10_max + value: 37.58739691370799 + - type: nauc_recall_at_10_std + value: 27.84314947266321 + - type: nauc_recall_at_1_diff1 + value: 35.990766700733666 + - type: nauc_recall_at_1_max + value: 29.711074426265377 + - type: nauc_recall_at_1_std + value: 8.80172791202737 + - type: nauc_recall_at_20_diff1 + value: 18.059452327875228 + - type: nauc_recall_at_20_max + value: 38.03444901646099 + - type: nauc_recall_at_20_std + value: 31.337948210431595 + - type: nauc_recall_at_3_diff1 + value: 23.003705225329192 + - type: nauc_recall_at_3_max + value: 32.68482322928325 + - type: nauc_recall_at_3_std + value: 17.04245899987673 + - type: nauc_recall_at_5_diff1 + value: 21.456295545433857 + - type: nauc_recall_at_5_max + value: 34.98739569425076 + - type: nauc_recall_at_5_std + value: 21.157564347220983 + - type: ndcg_at_1 + value: 38.371 + - type: ndcg_at_10 + value: 39.893 + - type: ndcg_at_100 + value: 47.058 + - type: ndcg_at_1000 + value: 50.085 + - type: ndcg_at_20 + value: 42.762 + - type: ndcg_at_3 + value: 33.535 + - type: ndcg_at_5 + value: 35.859 + - type: precision_at_1 + value: 38.371 + - type: precision_at_10 + value: 12.417 + - type: precision_at_100 + value: 2.018 + - type: precision_at_1000 + value: 0.259 + - type: precision_at_20 + value: 7.478999999999999 + - type: precision_at_3 + value: 25.342 + - type: precision_at_5 + value: 19.375 + - type: recall_at_1 + value: 17.015 + - type: recall_at_10 + value: 46.322 + - type: recall_at_100 + value: 70.395 + - type: recall_at_1000 + value: 86.965 + - type: recall_at_20 + value: 54.237 + - type: recall_at_3 + value: 30.453999999999997 + - type: recall_at_5 + value: 37.457 + task: + type: Retrieval + - dataset: + config: default + name: MTEB DBPedia (default) + revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659 + split: dev + type: mteb/dbpedia + metrics: + - type: main_score + value: 53.64600000000001 + - type: map_at_1 + value: 11.886 + - type: map_at_10 + value: 26.344 + - type: map_at_100 + value: 36.681000000000004 + - type: map_at_1000 + value: 38.562000000000005 + - type: map_at_20 + value: 30.517 + - type: map_at_3 + value: 18.134 + - type: map_at_5 + value: 21.684 + - type: mrr_at_1 + value: 82.08955223880598 + - type: mrr_at_10 + value: 86.94029850746269 + - type: mrr_at_100 + value: 87.23963912494942 + - type: mrr_at_1000 + value: 87.25187303735461 + - type: mrr_at_20 + value: 87.2003618272275 + - type: mrr_at_3 + value: 86.318407960199 + - type: mrr_at_5 + value: 86.69154228855722 + - type: nauc_map_at_1000_diff1 + value: 31.904858122789655 + - type: nauc_map_at_1000_max + value: 15.393069164987324 + - type: nauc_map_at_1000_std + value: -2.5516002495745536 + - type: nauc_map_at_100_diff1 + value: 33.12859014812164 + - type: nauc_map_at_100_max + value: 16.03984329898158 + - type: nauc_map_at_100_std + value: -6.3602270619421 + - type: nauc_map_at_10_diff1 + value: 42.812205659641265 + - type: nauc_map_at_10_max + value: 9.437532136242455 + - type: nauc_map_at_10_std + value: -26.630043592840718 + - type: nauc_map_at_1_diff1 + value: 52.39673975888479 + - type: nauc_map_at_1_max + value: -4.975155424552521 + - type: nauc_map_at_1_std + value: -43.35553798396772 + - type: nauc_map_at_20_diff1 + value: 38.81156608497937 + - type: nauc_map_at_20_max + value: 10.496922767791727 + - type: nauc_map_at_20_std + value: -20.03378456478267 + - type: nauc_map_at_3_diff1 + value: 51.488644281019816 + - type: nauc_map_at_3_max + value: 3.583641040987949 + - type: nauc_map_at_3_std + value: -40.69780713904084 + - type: nauc_map_at_5_diff1 + value: 49.07272163812163 + - type: nauc_map_at_5_max + value: 7.671957527642814 + - type: nauc_map_at_5_std + value: -34.91512826568007 + - type: nauc_mrr_at_1000_diff1 + value: 52.6355971077988 + - type: nauc_mrr_at_1000_max + value: 45.36479865434538 + - type: nauc_mrr_at_1000_std + value: 32.79332484713971 + - type: nauc_mrr_at_100_diff1 + value: 52.59498488683217 + - type: nauc_mrr_at_100_max + value: 45.39162017148813 + - type: nauc_mrr_at_100_std + value: 32.83268069547622 + - type: nauc_mrr_at_10_diff1 + value: 52.90078607010709 + - type: nauc_mrr_at_10_max + value: 46.69210937923942 + - type: nauc_mrr_at_10_std + value: 33.85843763986334 + - type: nauc_mrr_at_1_diff1 + value: 49.1516816465786 + - type: nauc_mrr_at_1_max + value: 34.06625521283758 + - type: nauc_mrr_at_1_std + value: 26.7487412661101 + - type: nauc_mrr_at_20_diff1 + value: 52.746246079487506 + - type: nauc_mrr_at_20_max + value: 45.565866053324086 + - type: nauc_mrr_at_20_std + value: 33.04699997796583 + - type: nauc_mrr_at_3_diff1 + value: 53.91035759263653 + - type: nauc_mrr_at_3_max + value: 44.50150247827879 + - type: nauc_mrr_at_3_std + value: 27.57511278982597 + - type: nauc_mrr_at_5_diff1 + value: 52.95112713443137 + - type: nauc_mrr_at_5_max + value: 45.450430582134246 + - type: nauc_mrr_at_5_std + value: 33.432362399682525 + - type: nauc_ndcg_at_1000_diff1 + value: 35.923468112209264 + - type: nauc_ndcg_at_1000_max + value: 27.489887941028258 + - type: nauc_ndcg_at_1000_std + value: 21.202401371634373 + - type: nauc_ndcg_at_100_diff1 + value: 38.391045263120546 + - type: nauc_ndcg_at_100_max + value: 26.972671038514445 + - type: nauc_ndcg_at_100_std + value: 7.426194860998732 + - type: nauc_ndcg_at_10_diff1 + value: 35.116506536831345 + - type: nauc_ndcg_at_10_max + value: 19.499556706036923 + - type: nauc_ndcg_at_10_std + value: 8.153191338687364 + - type: nauc_ndcg_at_1_diff1 + value: 37.608273621659485 + - type: nauc_ndcg_at_1_max + value: 26.309365209102463 + - type: nauc_ndcg_at_1_std + value: 12.608273594894905 + - type: nauc_ndcg_at_20_diff1 + value: 36.77903793396184 + - type: nauc_ndcg_at_20_max + value: 15.937086060520981 + - type: nauc_ndcg_at_20_std + value: -4.764058119363524 + - type: nauc_ndcg_at_3_diff1 + value: 34.34013983598627 + - type: nauc_ndcg_at_3_max + value: 22.17918915302654 + - type: nauc_ndcg_at_3_std + value: 5.840704603416985 + - type: nauc_ndcg_at_5_diff1 + value: 33.00765691902229 + - type: nauc_ndcg_at_5_max + value: 24.11780582205954 + - type: nauc_ndcg_at_5_std + value: 14.681321585103008 + - type: nauc_precision_at_1000_diff1 + value: -30.348815927104173 + - type: nauc_precision_at_1000_max + value: -0.34837235715550513 + - type: nauc_precision_at_1000_std + value: 48.85702188312139 + - type: nauc_precision_at_100_diff1 + value: -27.33309281230633 + - type: nauc_precision_at_100_max + value: 9.73442173022611 + - type: nauc_precision_at_100_std + value: 53.60828426620004 + - type: nauc_precision_at_10_diff1 + value: -22.49877954403723 + - type: nauc_precision_at_10_max + value: 13.583779756215566 + - type: nauc_precision_at_10_std + value: 46.2586545254228 + - type: nauc_precision_at_1_diff1 + value: 49.1516816465786 + - type: nauc_precision_at_1_max + value: 34.06625521283758 + - type: nauc_precision_at_1_std + value: 26.7487412661101 + - type: nauc_precision_at_20_diff1 + value: -27.3321150009186 + - type: nauc_precision_at_20_max + value: 11.298895669810724 + - type: nauc_precision_at_20_std + value: 47.64225907563871 + - type: nauc_precision_at_3_diff1 + value: 6.753806545317881 + - type: nauc_precision_at_3_max + value: 28.30120636725949 + - type: nauc_precision_at_3_std + value: 28.59868816299497 + - type: nauc_precision_at_5_diff1 + value: -10.985876397497565 + - type: nauc_precision_at_5_max + value: 26.256194956378437 + - type: nauc_precision_at_5_std + value: 44.790348803819846 + - type: nauc_recall_at_1000_diff1 + value: 23.401398066050845 + - type: nauc_recall_at_1000_max + value: 26.21354448282117 + - type: nauc_recall_at_1000_std + value: 36.8829642741755 + - type: nauc_recall_at_100_diff1 + value: 30.615565095003294 + - type: nauc_recall_at_100_max + value: 26.095372785252795 + - type: nauc_recall_at_100_std + value: 6.142605509708311 + - type: nauc_recall_at_10_diff1 + value: 38.240536826377266 + - type: nauc_recall_at_10_max + value: 12.280206720878708 + - type: nauc_recall_at_10_std + value: -25.2698685049082 + - type: nauc_recall_at_1_diff1 + value: 52.39673975888479 + - type: nauc_recall_at_1_max + value: -4.975155424552521 + - type: nauc_recall_at_1_std + value: -43.35553798396772 + - type: nauc_recall_at_20_diff1 + value: 26.874728676069175 + - type: nauc_recall_at_20_max + value: 5.312933810294398 + - type: nauc_recall_at_20_std + value: -23.875335283481437 + - type: nauc_recall_at_3_diff1 + value: 50.20535980461269 + - type: nauc_recall_at_3_max + value: 5.545187994233299 + - type: nauc_recall_at_3_std + value: -40.36081527039166 + - type: nauc_recall_at_5_diff1 + value: 46.47698540226754 + - type: nauc_recall_at_5_max + value: 8.8323523086315 + - type: nauc_recall_at_5_std + value: -34.88882328531045 + - type: ndcg_at_1 + value: 73.881 + - type: ndcg_at_10 + value: 53.64600000000001 + - type: ndcg_at_100 + value: 58.67 + - type: ndcg_at_1000 + value: 66.086 + - type: ndcg_at_20 + value: 52.563 + - type: ndcg_at_3 + value: 60.464 + - type: ndcg_at_5 + value: 57.964000000000006 + - type: precision_at_1 + value: 82.09 + - type: precision_at_10 + value: 40.0 + - type: precision_at_100 + value: 11.745999999999999 + - type: precision_at_1000 + value: 1.812 + - type: precision_at_20 + value: 29.627 + - type: precision_at_3 + value: 58.706 + - type: precision_at_5 + value: 51.94 + - type: recall_at_1 + value: 11.886 + - type: recall_at_10 + value: 31.328 + - type: recall_at_100 + value: 64.5 + - type: recall_at_1000 + value: 87.897 + - type: recall_at_20 + value: 42.458 + - type: recall_at_3 + value: 19.151 + - type: recall_at_5 + value: 23.892 + task: + type: Retrieval + - dataset: + config: default + name: MTEB DBPedia (default) + revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659 + split: test + type: mteb/dbpedia + metrics: + - type: main_score + value: 46.14 + - type: map_at_1 + value: 9.625 + - type: map_at_10 + value: 22.342000000000002 + - type: map_at_100 + value: 32.669 + - type: map_at_1000 + value: 34.481 + - type: map_at_20 + value: 26.247999999999998 + - type: map_at_3 + value: 15.801000000000002 + - type: map_at_5 + value: 18.671 + - type: mrr_at_1 + value: 73.25 + - type: mrr_at_10 + value: 80.84335317460315 + - type: mrr_at_100 + value: 81.057633670892 + - type: mrr_at_1000 + value: 81.06341463210379 + - type: mrr_at_20 + value: 81.00136738261736 + - type: mrr_at_3 + value: 80.08333333333333 + - type: mrr_at_5 + value: 80.53333333333333 + - type: nauc_map_at_1000_diff1 + value: 18.808416608853623 + - type: nauc_map_at_1000_max + value: 22.274749590304392 + - type: nauc_map_at_1000_std + value: 26.993422069361873 + - type: nauc_map_at_100_diff1 + value: 19.519767936883348 + - type: nauc_map_at_100_max + value: 20.56575582187748 + - type: nauc_map_at_100_std + value: 24.017114495402883 + - type: nauc_map_at_10_diff1 + value: 20.742656127758373 + - type: nauc_map_at_10_max + value: 7.573453667001434 + - type: nauc_map_at_10_std + value: -5.456524927507794 + - type: nauc_map_at_1_diff1 + value: 36.76303782773709 + - type: nauc_map_at_1_max + value: -0.4641162841266217 + - type: nauc_map_at_1_std + value: -21.924930847883296 + - type: nauc_map_at_20_diff1 + value: 19.639655231973045 + - type: nauc_map_at_20_max + value: 13.032068028607249 + - type: nauc_map_at_20_std + value: 5.799991072412417 + - type: nauc_map_at_3_diff1 + value: 23.129315693385262 + - type: nauc_map_at_3_max + value: -0.4857267219950859 + - type: nauc_map_at_3_std + value: -18.55512119874913 + - type: nauc_map_at_5_diff1 + value: 22.3872034169013 + - type: nauc_map_at_5_max + value: 2.5672533421233066 + - type: nauc_map_at_5_std + value: -14.023487160730742 + - type: nauc_mrr_at_1000_diff1 + value: 53.16047235133225 + - type: nauc_mrr_at_1000_max + value: 60.161857198524594 + - type: nauc_mrr_at_1000_std + value: 41.774830096690316 + - type: nauc_mrr_at_100_diff1 + value: 53.170646858899275 + - type: nauc_mrr_at_100_max + value: 60.16539833163827 + - type: nauc_mrr_at_100_std + value: 41.76287840069125 + - type: nauc_mrr_at_10_diff1 + value: 52.90449578254105 + - type: nauc_mrr_at_10_max + value: 60.103747480856605 + - type: nauc_mrr_at_10_std + value: 41.699592365522484 + - type: nauc_mrr_at_1_diff1 + value: 54.231933006155565 + - type: nauc_mrr_at_1_max + value: 56.28859648688401 + - type: nauc_mrr_at_1_std + value: 38.09012078502835 + - type: nauc_mrr_at_20_diff1 + value: 53.110389369392195 + - type: nauc_mrr_at_20_max + value: 60.21316329427088 + - type: nauc_mrr_at_20_std + value: 41.754011873038706 + - type: nauc_mrr_at_3_diff1 + value: 53.041629414814075 + - type: nauc_mrr_at_3_max + value: 60.590885039620225 + - type: nauc_mrr_at_3_std + value: 42.40239299103488 + - type: nauc_mrr_at_5_diff1 + value: 52.761597419908526 + - type: nauc_mrr_at_5_max + value: 60.08630101754503 + - type: nauc_mrr_at_5_std + value: 41.84106933929747 + - type: nauc_ndcg_at_1000_diff1 + value: 24.85133713668183 + - type: nauc_ndcg_at_1000_max + value: 37.41875071827513 + - type: nauc_ndcg_at_1000_std + value: 42.50452677406385 + - type: nauc_ndcg_at_100_diff1 + value: 25.091089889124234 + - type: nauc_ndcg_at_100_max + value: 30.804260054276146 + - type: nauc_ndcg_at_100_std + value: 33.36980648385951 + - type: nauc_ndcg_at_10_diff1 + value: 23.910946578398637 + - type: nauc_ndcg_at_10_max + value: 30.131895404991514 + - type: nauc_ndcg_at_10_std + value: 27.14384930922406 + - type: nauc_ndcg_at_1_diff1 + value: 45.378548739820104 + - type: nauc_ndcg_at_1_max + value: 38.20042154427698 + - type: nauc_ndcg_at_1_std + value: 25.84957004462828 + - type: nauc_ndcg_at_20_diff1 + value: 24.248214771631464 + - type: nauc_ndcg_at_20_max + value: 28.39760869669494 + - type: nauc_ndcg_at_20_std + value: 25.7940168728995 + - type: nauc_ndcg_at_3_diff1 + value: 26.543803404398997 + - type: nauc_ndcg_at_3_max + value: 33.76121086041755 + - type: nauc_ndcg_at_3_std + value: 27.890406379302128 + - type: nauc_ndcg_at_5_diff1 + value: 24.903161199686494 + - type: nauc_ndcg_at_5_max + value: 32.30909593534191 + - type: nauc_ndcg_at_5_std + value: 27.586478184869602 + - type: nauc_precision_at_1000_diff1 + value: -14.129241148980695 + - type: nauc_precision_at_1000_max + value: 6.5821742919315005 + - type: nauc_precision_at_1000_std + value: 20.778864942184995 + - type: nauc_precision_at_100_diff1 + value: -4.8893752487067 + - type: nauc_precision_at_100_max + value: 27.291931953840038 + - type: nauc_precision_at_100_std + value: 50.48461748905693 + - type: nauc_precision_at_10_diff1 + value: -0.7667210526343395 + - type: nauc_precision_at_10_max + value: 37.46677217439101 + - type: nauc_precision_at_10_std + value: 51.55142159022884 + - type: nauc_precision_at_1_diff1 + value: 54.231933006155565 + - type: nauc_precision_at_1_max + value: 56.28859648688401 + - type: nauc_precision_at_1_std + value: 38.09012078502835 + - type: nauc_precision_at_20_diff1 + value: -2.6321106434866794 + - type: nauc_precision_at_20_max + value: 33.76155137063395 + - type: nauc_precision_at_20_std + value: 53.125611051569244 + - type: nauc_precision_at_3_diff1 + value: 6.577876589414075 + - type: nauc_precision_at_3_max + value: 37.69316279300651 + - type: nauc_precision_at_3_std + value: 41.59455105304094 + - type: nauc_precision_at_5_diff1 + value: 2.794727527273706 + - type: nauc_precision_at_5_max + value: 37.62117970256494 + - type: nauc_precision_at_5_std + value: 45.84130506152856 + - type: nauc_recall_at_1000_diff1 + value: 9.909457444514592 + - type: nauc_recall_at_1000_max + value: 31.672472097999833 + - type: nauc_recall_at_1000_std + value: 54.756814638129526 + - type: nauc_recall_at_100_diff1 + value: 16.70858982603935 + - type: nauc_recall_at_100_max + value: 21.79730939844451 + - type: nauc_recall_at_100_std + value: 30.623781783612277 + - type: nauc_recall_at_10_diff1 + value: 15.79944658167252 + - type: nauc_recall_at_10_max + value: 3.5778016982061613 + - type: nauc_recall_at_10_std + value: -8.762254293915984 + - type: nauc_recall_at_1_diff1 + value: 36.76303782773709 + - type: nauc_recall_at_1_max + value: -0.4641162841266217 + - type: nauc_recall_at_1_std + value: -21.924930847883296 + - type: nauc_recall_at_20_diff1 + value: 13.654020745847738 + - type: nauc_recall_at_20_max + value: 8.661334654305582 + - type: nauc_recall_at_20_std + value: 2.2082534655673833 + - type: nauc_recall_at_3_diff1 + value: 18.325646606940328 + - type: nauc_recall_at_3_max + value: -4.54530258566652 + - type: nauc_recall_at_3_std + value: -21.028819902366013 + - type: nauc_recall_at_5_diff1 + value: 18.65353347594189 + - type: nauc_recall_at_5_max + value: -1.1818458344246656 + - type: nauc_recall_at_5_std + value: -17.153599696172893 + - type: ndcg_at_1 + value: 60.12499999999999 + - type: ndcg_at_10 + value: 46.14 + - type: ndcg_at_100 + value: 51.80200000000001 + - type: ndcg_at_1000 + value: 58.975 + - type: ndcg_at_20 + value: 45.742 + - type: ndcg_at_3 + value: 50.745 + - type: ndcg_at_5 + value: 48.324 + - type: precision_at_1 + value: 73.25 + - type: precision_at_10 + value: 37.425000000000004 + - type: precision_at_100 + value: 12.215 + - type: precision_at_1000 + value: 2.2689999999999997 + - type: precision_at_20 + value: 28.775000000000002 + - type: precision_at_3 + value: 54.833 + - type: precision_at_5 + value: 47.55 + - type: recall_at_1 + value: 9.625 + - type: recall_at_10 + value: 27.877000000000002 + - type: recall_at_100 + value: 58.797999999999995 + - type: recall_at_1000 + value: 81.645 + - type: recall_at_20 + value: 36.097 + - type: recall_at_3 + value: 17.492 + - type: recall_at_5 + value: 21.497 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FEVER (default) + revision: bea83ef9e8fb933d90a2f1d5515737465d613e12 + split: dev + type: mteb/fever + metrics: + - type: main_score + value: 90.834 + - type: map_at_1 + value: 80.604 + - type: map_at_10 + value: 87.81 + - type: map_at_100 + value: 88.015 + - type: map_at_1000 + value: 88.02799999999999 + - type: map_at_20 + value: 87.928 + - type: map_at_3 + value: 87.07300000000001 + - type: map_at_5 + value: 87.527 + - type: mrr_at_1 + value: 87.3987398739874 + - type: mrr_at_10 + value: 92.76807442649016 + - type: mrr_at_100 + value: 92.78949106669239 + - type: mrr_at_1000 + value: 92.78974222320679 + - type: mrr_at_20 + value: 92.78556988298675 + - type: mrr_at_3 + value: 92.4892489248924 + - type: mrr_at_5 + value: 92.68351835183503 + - type: nauc_map_at_1000_diff1 + value: 47.66982481342224 + - type: nauc_map_at_1000_max + value: 17.335595727415615 + - type: nauc_map_at_1000_std + value: -3.374731443502621 + - type: nauc_map_at_100_diff1 + value: 47.62390871144849 + - type: nauc_map_at_100_max + value: 17.333898019751715 + - type: nauc_map_at_100_std + value: -3.3676563570240834 + - type: nauc_map_at_10_diff1 + value: 47.00400674886937 + - type: nauc_map_at_10_max + value: 16.95808851125027 + - type: nauc_map_at_10_std + value: -3.7540636078709215 + - type: nauc_map_at_1_diff1 + value: 54.73601392293206 + - type: nauc_map_at_1_max + value: 12.196886082153208 + - type: nauc_map_at_1_std + value: -7.478404784374586 + - type: nauc_map_at_20_diff1 + value: 47.32361447078238 + - type: nauc_map_at_20_max + value: 17.23360070788687 + - type: nauc_map_at_20_std + value: -3.5054382123007053 + - type: nauc_map_at_3_diff1 + value: 46.41583091904632 + - type: nauc_map_at_3_max + value: 16.36159010582028 + - type: nauc_map_at_3_std + value: -4.658049536832921 + - type: nauc_map_at_5_diff1 + value: 46.52946366198386 + - type: nauc_map_at_5_max + value: 16.74450400580674 + - type: nauc_map_at_5_std + value: -4.108220142499625 + - type: nauc_mrr_at_1000_diff1 + value: 76.23174721823989 + - type: nauc_mrr_at_1000_max + value: 20.679036621825436 + - type: nauc_mrr_at_1000_std + value: -12.372646874535736 + - type: nauc_mrr_at_100_diff1 + value: 76.23235829015022 + - type: nauc_mrr_at_100_max + value: 20.68186609726051 + - type: nauc_mrr_at_100_std + value: -12.368979971123155 + - type: nauc_mrr_at_10_diff1 + value: 76.18677256786995 + - type: nauc_mrr_at_10_max + value: 20.766965521284185 + - type: nauc_mrr_at_10_std + value: -12.416808565439725 + - type: nauc_mrr_at_1_diff1 + value: 76.85474065189635 + - type: nauc_mrr_at_1_max + value: 17.22882213900808 + - type: nauc_mrr_at_1_std + value: -11.696873831763623 + - type: nauc_mrr_at_20_diff1 + value: 76.22993750308532 + - type: nauc_mrr_at_20_max + value: 20.712794831345168 + - type: nauc_mrr_at_20_std + value: -12.330623369594392 + - type: nauc_mrr_at_3_diff1 + value: 75.93451229919016 + - type: nauc_mrr_at_3_max + value: 21.610990192567662 + - type: nauc_mrr_at_3_std + value: -12.809588044640897 + - type: nauc_mrr_at_5_diff1 + value: 76.21371710254333 + - type: nauc_mrr_at_5_max + value: 21.074935296799822 + - type: nauc_mrr_at_5_std + value: -12.719619000388349 + - type: nauc_ndcg_at_1000_diff1 + value: 51.014409325297386 + - type: nauc_ndcg_at_1000_max + value: 19.699332852773484 + - type: nauc_ndcg_at_1000_std + value: -2.2709665879841845 + - type: nauc_ndcg_at_100_diff1 + value: 50.07911650536347 + - type: nauc_ndcg_at_100_max + value: 19.77727110232642 + - type: nauc_ndcg_at_100_std + value: -1.8464196973530789 + - type: nauc_ndcg_at_10_diff1 + value: 47.49483709069946 + - type: nauc_ndcg_at_10_max + value: 18.582490956167547 + - type: nauc_ndcg_at_10_std + value: -3.0256905511703143 + - type: nauc_ndcg_at_1_diff1 + value: 76.85474065189635 + - type: nauc_ndcg_at_1_max + value: 17.22882213900808 + - type: nauc_ndcg_at_1_std + value: -11.696873831763623 + - type: nauc_ndcg_at_20_diff1 + value: 48.40342672507231 + - type: nauc_ndcg_at_20_max + value: 19.228600038947512 + - type: nauc_ndcg_at_20_std + value: -2.3311737628995703 + - type: nauc_ndcg_at_3_diff1 + value: 47.61114139623747 + - type: nauc_ndcg_at_3_max + value: 18.69979123624098 + - type: nauc_ndcg_at_3_std + value: -4.558476300243176 + - type: nauc_ndcg_at_5_diff1 + value: 46.915186073259804 + - type: nauc_ndcg_at_5_max + value: 18.474697066818933 + - type: nauc_ndcg_at_5_std + value: -3.8495078567474654 + - type: nauc_precision_at_1000_diff1 + value: -7.7185510931166 + - type: nauc_precision_at_1000_max + value: 3.1094709117338106 + - type: nauc_precision_at_1000_std + value: 8.574501659251595 + - type: nauc_precision_at_100_diff1 + value: -7.575980314171725 + - type: nauc_precision_at_100_max + value: 6.844394394588453 + - type: nauc_precision_at_100_std + value: 11.979916297807362 + - type: nauc_precision_at_10_diff1 + value: -9.326564451079632 + - type: nauc_precision_at_10_max + value: 11.195895816468909 + - type: nauc_precision_at_10_std + value: 10.47911838738924 + - type: nauc_precision_at_1_diff1 + value: 76.85474065189635 + - type: nauc_precision_at_1_max + value: 17.22882213900808 + - type: nauc_precision_at_1_std + value: -11.696873831763623 + - type: nauc_precision_at_20_diff1 + value: -9.003158694428597 + - type: nauc_precision_at_20_max + value: 10.061127490589199 + - type: nauc_precision_at_20_std + value: 11.887489241368932 + - type: nauc_precision_at_3_diff1 + value: 0.16916287454937304 + - type: nauc_precision_at_3_max + value: 16.98083985881467 + - type: nauc_precision_at_3_std + value: 4.603256524512925 + - type: nauc_precision_at_5_diff1 + value: -7.50002391068408 + - type: nauc_precision_at_5_max + value: 14.407165406406792 + - type: nauc_precision_at_5_std + value: 8.410890993074611 + - type: nauc_recall_at_1000_diff1 + value: 4.719347205347491 + - type: nauc_recall_at_1000_max + value: 40.59300834847414 + - type: nauc_recall_at_1000_std + value: 49.284495449709155 + - type: nauc_recall_at_100_diff1 + value: 8.48851657784082 + - type: nauc_recall_at_100_max + value: 32.86730307842109 + - type: nauc_recall_at_100_std + value: 34.92891606130411 + - type: nauc_recall_at_10_diff1 + value: 9.091653822294713 + - type: nauc_recall_at_10_max + value: 19.97932228248086 + - type: nauc_recall_at_10_std + value: 11.726667783827 + - type: nauc_recall_at_1_diff1 + value: 54.73601392293206 + - type: nauc_recall_at_1_max + value: 12.196886082153208 + - type: nauc_recall_at_1_std + value: -7.478404784374586 + - type: nauc_recall_at_20_diff1 + value: 7.175104904115427 + - type: nauc_recall_at_20_max + value: 23.88719101063754 + - type: nauc_recall_at_20_std + value: 19.70879315323268 + - type: nauc_recall_at_3_diff1 + value: 20.75863060017615 + - type: nauc_recall_at_3_max + value: 19.292908601356697 + - type: nauc_recall_at_3_std + value: 0.8778108332026987 + - type: nauc_recall_at_5_diff1 + value: 14.388632469474533 + - type: nauc_recall_at_5_max + value: 19.74920228365671 + - type: nauc_recall_at_5_std + value: 5.125514707743805 + - type: ndcg_at_1 + value: 87.399 + - type: ndcg_at_10 + value: 90.834 + - type: ndcg_at_100 + value: 91.43900000000001 + - type: ndcg_at_1000 + value: 91.643 + - type: ndcg_at_20 + value: 91.10000000000001 + - type: ndcg_at_3 + value: 89.956 + - type: ndcg_at_5 + value: 90.4 + - type: precision_at_1 + value: 87.399 + - type: precision_at_10 + value: 10.695 + - type: precision_at_100 + value: 1.131 + - type: precision_at_1000 + value: 0.117 + - type: precision_at_20 + value: 5.453 + - type: precision_at_3 + value: 34.078 + - type: precision_at_5 + value: 20.879 + - type: recall_at_1 + value: 80.604 + - type: recall_at_10 + value: 95.22 + - type: recall_at_100 + value: 97.395 + - type: recall_at_1000 + value: 98.59599999999999 + - type: recall_at_20 + value: 96.06700000000001 + - type: recall_at_3 + value: 92.601 + - type: recall_at_5 + value: 93.95700000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FEVER (default) + revision: bea83ef9e8fb933d90a2f1d5515737465d613e12 + split: test + type: mteb/fever + metrics: + - type: main_score + value: 90.25800000000001 + - type: map_at_1 + value: 79.453 + - type: map_at_10 + value: 87.09 + - type: map_at_100 + value: 87.306 + - type: map_at_1000 + value: 87.318 + - type: map_at_20 + value: 87.223 + - type: map_at_3 + value: 86.181 + - type: map_at_5 + value: 86.771 + - type: mrr_at_1 + value: 85.7035703570357 + - type: mrr_at_10 + value: 91.59727877549649 + - type: mrr_at_100 + value: 91.62878189000877 + - type: mrr_at_1000 + value: 91.62956689674226 + - type: mrr_at_20 + value: 91.62221108033692 + - type: mrr_at_3 + value: 91.25662566256617 + - type: mrr_at_5 + value: 91.49889988998886 + - type: nauc_map_at_1000_diff1 + value: 48.95967465500532 + - type: nauc_map_at_1000_max + value: 13.552958155142544 + - type: nauc_map_at_1000_std + value: -6.4905216456926995 + - type: nauc_map_at_100_diff1 + value: 48.89414890666479 + - type: nauc_map_at_100_max + value: 13.535442300209844 + - type: nauc_map_at_100_std + value: -6.477107890977689 + - type: nauc_map_at_10_diff1 + value: 48.44305750037048 + - type: nauc_map_at_10_max + value: 13.3766772939877 + - type: nauc_map_at_10_std + value: -6.612176221534562 + - type: nauc_map_at_1_diff1 + value: 54.093905885669734 + - type: nauc_map_at_1_max + value: 7.7983577556125185 + - type: nauc_map_at_1_std + value: -10.652843802676353 + - type: nauc_map_at_20_diff1 + value: 48.67326568334062 + - type: nauc_map_at_20_max + value: 13.487634425801796 + - type: nauc_map_at_20_std + value: -6.410918320951313 + - type: nauc_map_at_3_diff1 + value: 48.29932184107172 + - type: nauc_map_at_3_max + value: 13.02326893220213 + - type: nauc_map_at_3_std + value: -7.722418221654614 + - type: nauc_map_at_5_diff1 + value: 48.181168136332595 + - type: nauc_map_at_5_max + value: 13.171144379780056 + - type: nauc_map_at_5_std + value: -7.136239742353514 + - type: nauc_mrr_at_1000_diff1 + value: 71.68636763108105 + - type: nauc_mrr_at_1000_max + value: 13.112870901835366 + - type: nauc_mrr_at_1000_std + value: -16.816140112002877 + - type: nauc_mrr_at_100_diff1 + value: 71.68340883100159 + - type: nauc_mrr_at_100_max + value: 13.118827070456662 + - type: nauc_mrr_at_100_std + value: -16.80570967438365 + - type: nauc_mrr_at_10_diff1 + value: 71.66920684557107 + - type: nauc_mrr_at_10_max + value: 13.322749754458684 + - type: nauc_mrr_at_10_std + value: -16.747882600201798 + - type: nauc_mrr_at_1_diff1 + value: 71.96165982322854 + - type: nauc_mrr_at_1_max + value: 9.584871421344907 + - type: nauc_mrr_at_1_std + value: -16.695718104507566 + - type: nauc_mrr_at_20_diff1 + value: 71.68644116651814 + - type: nauc_mrr_at_20_max + value: 13.175985071299964 + - type: nauc_mrr_at_20_std + value: -16.750815576718665 + - type: nauc_mrr_at_3_diff1 + value: 71.90727611125823 + - type: nauc_mrr_at_3_max + value: 13.626458987265941 + - type: nauc_mrr_at_3_std + value: -17.84496816797055 + - type: nauc_mrr_at_5_diff1 + value: 71.7854913757351 + - type: nauc_mrr_at_5_max + value: 13.575403146316267 + - type: nauc_mrr_at_5_std + value: -17.08521934193673 + - type: nauc_ndcg_at_1000_diff1 + value: 52.066585605962246 + - type: nauc_ndcg_at_1000_max + value: 15.361479005367206 + - type: nauc_ndcg_at_1000_std + value: -6.088585873735421 + - type: nauc_ndcg_at_100_diff1 + value: 50.72143377042151 + - type: nauc_ndcg_at_100_max + value: 15.238646937739356 + - type: nauc_ndcg_at_100_std + value: -5.4770070162753735 + - type: nauc_ndcg_at_10_diff1 + value: 48.647682887084095 + - type: nauc_ndcg_at_10_max + value: 15.035559255940637 + - type: nauc_ndcg_at_10_std + value: -5.547993952394186 + - type: nauc_ndcg_at_1_diff1 + value: 71.96165982322854 + - type: nauc_ndcg_at_1_max + value: 9.584871421344907 + - type: nauc_ndcg_at_1_std + value: -16.695718104507566 + - type: nauc_ndcg_at_20_diff1 + value: 49.36402205451731 + - type: nauc_ndcg_at_20_max + value: 15.165527068805423 + - type: nauc_ndcg_at_20_std + value: -5.020480694558102 + - type: nauc_ndcg_at_3_diff1 + value: 50.19393399264834 + - type: nauc_ndcg_at_3_max + value: 14.963993281745477 + - type: nauc_ndcg_at_3_std + value: -8.46649849858527 + - type: nauc_ndcg_at_5_diff1 + value: 48.748692877075804 + - type: nauc_ndcg_at_5_max + value: 14.973018370459279 + - type: nauc_ndcg_at_5_std + value: -6.884456655111388 + - type: nauc_precision_at_1000_diff1 + value: -3.426030106577485 + - type: nauc_precision_at_1000_max + value: 4.431468036965319 + - type: nauc_precision_at_1000_std + value: 4.2796433448579325 + - type: nauc_precision_at_100_diff1 + value: -7.225817278615556 + - type: nauc_precision_at_100_max + value: 5.727540376540685 + - type: nauc_precision_at_100_std + value: 7.621823945869666 + - type: nauc_precision_at_10_diff1 + value: -8.512808836712555 + - type: nauc_precision_at_10_max + value: 10.062000086858193 + - type: nauc_precision_at_10_std + value: 10.34606930713952 + - type: nauc_precision_at_1_diff1 + value: 71.96165982322854 + - type: nauc_precision_at_1_max + value: 9.584871421344907 + - type: nauc_precision_at_1_std + value: -16.695718104507566 + - type: nauc_precision_at_20_diff1 + value: -8.85903500015368 + - type: nauc_precision_at_20_max + value: 8.556783597219457 + - type: nauc_precision_at_20_std + value: 11.112190849799777 + - type: nauc_precision_at_3_diff1 + value: 9.315729981233503 + - type: nauc_precision_at_3_max + value: 19.025669680080902 + - type: nauc_precision_at_3_std + value: 3.8698499676908167 + - type: nauc_precision_at_5_diff1 + value: -3.7299965413574983 + - type: nauc_precision_at_5_max + value: 13.305616510081416 + - type: nauc_precision_at_5_std + value: 7.63924483161138 + - type: nauc_recall_at_1000_diff1 + value: 11.191174439777496 + - type: nauc_recall_at_1000_max + value: 54.4082555481569 + - type: nauc_recall_at_1000_std + value: 54.15386504969637 + - type: nauc_recall_at_100_diff1 + value: -0.03126529063446544 + - type: nauc_recall_at_100_max + value: 34.1035151022096 + - type: nauc_recall_at_100_std + value: 41.37762363765351 + - type: nauc_recall_at_10_diff1 + value: 11.244080211910003 + - type: nauc_recall_at_10_max + value: 24.618319335111526 + - type: nauc_recall_at_10_std + value: 17.87932822395642 + - type: nauc_recall_at_1_diff1 + value: 54.093905885669734 + - type: nauc_recall_at_1_max + value: 7.7983577556125185 + - type: nauc_recall_at_1_std + value: -10.652843802676353 + - type: nauc_recall_at_20_diff1 + value: 6.502935901039147 + - type: nauc_recall_at_20_max + value: 27.206947142553663 + - type: nauc_recall_at_20_std + value: 28.032976285160043 + - type: nauc_recall_at_3_diff1 + value: 27.41330897260636 + - type: nauc_recall_at_3_max + value: 19.142212328857084 + - type: nauc_recall_at_3_std + value: -1.6728318873826566 + - type: nauc_recall_at_5_diff1 + value: 19.199930042258764 + - type: nauc_recall_at_5_max + value: 21.354041676576575 + - type: nauc_recall_at_5_std + value: 5.798726913017626 + - type: ndcg_at_1 + value: 85.70400000000001 + - type: ndcg_at_10 + value: 90.25800000000001 + - type: ndcg_at_100 + value: 90.922 + - type: ndcg_at_1000 + value: 91.098 + - type: ndcg_at_20 + value: 90.572 + - type: ndcg_at_3 + value: 89.071 + - type: ndcg_at_5 + value: 89.742 + - type: precision_at_1 + value: 85.70400000000001 + - type: precision_at_10 + value: 10.671 + - type: precision_at_100 + value: 1.1280000000000001 + - type: precision_at_1000 + value: 0.116 + - type: precision_at_20 + value: 5.443 + - type: precision_at_3 + value: 33.708 + - type: precision_at_5 + value: 20.785999999999998 + - type: recall_at_1 + value: 79.453 + - type: recall_at_10 + value: 95.478 + - type: recall_at_100 + value: 97.898 + - type: recall_at_1000 + value: 98.931 + - type: recall_at_20 + value: 96.481 + - type: recall_at_3 + value: 92.26299999999999 + - type: recall_at_5 + value: 94.003 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FEVER (default) + revision: bea83ef9e8fb933d90a2f1d5515737465d613e12 + split: train + type: mteb/fever + metrics: + - type: main_score + value: 90.19 + - type: map_at_1 + value: 77.774 + - type: map_at_10 + value: 86.751 + - type: map_at_100 + value: 87.01 + - type: map_at_1000 + value: 87.027 + - type: map_at_20 + value: 86.902 + - type: map_at_3 + value: 85.82300000000001 + - type: map_at_5 + value: 86.414 + - type: mrr_at_1 + value: 86.4292869501867 + - type: mrr_at_10 + value: 92.42043327363166 + - type: mrr_at_100 + value: 92.43039837206005 + - type: mrr_at_1000 + value: 92.43049509357725 + - type: mrr_at_20 + value: 92.42830554720031 + - type: mrr_at_3 + value: 92.17299578058855 + - type: mrr_at_5 + value: 92.36906171265254 + - type: nauc_map_at_1000_diff1 + value: 38.51962805312923 + - type: nauc_map_at_1000_max + value: -2.584449676087596 + - type: nauc_map_at_1000_std + value: -14.754716102820922 + - type: nauc_map_at_100_diff1 + value: 38.45419775334632 + - type: nauc_map_at_100_max + value: -2.561182669607417 + - type: nauc_map_at_100_std + value: -14.710908124127153 + - type: nauc_map_at_10_diff1 + value: 37.73723021819721 + - type: nauc_map_at_10_max + value: -2.48929251013858 + - type: nauc_map_at_10_std + value: -14.617303353672304 + - type: nauc_map_at_1_diff1 + value: 45.59367471780984 + - type: nauc_map_at_1_max + value: -8.79394540843278 + - type: nauc_map_at_1_std + value: -18.592849963100452 + - type: nauc_map_at_20_diff1 + value: 38.094018510918545 + - type: nauc_map_at_20_max + value: -2.4946663351513765 + - type: nauc_map_at_20_std + value: -14.617529049915062 + - type: nauc_map_at_3_diff1 + value: 37.2825384594879 + - type: nauc_map_at_3_max + value: -2.7437227164766793 + - type: nauc_map_at_3_std + value: -15.615179684316619 + - type: nauc_map_at_5_diff1 + value: 37.29251719589608 + - type: nauc_map_at_5_max + value: -2.490965706373661 + - type: nauc_map_at_5_std + value: -14.750128097927359 + - type: nauc_mrr_at_1000_diff1 + value: 66.41322459273906 + - type: nauc_mrr_at_1000_max + value: -16.489014563746952 + - type: nauc_mrr_at_1000_std + value: -35.84365264710914 + - type: nauc_mrr_at_100_diff1 + value: 66.41345340021958 + - type: nauc_mrr_at_100_max + value: -16.487687647735363 + - type: nauc_mrr_at_100_std + value: -35.84266306778577 + - type: nauc_mrr_at_10_diff1 + value: 66.42934132853334 + - type: nauc_mrr_at_10_max + value: -16.413124135695732 + - type: nauc_mrr_at_10_std + value: -35.84526130937461 + - type: nauc_mrr_at_1_diff1 + value: 66.67028094421802 + - type: nauc_mrr_at_1_max + value: -17.165368836050753 + - type: nauc_mrr_at_1_std + value: -33.261826441743125 + - type: nauc_mrr_at_20_diff1 + value: 66.41476879339757 + - type: nauc_mrr_at_20_max + value: -16.46589905273922 + - type: nauc_mrr_at_20_std + value: -35.83503515437515 + - type: nauc_mrr_at_3_diff1 + value: 66.32619482173827 + - type: nauc_mrr_at_3_max + value: -16.345253832335274 + - type: nauc_mrr_at_3_std + value: -36.927314989541614 + - type: nauc_mrr_at_5_diff1 + value: 66.43639260714113 + - type: nauc_mrr_at_5_max + value: -16.257534579289114 + - type: nauc_mrr_at_5_std + value: -36.02427701751464 + - type: nauc_ndcg_at_1000_diff1 + value: 41.92531749803412 + - type: nauc_ndcg_at_1000_max + value: -2.9840601177890136 + - type: nauc_ndcg_at_1000_std + value: -15.9844531171351 + - type: nauc_ndcg_at_100_diff1 + value: 40.54880734081055 + - type: nauc_ndcg_at_100_max + value: -2.279659009100174 + - type: nauc_ndcg_at_100_std + value: -14.831218566119967 + - type: nauc_ndcg_at_10_diff1 + value: 37.86462563664053 + - type: nauc_ndcg_at_10_max + value: -1.530723987317026 + - type: nauc_ndcg_at_10_std + value: -14.082379523495161 + - type: nauc_ndcg_at_1_diff1 + value: 66.67028094421802 + - type: nauc_ndcg_at_1_max + value: -17.165368836050753 + - type: nauc_ndcg_at_1_std + value: -33.261826441743125 + - type: nauc_ndcg_at_20_diff1 + value: 38.76126966520895 + - type: nauc_ndcg_at_20_max + value: -1.632449728251427 + - type: nauc_ndcg_at_20_std + value: -14.086296686315958 + - type: nauc_ndcg_at_3_diff1 + value: 39.42102302865233 + - type: nauc_ndcg_at_3_max + value: -2.020301091355397 + - type: nauc_ndcg_at_3_std + value: -17.010312259332512 + - type: nauc_ndcg_at_5_diff1 + value: 37.69023629578657 + - type: nauc_ndcg_at_5_max + value: -1.4553705361052656 + - type: nauc_ndcg_at_5_std + value: -14.662676754364872 + - type: nauc_precision_at_1000_diff1 + value: -2.579666933975314 + - type: nauc_precision_at_1000_max + value: 0.4665939639671969 + - type: nauc_precision_at_1000_std + value: 0.8422650657008633 + - type: nauc_precision_at_100_diff1 + value: -4.9995368220526135 + - type: nauc_precision_at_100_max + value: 2.9446170123168565 + - type: nauc_precision_at_100_std + value: 4.229678644683377 + - type: nauc_precision_at_10_diff1 + value: -9.774162327456947 + - type: nauc_precision_at_10_max + value: 5.37831433315376 + - type: nauc_precision_at_10_std + value: 4.7842571438021375 + - type: nauc_precision_at_1_diff1 + value: 66.67028094421802 + - type: nauc_precision_at_1_max + value: -17.165368836050753 + - type: nauc_precision_at_1_std + value: -33.261826441743125 + - type: nauc_precision_at_20_diff1 + value: -8.053926975071425 + - type: nauc_precision_at_20_max + value: 5.015267267048775 + - type: nauc_precision_at_20_std + value: 5.39589484985495 + - type: nauc_precision_at_3_diff1 + value: -6.794243492247966 + - type: nauc_precision_at_3_max + value: 5.795585879471898 + - type: nauc_precision_at_3_std + value: -3.0991408572665646 + - type: nauc_precision_at_5_diff1 + value: -10.807385999336905 + - type: nauc_precision_at_5_max + value: 6.047919916817182 + - type: nauc_precision_at_5_std + value: 3.0744980914193816 + - type: nauc_recall_at_1000_diff1 + value: -8.83235270771091 + - type: nauc_recall_at_1000_max + value: 45.12283312420266 + - type: nauc_recall_at_1000_std + value: 52.208444882282635 + - type: nauc_recall_at_100_diff1 + value: -7.479713459569299 + - type: nauc_recall_at_100_max + value: 33.80130037870934 + - type: nauc_recall_at_100_std + value: 38.447888209366944 + - type: nauc_recall_at_10_diff1 + value: -1.196038439665805 + - type: nauc_recall_at_10_max + value: 22.634429878615137 + - type: nauc_recall_at_10_std + value: 18.560653718536248 + - type: nauc_recall_at_1_diff1 + value: 45.59367471780984 + - type: nauc_recall_at_1_max + value: -8.79394540843278 + - type: nauc_recall_at_1_std + value: -18.592849963100452 + - type: nauc_recall_at_20_diff1 + value: -3.919520514412911 + - type: nauc_recall_at_20_max + value: 26.943409555922642 + - type: nauc_recall_at_20_std + value: 25.528749887760394 + - type: nauc_recall_at_3_diff1 + value: 11.740112034107208 + - type: nauc_recall_at_3_max + value: 12.315680617281167 + - type: nauc_recall_at_3_std + value: -0.8629495880752692 + - type: nauc_recall_at_5_diff1 + value: 4.061518157560268 + - type: nauc_recall_at_5_max + value: 18.10733238594876 + - type: nauc_recall_at_5_std + value: 9.959026977903445 + - type: ndcg_at_1 + value: 86.429 + - type: ndcg_at_10 + value: 90.19 + - type: ndcg_at_100 + value: 90.911 + - type: ndcg_at_1000 + value: 91.15299999999999 + - type: ndcg_at_20 + value: 90.505 + - type: ndcg_at_3 + value: 89.239 + - type: ndcg_at_5 + value: 89.749 + - type: precision_at_1 + value: 86.429 + - type: precision_at_10 + value: 11.123 + - type: precision_at_100 + value: 1.193 + - type: precision_at_1000 + value: 0.124 + - type: precision_at_20 + value: 5.704 + - type: precision_at_3 + value: 35.002 + - type: precision_at_5 + value: 21.605 + - type: recall_at_1 + value: 77.774 + - type: recall_at_10 + value: 95.109 + - type: recall_at_100 + value: 97.565 + - type: recall_at_1000 + value: 98.92999999999999 + - type: recall_at_20 + value: 96.032 + - type: recall_at_3 + value: 92.191 + - type: recall_at_5 + value: 93.818 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FiQA2018 (default) + revision: 27a168819829fe9bcd655c2df245fb19452e8e06 + split: dev + type: mteb/fiqa + metrics: + - type: main_score + value: 47.259 + - type: map_at_1 + value: 25.491999999999997 + - type: map_at_10 + value: 39.363 + - type: map_at_100 + value: 41.281 + - type: map_at_1000 + value: 41.433 + - type: map_at_20 + value: 40.445 + - type: map_at_3 + value: 35.142 + - type: map_at_5 + value: 37.381 + - type: mrr_at_1 + value: 45.6 + - type: mrr_at_10 + value: 54.605396825396824 + - type: mrr_at_100 + value: 55.254906558729786 + - type: mrr_at_1000 + value: 55.2874716323657 + - type: mrr_at_20 + value: 54.980650513057626 + - type: mrr_at_3 + value: 52.46666666666668 + - type: mrr_at_5 + value: 53.65666666666667 + - type: nauc_map_at_1000_diff1 + value: 50.72180427594406 + - type: nauc_map_at_1000_max + value: 30.876082039757236 + - type: nauc_map_at_1000_std + value: 1.2116689446228104 + - type: nauc_map_at_100_diff1 + value: 50.68177752337445 + - type: nauc_map_at_100_max + value: 30.785700629549623 + - type: nauc_map_at_100_std + value: 1.1782094560924603 + - type: nauc_map_at_10_diff1 + value: 50.833761742669296 + - type: nauc_map_at_10_max + value: 30.023634696159178 + - type: nauc_map_at_10_std + value: -0.7690872824966695 + - type: nauc_map_at_1_diff1 + value: 53.688246820443666 + - type: nauc_map_at_1_max + value: 22.798469345843053 + - type: nauc_map_at_1_std + value: -8.116505032590933 + - type: nauc_map_at_20_diff1 + value: 50.72178455757237 + - type: nauc_map_at_20_max + value: 30.272614157124245 + - type: nauc_map_at_20_std + value: 0.3426372520251747 + - type: nauc_map_at_3_diff1 + value: 50.437105587236395 + - type: nauc_map_at_3_max + value: 25.94808478989068 + - type: nauc_map_at_3_std + value: -5.113415398823721 + - type: nauc_map_at_5_diff1 + value: 51.08971801336635 + - type: nauc_map_at_5_max + value: 28.44694164848957 + - type: nauc_map_at_5_std + value: -2.873984613746069 + - type: nauc_mrr_at_1000_diff1 + value: 58.05571877329422 + - type: nauc_mrr_at_1000_max + value: 43.50113665892488 + - type: nauc_mrr_at_1000_std + value: 7.235970711775945 + - type: nauc_mrr_at_100_diff1 + value: 58.04631900388618 + - type: nauc_mrr_at_100_max + value: 43.50169072674772 + - type: nauc_mrr_at_100_std + value: 7.260998970091722 + - type: nauc_mrr_at_10_diff1 + value: 58.10753352699932 + - type: nauc_mrr_at_10_max + value: 43.619051743963574 + - type: nauc_mrr_at_10_std + value: 7.252255743827588 + - type: nauc_mrr_at_1_diff1 + value: 60.814201282818416 + - type: nauc_mrr_at_1_max + value: 41.791851688554324 + - type: nauc_mrr_at_1_std + value: 3.7484647119956973 + - type: nauc_mrr_at_20_diff1 + value: 58.10213709860492 + - type: nauc_mrr_at_20_max + value: 43.46697000275535 + - type: nauc_mrr_at_20_std + value: 7.3321550640283855 + - type: nauc_mrr_at_3_diff1 + value: 57.84515480159098 + - type: nauc_mrr_at_3_max + value: 42.114438777545715 + - type: nauc_mrr_at_3_std + value: 4.989056401054402 + - type: nauc_mrr_at_5_diff1 + value: 57.83131361346262 + - type: nauc_mrr_at_5_max + value: 43.38780575879838 + - type: nauc_mrr_at_5_std + value: 6.128059137749874 + - type: nauc_ndcg_at_1000_diff1 + value: 52.24461941865872 + - type: nauc_ndcg_at_1000_max + value: 37.12605387314286 + - type: nauc_ndcg_at_1000_std + value: 8.15381458587306 + - type: nauc_ndcg_at_100_diff1 + value: 51.44175370568701 + - type: nauc_ndcg_at_100_max + value: 36.3883223360929 + - type: nauc_ndcg_at_100_std + value: 8.650088808677124 + - type: nauc_ndcg_at_10_diff1 + value: 51.5338374152218 + - type: nauc_ndcg_at_10_max + value: 34.47813857838582 + - type: nauc_ndcg_at_10_std + value: 4.3502022833370155 + - type: nauc_ndcg_at_1_diff1 + value: 60.814201282818416 + - type: nauc_ndcg_at_1_max + value: 41.791851688554324 + - type: nauc_ndcg_at_1_std + value: 3.7484647119956973 + - type: nauc_ndcg_at_20_diff1 + value: 51.420581933105716 + - type: nauc_ndcg_at_20_max + value: 34.63146192008459 + - type: nauc_ndcg_at_20_std + value: 6.307667763196827 + - type: nauc_ndcg_at_3_diff1 + value: 50.057122417501496 + - type: nauc_ndcg_at_3_max + value: 32.675529929464645 + - type: nauc_ndcg_at_3_std + value: 1.1764904379796604 + - type: nauc_ndcg_at_5_diff1 + value: 51.17028777004597 + - type: nauc_ndcg_at_5_max + value: 32.94317314476149 + - type: nauc_ndcg_at_5_std + value: 1.7130455231503465 + - type: nauc_precision_at_1000_diff1 + value: -0.9392230592457282 + - type: nauc_precision_at_1000_max + value: 22.345993491836673 + - type: nauc_precision_at_1000_std + value: 24.178427458007544 + - type: nauc_precision_at_100_diff1 + value: 4.036666149801863 + - type: nauc_precision_at_100_max + value: 26.548989769682702 + - type: nauc_precision_at_100_std + value: 29.2974364707028 + - type: nauc_precision_at_10_diff1 + value: 17.51353122807735 + - type: nauc_precision_at_10_max + value: 32.43490416766948 + - type: nauc_precision_at_10_std + value: 21.839530132265512 + - type: nauc_precision_at_1_diff1 + value: 60.814201282818416 + - type: nauc_precision_at_1_max + value: 41.791851688554324 + - type: nauc_precision_at_1_std + value: 3.7484647119956973 + - type: nauc_precision_at_20_diff1 + value: 11.223136725113264 + - type: nauc_precision_at_20_max + value: 28.98052334596371 + - type: nauc_precision_at_20_std + value: 26.97618173457716 + - type: nauc_precision_at_3_diff1 + value: 31.043853181414928 + - type: nauc_precision_at_3_max + value: 33.11108166956598 + - type: nauc_precision_at_3_std + value: 10.618685268602885 + - type: nauc_precision_at_5_diff1 + value: 25.520865305521102 + - type: nauc_precision_at_5_max + value: 33.48283016783097 + - type: nauc_precision_at_5_std + value: 15.157211083964592 + - type: nauc_recall_at_1000_diff1 + value: 42.74299853922106 + - type: nauc_recall_at_1000_max + value: 42.18810540059435 + - type: nauc_recall_at_1000_std + value: 40.599584937563954 + - type: nauc_recall_at_100_diff1 + value: 36.358133041879 + - type: nauc_recall_at_100_max + value: 32.89739462535902 + - type: nauc_recall_at_100_std + value: 28.262077367665505 + - type: nauc_recall_at_10_diff1 + value: 42.33807743083503 + - type: nauc_recall_at_10_max + value: 29.707954836506982 + - type: nauc_recall_at_10_std + value: 8.703138122786624 + - type: nauc_recall_at_1_diff1 + value: 53.688246820443666 + - type: nauc_recall_at_1_max + value: 22.798469345843053 + - type: nauc_recall_at_1_std + value: -8.116505032590933 + - type: nauc_recall_at_20_diff1 + value: 39.45484845593912 + - type: nauc_recall_at_20_max + value: 27.549514718986806 + - type: nauc_recall_at_20_std + value: 14.313387585153405 + - type: nauc_recall_at_3_diff1 + value: 44.0557079332108 + - type: nauc_recall_at_3_max + value: 21.89712965509793 + - type: nauc_recall_at_3_std + value: -4.486798859251415 + - type: nauc_recall_at_5_diff1 + value: 44.25967763312861 + - type: nauc_recall_at_5_max + value: 26.64082836717858 + - type: nauc_recall_at_5_std + value: 0.8448093114664144 + - type: ndcg_at_1 + value: 45.6 + - type: ndcg_at_10 + value: 47.259 + - type: ndcg_at_100 + value: 53.406 + - type: ndcg_at_1000 + value: 55.82 + - type: ndcg_at_20 + value: 49.784 + - type: ndcg_at_3 + value: 43.996 + - type: ndcg_at_5 + value: 44.686 + - type: precision_at_1 + value: 45.6 + - type: precision_at_10 + value: 12.22 + - type: precision_at_100 + value: 1.8679999999999999 + - type: precision_at_1000 + value: 0.22699999999999998 + - type: precision_at_20 + value: 7.22 + - type: precision_at_3 + value: 27.866999999999997 + - type: precision_at_5 + value: 19.84 + - type: recall_at_1 + value: 25.491999999999997 + - type: recall_at_10 + value: 54.521 + - type: recall_at_100 + value: 77.029 + - type: recall_at_1000 + value: 91.62400000000001 + - type: recall_at_20 + value: 62.397000000000006 + - type: recall_at_3 + value: 41.128 + - type: recall_at_5 + value: 46.609 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FiQA2018 (default) + revision: 27a168819829fe9bcd655c2df245fb19452e8e06 + split: test + type: mteb/fiqa + metrics: + - type: main_score + value: 45.414 + - type: map_at_1 + value: 23.639 + - type: map_at_10 + value: 37.714 + - type: map_at_100 + value: 39.649 + - type: map_at_1000 + value: 39.83 + - type: map_at_20 + value: 38.839 + - type: map_at_3 + value: 33.318 + - type: map_at_5 + value: 35.702 + - type: mrr_at_1 + value: 45.67901234567901 + - type: mrr_at_10 + value: 53.80787037037037 + - type: mrr_at_100 + value: 54.57292395037695 + - type: mrr_at_1000 + value: 54.60732599969202 + - type: mrr_at_20 + value: 54.29808731528356 + - type: mrr_at_3 + value: 51.23456790123457 + - type: mrr_at_5 + value: 52.70061728395061 + - type: nauc_map_at_1000_diff1 + value: 40.79693555111124 + - type: nauc_map_at_1000_max + value: 31.93826160734657 + - type: nauc_map_at_1000_std + value: 3.8184369075253506 + - type: nauc_map_at_100_diff1 + value: 40.70327893882628 + - type: nauc_map_at_100_max + value: 31.84401788672614 + - type: nauc_map_at_100_std + value: 3.8218534829376116 + - type: nauc_map_at_10_diff1 + value: 40.46730792783253 + - type: nauc_map_at_10_max + value: 30.465616503734296 + - type: nauc_map_at_10_std + value: 2.3830531439591813 + - type: nauc_map_at_1_diff1 + value: 45.25513310331309 + - type: nauc_map_at_1_max + value: 24.33267518396139 + - type: nauc_map_at_1_std + value: 0.058189822716279094 + - type: nauc_map_at_20_diff1 + value: 40.49922197626706 + - type: nauc_map_at_20_max + value: 31.384589763567554 + - type: nauc_map_at_20_std + value: 3.3254300279996887 + - type: nauc_map_at_3_diff1 + value: 41.275011370302664 + - type: nauc_map_at_3_max + value: 26.67085750744137 + - type: nauc_map_at_3_std + value: -0.14108246056717116 + - type: nauc_map_at_5_diff1 + value: 41.32360383192518 + - type: nauc_map_at_5_max + value: 29.479218102656574 + - type: nauc_map_at_5_std + value: 1.5963967229908553 + - type: nauc_mrr_at_1000_diff1 + value: 53.54649350159304 + - type: nauc_mrr_at_1000_max + value: 41.12026635845684 + - type: nauc_mrr_at_1000_std + value: 6.013167900259446 + - type: nauc_mrr_at_100_diff1 + value: 53.52118112726805 + - type: nauc_mrr_at_100_max + value: 41.128658883492825 + - type: nauc_mrr_at_100_std + value: 6.043033929594675 + - type: nauc_mrr_at_10_diff1 + value: 53.3548135763686 + - type: nauc_mrr_at_10_max + value: 40.793671799611616 + - type: nauc_mrr_at_10_std + value: 5.399733307841459 + - type: nauc_mrr_at_1_diff1 + value: 56.51206647461513 + - type: nauc_mrr_at_1_max + value: 41.0802790143328 + - type: nauc_mrr_at_1_std + value: 4.82373453507849 + - type: nauc_mrr_at_20_diff1 + value: 53.42197135923319 + - type: nauc_mrr_at_20_max + value: 41.04568171837774 + - type: nauc_mrr_at_20_std + value: 5.940443558911226 + - type: nauc_mrr_at_3_diff1 + value: 54.08570617279002 + - type: nauc_mrr_at_3_max + value: 40.6764139846654 + - type: nauc_mrr_at_3_std + value: 4.799356031469672 + - type: nauc_mrr_at_5_diff1 + value: 53.80809476676707 + - type: nauc_mrr_at_5_max + value: 40.91131831488185 + - type: nauc_mrr_at_5_std + value: 5.384337462838449 + - type: nauc_ndcg_at_1000_diff1 + value: 44.40936042120287 + - type: nauc_ndcg_at_1000_max + value: 37.36720837647671 + - type: nauc_ndcg_at_1000_std + value: 8.469923413119727 + - type: nauc_ndcg_at_100_diff1 + value: 43.13653122471379 + - type: nauc_ndcg_at_100_max + value: 36.493253754425254 + - type: nauc_ndcg_at_100_std + value: 9.056782965364897 + - type: nauc_ndcg_at_10_diff1 + value: 42.09654106679639 + - type: nauc_ndcg_at_10_max + value: 33.240520203888835 + - type: nauc_ndcg_at_10_std + value: 4.007667430079706 + - type: nauc_ndcg_at_1_diff1 + value: 56.51206647461513 + - type: nauc_ndcg_at_1_max + value: 41.0802790143328 + - type: nauc_ndcg_at_1_std + value: 4.82373453507849 + - type: nauc_ndcg_at_20_diff1 + value: 42.31585208388624 + - type: nauc_ndcg_at_20_max + value: 34.925001463841596 + - type: nauc_ndcg_at_20_std + value: 6.57677059528588 + - type: nauc_ndcg_at_3_diff1 + value: 43.11230344169659 + - type: nauc_ndcg_at_3_max + value: 33.371126631700236 + - type: nauc_ndcg_at_3_std + value: 2.6157697075832194 + - type: nauc_ndcg_at_5_diff1 + value: 43.08901943201737 + - type: nauc_ndcg_at_5_max + value: 33.551785743184034 + - type: nauc_ndcg_at_5_std + value: 3.5514401825959503 + - type: nauc_precision_at_1000_diff1 + value: 2.0038085725274124 + - type: nauc_precision_at_1000_max + value: 23.892791793393855 + - type: nauc_precision_at_1000_std + value: 15.576002858560829 + - type: nauc_precision_at_100_diff1 + value: 6.758228283812937 + - type: nauc_precision_at_100_max + value: 29.143629344714544 + - type: nauc_precision_at_100_std + value: 19.276601462356844 + - type: nauc_precision_at_10_diff1 + value: 16.68790806980712 + - type: nauc_precision_at_10_max + value: 33.57227603366303 + - type: nauc_precision_at_10_std + value: 10.916847025085708 + - type: nauc_precision_at_1_diff1 + value: 56.51206647461513 + - type: nauc_precision_at_1_max + value: 41.0802790143328 + - type: nauc_precision_at_1_std + value: 4.82373453507849 + - type: nauc_precision_at_20_diff1 + value: 12.805224266236184 + - type: nauc_precision_at_20_max + value: 32.95072251134881 + - type: nauc_precision_at_20_std + value: 15.567649053765525 + - type: nauc_precision_at_3_diff1 + value: 29.79457909141301 + - type: nauc_precision_at_3_max + value: 34.100479917557955 + - type: nauc_precision_at_3_std + value: 4.8228489191687 + - type: nauc_precision_at_5_diff1 + value: 24.780101003334558 + - type: nauc_precision_at_5_max + value: 35.38026352988683 + - type: nauc_precision_at_5_std + value: 8.251451137390243 + - type: nauc_recall_at_1000_diff1 + value: 43.45779284015868 + - type: nauc_recall_at_1000_max + value: 47.6560075930806 + - type: nauc_recall_at_1000_std + value: 43.754965098108165 + - type: nauc_recall_at_100_diff1 + value: 28.173230312706366 + - type: nauc_recall_at_100_max + value: 32.85300138122667 + - type: nauc_recall_at_100_std + value: 28.345517496198084 + - type: nauc_recall_at_10_diff1 + value: 30.857388512991218 + - type: nauc_recall_at_10_max + value: 25.911017012295684 + - type: nauc_recall_at_10_std + value: 4.1564118591506425 + - type: nauc_recall_at_1_diff1 + value: 45.25513310331309 + - type: nauc_recall_at_1_max + value: 24.33267518396139 + - type: nauc_recall_at_1_std + value: 0.058189822716279094 + - type: nauc_recall_at_20_diff1 + value: 30.114352125861572 + - type: nauc_recall_at_20_max + value: 30.042878295078072 + - type: nauc_recall_at_20_std + value: 12.418600151564057 + - type: nauc_recall_at_3_diff1 + value: 36.372282024548255 + - type: nauc_recall_at_3_max + value: 23.2065904151668 + - type: nauc_recall_at_3_std + value: -0.7920546172242813 + - type: nauc_recall_at_5_diff1 + value: 35.60442261227824 + - type: nauc_recall_at_5_max + value: 26.769767553883923 + - type: nauc_recall_at_5_std + value: 3.173463160439195 + - type: ndcg_at_1 + value: 45.678999999999995 + - type: ndcg_at_10 + value: 45.414 + - type: ndcg_at_100 + value: 52.09 + - type: ndcg_at_1000 + value: 54.854 + - type: ndcg_at_20 + value: 48.156 + - type: ndcg_at_3 + value: 41.813 + - type: ndcg_at_5 + value: 42.794 + - type: precision_at_1 + value: 45.678999999999995 + - type: precision_at_10 + value: 12.315 + - type: precision_at_100 + value: 1.917 + - type: precision_at_1000 + value: 0.241 + - type: precision_at_20 + value: 7.361 + - type: precision_at_3 + value: 27.366 + - type: precision_at_5 + value: 20.093 + - type: recall_at_1 + value: 23.639 + - type: recall_at_10 + value: 51.517 + - type: recall_at_100 + value: 76.151 + - type: recall_at_1000 + value: 92.14099999999999 + - type: recall_at_20 + value: 59.782999999999994 + - type: recall_at_3 + value: 37.535000000000004 + - type: recall_at_5 + value: 43.181000000000004 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FiQA2018 (default) + revision: 27a168819829fe9bcd655c2df245fb19452e8e06 + split: train + type: mteb/fiqa + metrics: + - type: main_score + value: 46.587 + - type: map_at_1 + value: 24.194 + - type: map_at_10 + value: 38.505 + - type: map_at_100 + value: 40.35 + - type: map_at_1000 + value: 40.513 + - type: map_at_20 + value: 39.548 + - type: map_at_3 + value: 33.888 + - type: map_at_5 + value: 36.370999999999995 + - type: mrr_at_1 + value: 44.85454545454545 + - type: mrr_at_10 + value: 53.74769841269854 + - type: mrr_at_100 + value: 54.45303445294334 + - type: mrr_at_1000 + value: 54.48781203381574 + - type: mrr_at_20 + value: 54.17068368187784 + - type: mrr_at_3 + value: 51.275757575757716 + - type: mrr_at_5 + value: 52.71484848484853 + - type: nauc_map_at_1000_diff1 + value: 45.59497741451223 + - type: nauc_map_at_1000_max + value: 29.29369747229761 + - type: nauc_map_at_1000_std + value: 1.7760673707937762 + - type: nauc_map_at_100_diff1 + value: 45.58884528458884 + - type: nauc_map_at_100_max + value: 29.244033513094276 + - type: nauc_map_at_100_std + value: 1.7313619143262027 + - type: nauc_map_at_10_diff1 + value: 45.37380105017834 + - type: nauc_map_at_10_max + value: 28.222088708674164 + - type: nauc_map_at_10_std + value: 0.35078417002628504 + - type: nauc_map_at_1_diff1 + value: 48.69936193448306 + - type: nauc_map_at_1_max + value: 19.086854504806546 + - type: nauc_map_at_1_std + value: -4.76102930260662 + - type: nauc_map_at_20_diff1 + value: 45.442731723960286 + - type: nauc_map_at_20_max + value: 28.78189737953754 + - type: nauc_map_at_20_std + value: 1.1443897851780256 + - type: nauc_map_at_3_diff1 + value: 45.66389119353029 + - type: nauc_map_at_3_max + value: 24.8929203407857 + - type: nauc_map_at_3_std + value: -2.9734539769983295 + - type: nauc_map_at_5_diff1 + value: 45.332909024668645 + - type: nauc_map_at_5_max + value: 26.572901966592838 + - type: nauc_map_at_5_std + value: -1.4890802294775731 + - type: nauc_mrr_at_1000_diff1 + value: 52.24905947258471 + - type: nauc_mrr_at_1000_max + value: 36.68992514726046 + - type: nauc_mrr_at_1000_std + value: 4.036175007509102 + - type: nauc_mrr_at_100_diff1 + value: 52.24111877954976 + - type: nauc_mrr_at_100_max + value: 36.69904806662672 + - type: nauc_mrr_at_100_std + value: 4.057041685352858 + - type: nauc_mrr_at_10_diff1 + value: 52.20317475120265 + - type: nauc_mrr_at_10_max + value: 36.73687859138565 + - type: nauc_mrr_at_10_std + value: 3.9791960914853965 + - type: nauc_mrr_at_1_diff1 + value: 55.67342562037457 + - type: nauc_mrr_at_1_max + value: 35.42443099163554 + - type: nauc_mrr_at_1_std + value: 0.5448839756707202 + - type: nauc_mrr_at_20_diff1 + value: 52.18210662127548 + - type: nauc_mrr_at_20_max + value: 36.711107440726536 + - type: nauc_mrr_at_20_std + value: 4.058202442757198 + - type: nauc_mrr_at_3_diff1 + value: 52.26160206801409 + - type: nauc_mrr_at_3_max + value: 36.205799065502866 + - type: nauc_mrr_at_3_std + value: 2.9681462087067696 + - type: nauc_mrr_at_5_diff1 + value: 52.247034238031695 + - type: nauc_mrr_at_5_max + value: 36.51493166384178 + - type: nauc_mrr_at_5_std + value: 3.570981650048746 + - type: nauc_ndcg_at_1000_diff1 + value: 46.50975125927248 + - type: nauc_ndcg_at_1000_max + value: 33.853860913478904 + - type: nauc_ndcg_at_1000_std + value: 6.660306015939721 + - type: nauc_ndcg_at_100_diff1 + value: 46.27042408923828 + - type: nauc_ndcg_at_100_max + value: 33.607892772154344 + - type: nauc_ndcg_at_100_std + value: 7.008572795576808 + - type: nauc_ndcg_at_10_diff1 + value: 45.59321855035331 + - type: nauc_ndcg_at_10_max + value: 31.539072784071976 + - type: nauc_ndcg_at_10_std + value: 3.6682407658236604 + - type: nauc_ndcg_at_1_diff1 + value: 55.67342562037457 + - type: nauc_ndcg_at_1_max + value: 35.42443099163554 + - type: nauc_ndcg_at_1_std + value: 0.5448839756707202 + - type: nauc_ndcg_at_20_diff1 + value: 45.65487959293365 + - type: nauc_ndcg_at_20_max + value: 32.121270115512445 + - type: nauc_ndcg_at_20_std + value: 4.895201727104387 + - type: nauc_ndcg_at_3_diff1 + value: 46.01634177986288 + - type: nauc_ndcg_at_3_max + value: 31.039722303081156 + - type: nauc_ndcg_at_3_std + value: 1.2634852059305737 + - type: nauc_ndcg_at_5_diff1 + value: 45.39242013370225 + - type: nauc_ndcg_at_5_max + value: 30.3812965425671 + - type: nauc_ndcg_at_5_std + value: 1.819391722091453 + - type: nauc_precision_at_1000_diff1 + value: -1.5590692190860904 + - type: nauc_precision_at_1000_max + value: 20.979544329360568 + - type: nauc_precision_at_1000_std + value: 19.519200019393033 + - type: nauc_precision_at_100_diff1 + value: 6.997385462318145 + - type: nauc_precision_at_100_max + value: 28.698708128988503 + - type: nauc_precision_at_100_std + value: 23.97830418343215 + - type: nauc_precision_at_10_diff1 + value: 19.389059235749873 + - type: nauc_precision_at_10_max + value: 34.079184666167414 + - type: nauc_precision_at_10_std + value: 16.574183149641954 + - type: nauc_precision_at_1_diff1 + value: 55.67342562037457 + - type: nauc_precision_at_1_max + value: 35.42443099163554 + - type: nauc_precision_at_1_std + value: 0.5448839756707202 + - type: nauc_precision_at_20_diff1 + value: 14.913591644410861 + - type: nauc_precision_at_20_max + value: 32.50938357784388 + - type: nauc_precision_at_20_std + value: 19.850776471349782 + - type: nauc_precision_at_3_diff1 + value: 31.38779333263097 + - type: nauc_precision_at_3_max + value: 35.03244136415842 + - type: nauc_precision_at_3_std + value: 6.719699147068504 + - type: nauc_precision_at_5_diff1 + value: 25.30837187699263 + - type: nauc_precision_at_5_max + value: 34.590856041914606 + - type: nauc_precision_at_5_std + value: 10.788586397125274 + - type: nauc_recall_at_1000_diff1 + value: 25.821877997412066 + - type: nauc_recall_at_1000_max + value: 36.21259453607436 + - type: nauc_recall_at_1000_std + value: 39.93155434096384 + - type: nauc_recall_at_100_diff1 + value: 32.050903304097766 + - type: nauc_recall_at_100_max + value: 31.360916971322116 + - type: nauc_recall_at_100_std + value: 23.22588972952997 + - type: nauc_recall_at_10_diff1 + value: 35.05124064290814 + - type: nauc_recall_at_10_max + value: 26.73964977315679 + - type: nauc_recall_at_10_std + value: 6.740995331893972 + - type: nauc_recall_at_1_diff1 + value: 48.69936193448306 + - type: nauc_recall_at_1_max + value: 19.086854504806546 + - type: nauc_recall_at_1_std + value: -4.76102930260662 + - type: nauc_recall_at_20_diff1 + value: 33.3672021367699 + - type: nauc_recall_at_20_max + value: 27.043596259272075 + - type: nauc_recall_at_20_std + value: 10.64844704773122 + - type: nauc_recall_at_3_diff1 + value: 39.249554420545245 + - type: nauc_recall_at_3_max + value: 22.93148009562249 + - type: nauc_recall_at_3_std + value: -1.52369678059749 + - type: nauc_recall_at_5_diff1 + value: 36.93076382798328 + - type: nauc_recall_at_5_max + value: 23.938693625294675 + - type: nauc_recall_at_5_std + value: 1.2542630444480962 + - type: ndcg_at_1 + value: 44.855000000000004 + - type: ndcg_at_10 + value: 46.587 + - type: ndcg_at_100 + value: 52.864 + - type: ndcg_at_1000 + value: 55.472 + - type: ndcg_at_20 + value: 49.053999999999995 + - type: ndcg_at_3 + value: 42.335 + - type: ndcg_at_5 + value: 43.693 + - type: precision_at_1 + value: 44.855000000000004 + - type: precision_at_10 + value: 12.427000000000001 + - type: precision_at_100 + value: 1.905 + - type: precision_at_1000 + value: 0.23600000000000002 + - type: precision_at_20 + value: 7.295999999999999 + - type: precision_at_3 + value: 27.442 + - type: precision_at_5 + value: 20.003999999999998 + - type: recall_at_1 + value: 24.194 + - type: recall_at_10 + value: 54.31399999999999 + - type: recall_at_100 + value: 77.653 + - type: recall_at_1000 + value: 93.31099999999999 + - type: recall_at_20 + value: 62.022999999999996 + - type: recall_at_3 + value: 39.163 + - type: recall_at_5 + value: 45.541 + task: + type: Retrieval + - dataset: + config: default + name: MTEB HotpotQA (default) + revision: ab518f4d6fcca38d87c25209f94beba119d02014 + split: dev + type: mteb/hotpotqa + metrics: + - type: main_score + value: 78.594 + - type: map_at_1 + value: 43.143 + - type: map_at_10 + value: 71.378 + - type: map_at_100 + value: 72.136 + - type: map_at_1000 + value: 72.18299999999999 + - type: map_at_20 + value: 71.851 + - type: map_at_3 + value: 68.27499999999999 + - type: map_at_5 + value: 70.229 + - type: mrr_at_1 + value: 86.28602900679273 + - type: mrr_at_10 + value: 90.57545874968305 + - type: mrr_at_100 + value: 90.66400671194192 + - type: mrr_at_1000 + value: 90.66560423144004 + - type: mrr_at_20 + value: 90.63388375183823 + - type: mrr_at_3 + value: 90.00979132244048 + - type: mrr_at_5 + value: 90.38339146931027 + - type: nauc_map_at_1000_diff1 + value: 9.388430489926671 + - type: nauc_map_at_1000_max + value: 30.585113457632847 + - type: nauc_map_at_1000_std + value: 23.04278851501047 + - type: nauc_map_at_100_diff1 + value: 9.356444463077015 + - type: nauc_map_at_100_max + value: 30.58814044634665 + - type: nauc_map_at_100_std + value: 23.07560185915737 + - type: nauc_map_at_10_diff1 + value: 8.877530965043936 + - type: nauc_map_at_10_max + value: 30.366402397278875 + - type: nauc_map_at_10_std + value: 22.833144668339592 + - type: nauc_map_at_1_diff1 + value: 70.06248041256521 + - type: nauc_map_at_1_max + value: 48.0800497365783 + - type: nauc_map_at_1_std + value: 4.864111306341327 + - type: nauc_map_at_20_diff1 + value: 9.223705261670693 + - type: nauc_map_at_20_max + value: 30.560650634461922 + - type: nauc_map_at_20_std + value: 23.029467586071004 + - type: nauc_map_at_3_diff1 + value: 8.882099696768105 + - type: nauc_map_at_3_max + value: 28.633735347192648 + - type: nauc_map_at_3_std + value: 18.91510493931095 + - type: nauc_map_at_5_diff1 + value: 9.1266109061697 + - type: nauc_map_at_5_max + value: 29.856683878528745 + - type: nauc_map_at_5_std + value: 21.556900429541464 + - type: nauc_mrr_at_1000_diff1 + value: 69.56709020548416 + - type: nauc_mrr_at_1000_max + value: 51.801789590600734 + - type: nauc_mrr_at_1000_std + value: 8.654124094099393 + - type: nauc_mrr_at_100_diff1 + value: 69.56704937674587 + - type: nauc_mrr_at_100_max + value: 51.807968550977044 + - type: nauc_mrr_at_100_std + value: 8.66028971665739 + - type: nauc_mrr_at_10_diff1 + value: 69.58545319727206 + - type: nauc_mrr_at_10_max + value: 51.91765539257316 + - type: nauc_mrr_at_10_std + value: 8.76708974052272 + - type: nauc_mrr_at_1_diff1 + value: 70.06248041256521 + - type: nauc_mrr_at_1_max + value: 48.0800497365783 + - type: nauc_mrr_at_1_std + value: 4.864111306341327 + - type: nauc_mrr_at_20_diff1 + value: 69.51864854234509 + - type: nauc_mrr_at_20_max + value: 51.86043525767574 + - type: nauc_mrr_at_20_std + value: 8.779547981143692 + - type: nauc_mrr_at_3_diff1 + value: 69.54615016831823 + - type: nauc_mrr_at_3_max + value: 52.17745956421602 + - type: nauc_mrr_at_3_std + value: 7.935204865903847 + - type: nauc_mrr_at_5_diff1 + value: 69.69710713238221 + - type: nauc_mrr_at_5_max + value: 52.08654294666856 + - type: nauc_mrr_at_5_std + value: 8.654083438282056 + - type: nauc_ndcg_at_1000_diff1 + value: 16.242605765688694 + - type: nauc_ndcg_at_1000_max + value: 35.07165720958159 + - type: nauc_ndcg_at_1000_std + value: 25.67569052091955 + - type: nauc_ndcg_at_100_diff1 + value: 15.187960766782545 + - type: nauc_ndcg_at_100_max + value: 35.04207977681353 + - type: nauc_ndcg_at_100_std + value: 26.641180659185228 + - type: nauc_ndcg_at_10_diff1 + value: 13.437171988178745 + - type: nauc_ndcg_at_10_max + value: 34.25673377763555 + - type: nauc_ndcg_at_10_std + value: 25.7326001239102 + - type: nauc_ndcg_at_1_diff1 + value: 70.06248041256521 + - type: nauc_ndcg_at_1_max + value: 48.0800497365783 + - type: nauc_ndcg_at_1_std + value: 4.864111306341327 + - type: nauc_ndcg_at_20_diff1 + value: 14.329469941566877 + - type: nauc_ndcg_at_20_max + value: 34.84766744310814 + - type: nauc_ndcg_at_20_std + value: 26.494826227161145 + - type: nauc_ndcg_at_3_diff1 + value: 14.144945529516598 + - type: nauc_ndcg_at_3_max + value: 31.862175768992007 + - type: nauc_ndcg_at_3_std + value: 19.26728699023659 + - type: nauc_ndcg_at_5_diff1 + value: 14.174567290867287 + - type: nauc_ndcg_at_5_max + value: 33.40724910369022 + - type: nauc_ndcg_at_5_std + value: 23.181873724075082 + - type: nauc_precision_at_1000_diff1 + value: -2.954891416976416 + - type: nauc_precision_at_1000_max + value: 42.184537786000774 + - type: nauc_precision_at_1000_std + value: 66.64900526421333 + - type: nauc_precision_at_100_diff1 + value: -4.161796796363399 + - type: nauc_precision_at_100_max + value: 36.37411208428507 + - type: nauc_precision_at_100_std + value: 53.17437800549103 + - type: nauc_precision_at_10_diff1 + value: -2.3517941676669323 + - type: nauc_precision_at_10_max + value: 33.45090205320283 + - type: nauc_precision_at_10_std + value: 39.094125569808504 + - type: nauc_precision_at_1_diff1 + value: 70.06248041256521 + - type: nauc_precision_at_1_max + value: 48.0800497365783 + - type: nauc_precision_at_1_std + value: 4.864111306341327 + - type: nauc_precision_at_20_diff1 + value: -1.987456332062962 + - type: nauc_precision_at_20_max + value: 35.02025399087546 + - type: nauc_precision_at_20_std + value: 43.9516095251227 + - type: nauc_precision_at_3_diff1 + value: 3.3504605594055192 + - type: nauc_precision_at_3_max + value: 29.700751073309668 + - type: nauc_precision_at_3_std + value: 23.621821011080876 + - type: nauc_precision_at_5_diff1 + value: 1.8873559078558009 + - type: nauc_precision_at_5_max + value: 31.87602995170986 + - type: nauc_precision_at_5_std + value: 31.31868467173099 + - type: nauc_recall_at_1000_diff1 + value: -2.9548914169769955 + - type: nauc_recall_at_1000_max + value: 42.184537786000284 + - type: nauc_recall_at_1000_std + value: 66.64900526421359 + - type: nauc_recall_at_100_diff1 + value: -4.161796796363361 + - type: nauc_recall_at_100_max + value: 36.37411208428513 + - type: nauc_recall_at_100_std + value: 53.174378005490794 + - type: nauc_recall_at_10_diff1 + value: -2.35179416766723 + - type: nauc_recall_at_10_max + value: 33.45090205320268 + - type: nauc_recall_at_10_std + value: 39.094125569808305 + - type: nauc_recall_at_1_diff1 + value: 70.06248041256521 + - type: nauc_recall_at_1_max + value: 48.0800497365783 + - type: nauc_recall_at_1_std + value: 4.864111306341327 + - type: nauc_recall_at_20_diff1 + value: -1.9874563320631378 + - type: nauc_recall_at_20_max + value: 35.020253990875204 + - type: nauc_recall_at_20_std + value: 43.95160952512256 + - type: nauc_recall_at_3_diff1 + value: 3.3504605594055166 + - type: nauc_recall_at_3_max + value: 29.70075107330964 + - type: nauc_recall_at_3_std + value: 23.621821011080808 + - type: nauc_recall_at_5_diff1 + value: 1.8873559078557478 + - type: nauc_recall_at_5_max + value: 31.8760299517097 + - type: nauc_recall_at_5_std + value: 31.318684671730807 + - type: ndcg_at_1 + value: 86.286 + - type: ndcg_at_10 + value: 78.594 + - type: ndcg_at_100 + value: 81.00699999999999 + - type: ndcg_at_1000 + value: 81.855 + - type: ndcg_at_20 + value: 79.702 + - type: ndcg_at_3 + value: 74.417 + - type: ndcg_at_5 + value: 76.775 + - type: precision_at_1 + value: 86.286 + - type: precision_at_10 + value: 16.293 + - type: precision_at_100 + value: 1.814 + - type: precision_at_1000 + value: 0.193 + - type: precision_at_20 + value: 8.502 + - type: precision_at_3 + value: 48.186 + - type: precision_at_5 + value: 30.773 + - type: recall_at_1 + value: 43.143 + - type: recall_at_10 + value: 81.467 + - type: recall_at_100 + value: 90.71000000000001 + - type: recall_at_1000 + value: 96.26400000000001 + - type: recall_at_20 + value: 85.019 + - type: recall_at_3 + value: 72.27799999999999 + - type: recall_at_5 + value: 76.932 + task: + type: Retrieval + - dataset: + config: default + name: MTEB HotpotQA (default) + revision: ab518f4d6fcca38d87c25209f94beba119d02014 + split: test + type: mteb/hotpotqa + metrics: + - type: main_score + value: 76.875 + - type: map_at_1 + value: 42.269 + - type: map_at_10 + value: 69.292 + - type: map_at_100 + value: 70.114 + - type: map_at_1000 + value: 70.16 + - type: map_at_20 + value: 69.796 + - type: map_at_3 + value: 65.935 + - type: map_at_5 + value: 68.123 + - type: mrr_at_1 + value: 84.53747467927076 + - type: mrr_at_10 + value: 89.33357448313538 + - type: mrr_at_100 + value: 89.43144514614639 + - type: mrr_at_1000 + value: 89.43439453618649 + - type: mrr_at_20 + value: 89.3957836961129 + - type: mrr_at_3 + value: 88.69232500562664 + - type: mrr_at_5 + value: 89.11163628179128 + - type: nauc_map_at_1000_diff1 + value: 8.109703373251145 + - type: nauc_map_at_1000_max + value: 23.994888983388254 + - type: nauc_map_at_1000_std + value: 17.121601939043398 + - type: nauc_map_at_100_diff1 + value: 8.08595887114707 + - type: nauc_map_at_100_max + value: 23.998897104654144 + - type: nauc_map_at_100_std + value: 17.163165111294518 + - type: nauc_map_at_10_diff1 + value: 7.684758341960099 + - type: nauc_map_at_10_max + value: 23.724799190953018 + - type: nauc_map_at_10_std + value: 16.68447774854757 + - type: nauc_map_at_1_diff1 + value: 66.81583059408007 + - type: nauc_map_at_1_max + value: 44.99007039930331 + - type: nauc_map_at_1_std + value: 1.9672750979981743 + - type: nauc_map_at_20_diff1 + value: 7.889868542104006 + - type: nauc_map_at_20_max + value: 23.842069475755636 + - type: nauc_map_at_20_std + value: 17.00465387238006 + - type: nauc_map_at_3_diff1 + value: 8.23318084259305 + - type: nauc_map_at_3_max + value: 22.041544528023994 + - type: nauc_map_at_3_std + value: 13.414893323049728 + - type: nauc_map_at_5_diff1 + value: 7.439725925625305 + - type: nauc_map_at_5_max + value: 23.33003483895132 + - type: nauc_map_at_5_std + value: 15.753908206597092 + - type: nauc_mrr_at_1000_diff1 + value: 66.15955168618606 + - type: nauc_mrr_at_1000_max + value: 48.56436483917718 + - type: nauc_mrr_at_1000_std + value: 4.474538780987628 + - type: nauc_mrr_at_100_diff1 + value: 66.16502155887812 + - type: nauc_mrr_at_100_max + value: 48.57268936821578 + - type: nauc_mrr_at_100_std + value: 4.4830649246242205 + - type: nauc_mrr_at_10_diff1 + value: 66.2089647691383 + - type: nauc_mrr_at_10_max + value: 48.75908726580652 + - type: nauc_mrr_at_10_std + value: 4.616333858518716 + - type: nauc_mrr_at_1_diff1 + value: 66.81583059408007 + - type: nauc_mrr_at_1_max + value: 44.99007039930331 + - type: nauc_mrr_at_1_std + value: 1.9672750979981743 + - type: nauc_mrr_at_20_diff1 + value: 66.16404221325158 + - type: nauc_mrr_at_20_max + value: 48.62361247765342 + - type: nauc_mrr_at_20_std + value: 4.530845984725845 + - type: nauc_mrr_at_3_diff1 + value: 65.92961125630885 + - type: nauc_mrr_at_3_max + value: 48.99162797399524 + - type: nauc_mrr_at_3_std + value: 4.4598293131250335 + - type: nauc_mrr_at_5_diff1 + value: 66.19071575749183 + - type: nauc_mrr_at_5_max + value: 49.09480826369942 + - type: nauc_mrr_at_5_std + value: 4.65021545718115 + - type: nauc_ndcg_at_1000_diff1 + value: 14.738756964862642 + - type: nauc_ndcg_at_1000_max + value: 28.933401252638998 + - type: nauc_ndcg_at_1000_std + value: 19.53690398894528 + - type: nauc_ndcg_at_100_diff1 + value: 13.925580127074463 + - type: nauc_ndcg_at_100_max + value: 28.916986648599526 + - type: nauc_ndcg_at_100_std + value: 20.560721835265138 + - type: nauc_ndcg_at_10_diff1 + value: 12.27023264472293 + - type: nauc_ndcg_at_10_max + value: 27.625796820636772 + - type: nauc_ndcg_at_10_std + value: 18.621751232958527 + - type: nauc_ndcg_at_1_diff1 + value: 66.81583059408007 + - type: nauc_ndcg_at_1_max + value: 44.99007039930331 + - type: nauc_ndcg_at_1_std + value: 1.9672750979981743 + - type: nauc_ndcg_at_20_diff1 + value: 12.796824428751854 + - type: nauc_ndcg_at_20_max + value: 27.94600059457669 + - type: nauc_ndcg_at_20_std + value: 19.56124604706208 + - type: nauc_ndcg_at_3_diff1 + value: 13.497928598266885 + - type: nauc_ndcg_at_3_max + value: 25.512659339160404 + - type: nauc_ndcg_at_3_std + value: 13.492408305021456 + - type: nauc_ndcg_at_5_diff1 + value: 12.129094782065444 + - type: nauc_ndcg_at_5_max + value: 27.09821685754243 + - type: nauc_ndcg_at_5_std + value: 16.749664591916023 + - type: nauc_precision_at_1000_diff1 + value: -10.482006811274527 + - type: nauc_precision_at_1000_max + value: 35.47844891285956 + - type: nauc_precision_at_1000_std + value: 60.3055227587308 + - type: nauc_precision_at_100_diff1 + value: -6.4516803404104 + - type: nauc_precision_at_100_max + value: 30.571341321897126 + - type: nauc_precision_at_100_std + value: 48.83583502335811 + - type: nauc_precision_at_10_diff1 + value: -4.134290594596903 + - type: nauc_precision_at_10_max + value: 24.50731126991085 + - type: nauc_precision_at_10_std + value: 28.90753659067843 + - type: nauc_precision_at_1_diff1 + value: 66.81583059408007 + - type: nauc_precision_at_1_max + value: 44.99007039930331 + - type: nauc_precision_at_1_std + value: 1.9672750979981743 + - type: nauc_precision_at_20_diff1 + value: -4.9287309601836515 + - type: nauc_precision_at_20_max + value: 24.973388438210566 + - type: nauc_precision_at_20_std + value: 33.837584207454015 + - type: nauc_precision_at_3_diff1 + value: 1.9725862462476325 + - type: nauc_precision_at_3_max + value: 21.78580614273071 + - type: nauc_precision_at_3_std + value: 16.830358454546698 + - type: nauc_precision_at_5_diff1 + value: -2.1036896627637374 + - type: nauc_precision_at_5_max + value: 23.95905039644115 + - type: nauc_precision_at_5_std + value: 23.158073886809635 + - type: nauc_recall_at_1000_diff1 + value: -10.482006811273724 + - type: nauc_recall_at_1000_max + value: 35.478448912859825 + - type: nauc_recall_at_1000_std + value: 60.30552275873128 + - type: nauc_recall_at_100_diff1 + value: -6.4516803404104595 + - type: nauc_recall_at_100_max + value: 30.57134132189695 + - type: nauc_recall_at_100_std + value: 48.83583502335814 + - type: nauc_recall_at_10_diff1 + value: -4.134290594596836 + - type: nauc_recall_at_10_max + value: 24.507311269910964 + - type: nauc_recall_at_10_std + value: 28.907536590678358 + - type: nauc_recall_at_1_diff1 + value: 66.81583059408007 + - type: nauc_recall_at_1_max + value: 44.99007039930331 + - type: nauc_recall_at_1_std + value: 1.9672750979981743 + - type: nauc_recall_at_20_diff1 + value: -4.92873096018357 + - type: nauc_recall_at_20_max + value: 24.97338843821055 + - type: nauc_recall_at_20_std + value: 33.837584207454064 + - type: nauc_recall_at_3_diff1 + value: 1.9725862462476491 + - type: nauc_recall_at_3_max + value: 21.785806142730692 + - type: nauc_recall_at_3_std + value: 16.83035845454663 + - type: nauc_recall_at_5_diff1 + value: -2.1036896627637502 + - type: nauc_recall_at_5_max + value: 23.95905039644112 + - type: nauc_recall_at_5_std + value: 23.15807388680964 + - type: ndcg_at_1 + value: 84.53699999999999 + - type: ndcg_at_10 + value: 76.875 + - type: ndcg_at_100 + value: 79.549 + - type: ndcg_at_1000 + value: 80.39 + - type: ndcg_at_20 + value: 78.057 + - type: ndcg_at_3 + value: 72.356 + - type: ndcg_at_5 + value: 74.988 + - type: precision_at_1 + value: 84.53699999999999 + - type: precision_at_10 + value: 15.998999999999999 + - type: precision_at_100 + value: 1.807 + - type: precision_at_1000 + value: 0.192 + - type: precision_at_20 + value: 8.378 + - type: precision_at_3 + value: 46.721000000000004 + - type: precision_at_5 + value: 30.104 + - type: recall_at_1 + value: 42.269 + - type: recall_at_10 + value: 79.99300000000001 + - type: recall_at_100 + value: 90.338 + - type: recall_at_1000 + value: 95.881 + - type: recall_at_20 + value: 83.781 + - type: recall_at_3 + value: 70.081 + - type: recall_at_5 + value: 75.26 + task: + type: Retrieval + - dataset: + config: default + name: MTEB HotpotQA (default) + revision: ab518f4d6fcca38d87c25209f94beba119d02014 + split: train + type: mteb/hotpotqa + metrics: + - type: main_score + value: 79.148 + - type: map_at_1 + value: 43.106 + - type: map_at_10 + value: 71.986 + - type: map_at_100 + value: 72.721 + - type: map_at_1000 + value: 72.764 + - type: map_at_20 + value: 72.437 + - type: map_at_3 + value: 68.67699999999999 + - type: map_at_5 + value: 70.814 + - type: mrr_at_1 + value: 86.2129411764706 + - type: mrr_at_10 + value: 90.55639028944775 + - type: mrr_at_100 + value: 90.64496155524004 + - type: mrr_at_1000 + value: 90.64726989555281 + - type: mrr_at_20 + value: 90.61713734241617 + - type: mrr_at_3 + value: 89.98000000000013 + - type: mrr_at_5 + value: 90.36052941176395 + - type: nauc_map_at_1000_diff1 + value: 10.05761251509696 + - type: nauc_map_at_1000_max + value: 31.618440470080557 + - type: nauc_map_at_1000_std + value: 23.820130377632186 + - type: nauc_map_at_100_diff1 + value: 10.035879754654816 + - type: nauc_map_at_100_max + value: 31.62901297205673 + - type: nauc_map_at_100_std + value: 23.862472322221137 + - type: nauc_map_at_10_diff1 + value: 9.684570817984941 + - type: nauc_map_at_10_max + value: 31.339281257545803 + - type: nauc_map_at_10_std + value: 23.424796468856435 + - type: nauc_map_at_1_diff1 + value: 69.61856350238 + - type: nauc_map_at_1_max + value: 50.65234679645428 + - type: nauc_map_at_1_std + value: 10.64580237874998 + - type: nauc_map_at_20_diff1 + value: 9.873820218600816 + - type: nauc_map_at_20_max + value: 31.57189678474991 + - type: nauc_map_at_20_std + value: 23.794450408848927 + - type: nauc_map_at_3_diff1 + value: 9.274847308664114 + - type: nauc_map_at_3_max + value: 29.1281557454772 + - type: nauc_map_at_3_std + value: 19.88536868010767 + - type: nauc_map_at_5_diff1 + value: 9.45097370507114 + - type: nauc_map_at_5_max + value: 30.712377117427085 + - type: nauc_map_at_5_std + value: 22.246596762146282 + - type: nauc_mrr_at_1000_diff1 + value: 69.13234866095783 + - type: nauc_mrr_at_1000_max + value: 54.139390857725445 + - type: nauc_mrr_at_1000_std + value: 13.75136650615866 + - type: nauc_mrr_at_100_diff1 + value: 69.1340081434531 + - type: nauc_mrr_at_100_max + value: 54.14611791638828 + - type: nauc_mrr_at_100_std + value: 13.762847786557838 + - type: nauc_mrr_at_10_diff1 + value: 69.11212673452002 + - type: nauc_mrr_at_10_max + value: 54.27353042757507 + - type: nauc_mrr_at_10_std + value: 13.858522998058328 + - type: nauc_mrr_at_1_diff1 + value: 69.61856350238 + - type: nauc_mrr_at_1_max + value: 50.65234679645428 + - type: nauc_mrr_at_1_std + value: 10.64580237874998 + - type: nauc_mrr_at_20_diff1 + value: 69.13074505610524 + - type: nauc_mrr_at_20_max + value: 54.197116208216514 + - type: nauc_mrr_at_20_std + value: 13.813041480367966 + - type: nauc_mrr_at_3_diff1 + value: 68.99421809574834 + - type: nauc_mrr_at_3_max + value: 54.328484880230675 + - type: nauc_mrr_at_3_std + value: 13.379393094514672 + - type: nauc_mrr_at_5_diff1 + value: 69.0804931701423 + - type: nauc_mrr_at_5_max + value: 54.38248914021001 + - type: nauc_mrr_at_5_std + value: 13.798137560154458 + - type: nauc_ndcg_at_1000_diff1 + value: 16.967796236537282 + - type: nauc_ndcg_at_1000_max + value: 36.45392260390786 + - type: nauc_ndcg_at_1000_std + value: 26.656134038811558 + - type: nauc_ndcg_at_100_diff1 + value: 16.12113715473043 + - type: nauc_ndcg_at_100_max + value: 36.58658672941053 + - type: nauc_ndcg_at_100_std + value: 27.79684895123286 + - type: nauc_ndcg_at_10_diff1 + value: 14.454670518223331 + - type: nauc_ndcg_at_10_max + value: 35.41210078818887 + - type: nauc_ndcg_at_10_std + value: 26.110404553906253 + - type: nauc_ndcg_at_1_diff1 + value: 69.61856350238 + - type: nauc_ndcg_at_1_max + value: 50.65234679645428 + - type: nauc_ndcg_at_1_std + value: 10.64580237874998 + - type: nauc_ndcg_at_20_diff1 + value: 14.982971200193848 + - type: nauc_ndcg_at_20_max + value: 36.11694377424367 + - type: nauc_ndcg_at_20_std + value: 27.310725448903707 + - type: nauc_ndcg_at_3_diff1 + value: 14.247702904153513 + - type: nauc_ndcg_at_3_max + value: 32.195697331028185 + - type: nauc_ndcg_at_3_std + value: 20.328126768668824 + - type: nauc_ndcg_at_5_diff1 + value: 14.187006798751135 + - type: nauc_ndcg_at_5_max + value: 34.2517672713244 + - type: nauc_ndcg_at_5_std + value: 23.721875747074396 + - type: nauc_precision_at_1000_diff1 + value: -1.8032283061535919 + - type: nauc_precision_at_1000_max + value: 45.979727285748886 + - type: nauc_precision_at_1000_std + value: 66.62284452233656 + - type: nauc_precision_at_100_diff1 + value: -1.8731653482451494 + - type: nauc_precision_at_100_max + value: 40.81950868326732 + - type: nauc_precision_at_100_std + value: 56.669049299387645 + - type: nauc_precision_at_10_diff1 + value: -0.7360893556216586 + - type: nauc_precision_at_10_max + value: 34.234798886249465 + - type: nauc_precision_at_10_std + value: 37.066580831223526 + - type: nauc_precision_at_1_diff1 + value: 69.61856350238 + - type: nauc_precision_at_1_max + value: 50.65234679645428 + - type: nauc_precision_at_1_std + value: 10.64580237874998 + - type: nauc_precision_at_20_diff1 + value: -1.5994058940261544 + - type: nauc_precision_at_20_max + value: 36.56691220785231 + - type: nauc_precision_at_20_std + value: 43.66158032869279 + - type: nauc_precision_at_3_diff1 + value: 2.973992767232654 + - type: nauc_precision_at_3_max + value: 28.98427041638594 + - type: nauc_precision_at_3_std + value: 23.251400594453248 + - type: nauc_precision_at_5_diff1 + value: 1.1286753856774216 + - type: nauc_precision_at_5_max + value: 31.86301574615224 + - type: nauc_precision_at_5_std + value: 29.72307522609133 + - type: nauc_recall_at_1000_diff1 + value: -1.803228306153869 + - type: nauc_recall_at_1000_max + value: 45.979727285748616 + - type: nauc_recall_at_1000_std + value: 66.62284452233628 + - type: nauc_recall_at_100_diff1 + value: -1.8731653482454282 + - type: nauc_recall_at_100_max + value: 40.819508683267465 + - type: nauc_recall_at_100_std + value: 56.66904929938734 + - type: nauc_recall_at_10_diff1 + value: -0.7360893556216586 + - type: nauc_recall_at_10_max + value: 34.23479888624942 + - type: nauc_recall_at_10_std + value: 37.066580831223575 + - type: nauc_recall_at_1_diff1 + value: 69.61856350238 + - type: nauc_recall_at_1_max + value: 50.65234679645428 + - type: nauc_recall_at_1_std + value: 10.64580237874998 + - type: nauc_recall_at_20_diff1 + value: -1.5994058940263893 + - type: nauc_recall_at_20_max + value: 36.566912207852084 + - type: nauc_recall_at_20_std + value: 43.66158032869262 + - type: nauc_recall_at_3_diff1 + value: 2.973992767232573 + - type: nauc_recall_at_3_max + value: 28.984270416385762 + - type: nauc_recall_at_3_std + value: 23.251400594453074 + - type: nauc_recall_at_5_diff1 + value: 1.1286753856772571 + - type: nauc_recall_at_5_max + value: 31.863015746152207 + - type: nauc_recall_at_5_std + value: 29.723075226091378 + - type: ndcg_at_1 + value: 86.213 + - type: ndcg_at_10 + value: 79.148 + - type: ndcg_at_100 + value: 81.528 + - type: ndcg_at_1000 + value: 82.292 + - type: ndcg_at_20 + value: 80.215 + - type: ndcg_at_3 + value: 74.726 + - type: ndcg_at_5 + value: 77.285 + - type: precision_at_1 + value: 86.213 + - type: precision_at_10 + value: 16.479 + - type: precision_at_100 + value: 1.8319999999999999 + - type: precision_at_1000 + value: 0.193 + - type: precision_at_20 + value: 8.581999999999999 + - type: precision_at_3 + value: 48.464 + - type: precision_at_5 + value: 31.096 + - type: recall_at_1 + value: 43.106 + - type: recall_at_10 + value: 82.396 + - type: recall_at_100 + value: 91.581 + - type: recall_at_1000 + value: 96.576 + - type: recall_at_20 + value: 85.81800000000001 + - type: recall_at_3 + value: 72.696 + - type: recall_at_5 + value: 77.74 + task: + type: Retrieval + - dataset: + config: default + name: MTEB MSMARCO (default) + revision: c5a29a104738b98a9e76336939199e264163d4a0 + split: dev + type: mteb/msmarco + metrics: + - type: main_score + value: 43.092999999999996 + - type: map_at_1 + value: 22.796 + - type: map_at_10 + value: 35.897 + - type: map_at_100 + value: 37.043 + - type: map_at_1000 + value: 37.085 + - type: map_at_20 + value: 36.618 + - type: map_at_3 + value: 31.972 + - type: map_at_5 + value: 34.260000000000005 + - type: mrr_at_1 + value: 23.45272206303725 + - type: mrr_at_10 + value: 36.522098285350395 + - type: mrr_at_100 + value: 37.60415458447466 + - type: mrr_at_1000 + value: 37.64023510314476 + - type: mrr_at_20 + value: 37.210080850755766 + - type: mrr_at_3 + value: 32.681470869149834 + - type: mrr_at_5 + value: 34.937201528175734 + - type: nauc_map_at_1000_diff1 + value: 36.69898209074347 + - type: nauc_map_at_1000_max + value: 3.043965768557629 + - type: nauc_map_at_1000_std + value: -20.655216991448096 + - type: nauc_map_at_100_diff1 + value: 36.69030351985322 + - type: nauc_map_at_100_max + value: 3.0424899906280554 + - type: nauc_map_at_100_std + value: -20.63677399137818 + - type: nauc_map_at_10_diff1 + value: 36.69678708168894 + - type: nauc_map_at_10_max + value: 2.8808847385755403 + - type: nauc_map_at_10_std + value: -21.381065801657765 + - type: nauc_map_at_1_diff1 + value: 40.45282223147299 + - type: nauc_map_at_1_max + value: 2.4561406651468616 + - type: nauc_map_at_1_std + value: -18.20707520387699 + - type: nauc_map_at_20_diff1 + value: 36.67036642544658 + - type: nauc_map_at_20_max + value: 2.9995644731908193 + - type: nauc_map_at_20_std + value: -20.89094588435063 + - type: nauc_map_at_3_diff1 + value: 36.88718383236647 + - type: nauc_map_at_3_max + value: 2.692698975254288 + - type: nauc_map_at_3_std + value: -21.233896311915682 + - type: nauc_map_at_5_diff1 + value: 36.7331420446183 + - type: nauc_map_at_5_max + value: 2.7892096779560225 + - type: nauc_map_at_5_std + value: -21.354306663721907 + - type: nauc_mrr_at_1000_diff1 + value: 36.71782785781189 + - type: nauc_mrr_at_1000_max + value: 3.1005283778356745 + - type: nauc_mrr_at_1000_std + value: -20.2930295352988 + - type: nauc_mrr_at_100_diff1 + value: 36.707115695249854 + - type: nauc_mrr_at_100_max + value: 3.102295624252767 + - type: nauc_mrr_at_100_std + value: -20.26978377415289 + - type: nauc_mrr_at_10_diff1 + value: 36.717057516741825 + - type: nauc_mrr_at_10_max + value: 2.9860901265547737 + - type: nauc_mrr_at_10_std + value: -20.95568841004614 + - type: nauc_mrr_at_1_diff1 + value: 40.486539775597564 + - type: nauc_mrr_at_1_max + value: 2.5243704406671514 + - type: nauc_mrr_at_1_std + value: -18.04766011045231 + - type: nauc_mrr_at_20_diff1 + value: 36.67592961433832 + - type: nauc_mrr_at_20_max + value: 3.1025755273393787 + - type: nauc_mrr_at_20_std + value: -20.480021325210647 + - type: nauc_mrr_at_3_diff1 + value: 36.94309899232831 + - type: nauc_mrr_at_3_max + value: 2.6902959836823315 + - type: nauc_mrr_at_3_std + value: -20.99439547354983 + - type: nauc_mrr_at_5_diff1 + value: 36.762334959695295 + - type: nauc_mrr_at_5_max + value: 2.7983345425410175 + - type: nauc_mrr_at_5_std + value: -21.02847418758874 + - type: nauc_ndcg_at_1000_diff1 + value: 35.968719906618155 + - type: nauc_ndcg_at_1000_max + value: 3.787266631742529 + - type: nauc_ndcg_at_1000_std + value: -19.36095787174365 + - type: nauc_ndcg_at_100_diff1 + value: 35.73759695176694 + - type: nauc_ndcg_at_100_max + value: 3.926995494072129 + - type: nauc_ndcg_at_100_std + value: -18.455422708670035 + - type: nauc_ndcg_at_10_diff1 + value: 35.63192622556607 + - type: nauc_ndcg_at_10_max + value: 3.24107366499774 + - type: nauc_ndcg_at_10_std + value: -22.087888946880593 + - type: nauc_ndcg_at_1_diff1 + value: 40.486539775597564 + - type: nauc_ndcg_at_1_max + value: 2.5243704406671514 + - type: nauc_ndcg_at_1_std + value: -18.04766011045231 + - type: nauc_ndcg_at_20_diff1 + value: 35.525389559431154 + - type: nauc_ndcg_at_20_max + value: 3.6851086663206356 + - type: nauc_ndcg_at_20_std + value: -20.252260145761472 + - type: nauc_ndcg_at_3_diff1 + value: 36.04982237514636 + - type: nauc_ndcg_at_3_max + value: 2.758048272516093 + - type: nauc_ndcg_at_3_std + value: -21.940294918937997 + - type: nauc_ndcg_at_5_diff1 + value: 35.77858212500885 + - type: nauc_ndcg_at_5_max + value: 2.909959816503153 + - type: nauc_ndcg_at_5_std + value: -22.16896152451512 + - type: nauc_precision_at_1000_diff1 + value: -4.7162448246755195 + - type: nauc_precision_at_1000_max + value: 10.373650619252393 + - type: nauc_precision_at_1000_std + value: 14.357310662396907 + - type: nauc_precision_at_100_diff1 + value: 13.055295102773561 + - type: nauc_precision_at_100_max + value: 11.688756971071097 + - type: nauc_precision_at_100_std + value: 13.837542964600951 + - type: nauc_precision_at_10_diff1 + value: 29.285778868509432 + - type: nauc_precision_at_10_max + value: 4.167613880995443 + - type: nauc_precision_at_10_std + value: -23.171725652286263 + - type: nauc_precision_at_1_diff1 + value: 40.486539775597564 + - type: nauc_precision_at_1_max + value: 2.5243704406671514 + - type: nauc_precision_at_1_std + value: -18.04766011045231 + - type: nauc_precision_at_20_diff1 + value: 25.434935125833764 + - type: nauc_precision_at_20_max + value: 6.955075791618158 + - type: nauc_precision_at_20_std + value: -12.90153549510208 + - type: nauc_precision_at_3_diff1 + value: 33.436440867459595 + - type: nauc_precision_at_3_max + value: 3.004261639598586 + - type: nauc_precision_at_3_std + value: -23.94804289351618 + - type: nauc_precision_at_5_diff1 + value: 32.015481834045765 + - type: nauc_precision_at_5_max + value: 3.391910076030244 + - type: nauc_precision_at_5_std + value: -23.96787614532766 + - type: nauc_recall_at_1000_diff1 + value: 22.19644459090854 + - type: nauc_recall_at_1000_max + value: 40.140275718130034 + - type: nauc_recall_at_1000_std + value: 62.09666611742715 + - type: nauc_recall_at_100_diff1 + value: 28.019398682042812 + - type: nauc_recall_at_100_max + value: 13.86909861143805 + - type: nauc_recall_at_100_std + value: 16.073534949399807 + - type: nauc_recall_at_10_diff1 + value: 31.754844040271763 + - type: nauc_recall_at_10_max + value: 4.4294441953177905 + - type: nauc_recall_at_10_std + value: -24.294975062896643 + - type: nauc_recall_at_1_diff1 + value: 40.45282223147299 + - type: nauc_recall_at_1_max + value: 2.4561406651468616 + - type: nauc_recall_at_1_std + value: -18.20707520387699 + - type: nauc_recall_at_20_diff1 + value: 30.409841241464203 + - type: nauc_recall_at_20_max + value: 6.817793927374914 + - type: nauc_recall_at_20_std + value: -15.545580889638233 + - type: nauc_recall_at_3_diff1 + value: 33.50070203381667 + - type: nauc_recall_at_3_max + value: 2.925541620353969 + - type: nauc_recall_at_3_std + value: -23.772789254855955 + - type: nauc_recall_at_5_diff1 + value: 32.735790237010995 + - type: nauc_recall_at_5_max + value: 3.198760172060438 + - type: nauc_recall_at_5_std + value: -24.47406413878985 + - type: ndcg_at_1 + value: 23.453 + - type: ndcg_at_10 + value: 43.092999999999996 + - type: ndcg_at_100 + value: 48.588 + - type: ndcg_at_1000 + value: 49.633 + - type: ndcg_at_20 + value: 45.658 + - type: ndcg_at_3 + value: 35.15 + - type: ndcg_at_5 + value: 39.225 + - type: precision_at_1 + value: 23.453 + - type: precision_at_10 + value: 6.7989999999999995 + - type: precision_at_100 + value: 0.9560000000000001 + - type: precision_at_1000 + value: 0.105 + - type: precision_at_20 + value: 3.9370000000000003 + - type: precision_at_3 + value: 15.033 + - type: precision_at_5 + value: 11.103 + - type: recall_at_1 + value: 22.796 + - type: recall_at_10 + value: 65.146 + - type: recall_at_100 + value: 90.521 + - type: recall_at_1000 + value: 98.478 + - type: recall_at_20 + value: 75.12100000000001 + - type: recall_at_3 + value: 43.653999999999996 + - type: recall_at_5 + value: 53.419000000000004 + task: + type: Retrieval + - dataset: + config: default + name: MTEB MSMARCO (default) + revision: c5a29a104738b98a9e76336939199e264163d4a0 + split: test + type: mteb/msmarco + metrics: + - type: main_score + value: 72.355 + - type: map_at_1 + value: 2.443 + - type: map_at_10 + value: 16.415 + - type: map_at_100 + value: 43.407000000000004 + - type: map_at_1000 + value: 51.77 + - type: map_at_20 + value: 25.384 + - type: map_at_3 + value: 6.6000000000000005 + - type: map_at_5 + value: 9.716 + - type: mrr_at_1 + value: 93.02325581395348 + - type: mrr_at_10 + value: 96.12403100775194 + - type: mrr_at_100 + value: 96.12403100775194 + - type: mrr_at_1000 + value: 96.12403100775194 + - type: mrr_at_20 + value: 96.12403100775194 + - type: mrr_at_3 + value: 96.12403100775194 + - type: mrr_at_5 + value: 96.12403100775194 + - type: nauc_map_at_1000_diff1 + value: -13.095612894095154 + - type: nauc_map_at_1000_max + value: 26.995852134511583 + - type: nauc_map_at_1000_std + value: 54.67201845009677 + - type: nauc_map_at_100_diff1 + value: 0.9564790713862549 + - type: nauc_map_at_100_max + value: 8.799594495465193 + - type: nauc_map_at_100_std + value: 27.94895226058377 + - type: nauc_map_at_10_diff1 + value: 24.483469271673698 + - type: nauc_map_at_10_max + value: -20.21886527023598 + - type: nauc_map_at_10_std + value: -19.286796751707733 + - type: nauc_map_at_1_diff1 + value: 33.31026180296999 + - type: nauc_map_at_1_max + value: -39.24980445669469 + - type: nauc_map_at_1_std + value: -39.09477812455557 + - type: nauc_map_at_20_diff1 + value: 20.392448985910963 + - type: nauc_map_at_20_max + value: -7.7454812534735975 + - type: nauc_map_at_20_std + value: -6.208561974476969 + - type: nauc_map_at_3_diff1 + value: 38.18364178164642 + - type: nauc_map_at_3_max + value: -36.44921485304263 + - type: nauc_map_at_3_std + value: -35.31959806743695 + - type: nauc_map_at_5_diff1 + value: 35.33201978468498 + - type: nauc_map_at_5_max + value: -29.298453040830736 + - type: nauc_map_at_5_std + value: -28.920617587362997 + - type: nauc_mrr_at_1000_diff1 + value: -30.606157395084576 + - type: nauc_mrr_at_1000_max + value: -53.19264357326626 + - type: nauc_mrr_at_1000_std + value: -8.305258755085841 + - type: nauc_mrr_at_100_diff1 + value: -30.606157395084576 + - type: nauc_mrr_at_100_max + value: -53.19264357326626 + - type: nauc_mrr_at_100_std + value: -8.305258755085841 + - type: nauc_mrr_at_10_diff1 + value: -30.606157395084576 + - type: nauc_mrr_at_10_max + value: -53.19264357326626 + - type: nauc_mrr_at_10_std + value: -8.305258755085841 + - type: nauc_mrr_at_1_diff1 + value: -27.68112574340929 + - type: nauc_mrr_at_1_max + value: -52.777221496944726 + - type: nauc_mrr_at_1_std + value: -15.371067481794132 + - type: nauc_mrr_at_20_diff1 + value: -30.606157395084576 + - type: nauc_mrr_at_20_max + value: -53.19264357326626 + - type: nauc_mrr_at_20_std + value: -8.305258755085841 + - type: nauc_mrr_at_3_diff1 + value: -30.606157395084576 + - type: nauc_mrr_at_3_max + value: -53.19264357326626 + - type: nauc_mrr_at_3_std + value: -8.305258755085841 + - type: nauc_mrr_at_5_diff1 + value: -30.606157395084576 + - type: nauc_mrr_at_5_max + value: -53.19264357326626 + - type: nauc_mrr_at_5_std + value: -8.305258755085841 + - type: nauc_ndcg_at_1000_diff1 + value: -16.034483752772204 + - type: nauc_ndcg_at_1000_max + value: 45.46247864860879 + - type: nauc_ndcg_at_1000_std + value: 58.89930063809204 + - type: nauc_ndcg_at_100_diff1 + value: -10.748580136138578 + - type: nauc_ndcg_at_100_max + value: 28.946017610642706 + - type: nauc_ndcg_at_100_std + value: 52.582208083028625 + - type: nauc_ndcg_at_10_diff1 + value: 0.03984937943496068 + - type: nauc_ndcg_at_10_max + value: 19.79086384332024 + - type: nauc_ndcg_at_10_std + value: 19.155618836589376 + - type: nauc_ndcg_at_1_diff1 + value: 26.036794714344047 + - type: nauc_ndcg_at_1_max + value: -1.819122910416097 + - type: nauc_ndcg_at_1_std + value: -10.452802753949953 + - type: nauc_ndcg_at_20_diff1 + value: -1.6467702014700067 + - type: nauc_ndcg_at_20_max + value: 23.751741150948085 + - type: nauc_ndcg_at_20_std + value: 37.114835788141995 + - type: nauc_ndcg_at_3_diff1 + value: 11.021194985583985 + - type: nauc_ndcg_at_3_max + value: 16.444341503891035 + - type: nauc_ndcg_at_3_std + value: 9.583892749673794 + - type: nauc_ndcg_at_5_diff1 + value: 5.707152653487285 + - type: nauc_ndcg_at_5_max + value: 19.29303430227641 + - type: nauc_ndcg_at_5_std + value: 13.921993396555694 + - type: nauc_precision_at_1000_diff1 + value: -35.27681517614476 + - type: nauc_precision_at_1000_max + value: 30.735509903891266 + - type: nauc_precision_at_1000_std + value: 51.761061970373234 + - type: nauc_precision_at_100_diff1 + value: -32.90941039646208 + - type: nauc_precision_at_100_max + value: 33.710251139092136 + - type: nauc_precision_at_100_std + value: 58.62288946778689 + - type: nauc_precision_at_10_diff1 + value: -29.9325099508508 + - type: nauc_precision_at_10_max + value: 41.16949698700421 + - type: nauc_precision_at_10_std + value: 67.483536639692 + - type: nauc_precision_at_1_diff1 + value: -27.68112574340929 + - type: nauc_precision_at_1_max + value: -52.777221496944726 + - type: nauc_precision_at_1_std + value: -15.371067481794132 + - type: nauc_precision_at_20_diff1 + value: -26.466718701309933 + - type: nauc_precision_at_20_max + value: 42.4258197087897 + - type: nauc_precision_at_20_std + value: 68.28570412537994 + - type: nauc_precision_at_3_diff1 + value: -17.5537402528335 + - type: nauc_precision_at_3_max + value: 11.948422610902803 + - type: nauc_precision_at_3_std + value: 42.73257543682195 + - type: nauc_precision_at_5_diff1 + value: -9.71770975873051 + - type: nauc_precision_at_5_max + value: 42.91671438570917 + - type: nauc_precision_at_5_std + value: 65.11066579711269 + - type: nauc_recall_at_1000_diff1 + value: -31.28280669952206 + - type: nauc_recall_at_1000_max + value: 51.299313317147664 + - type: nauc_recall_at_1000_std + value: 62.264776617261845 + - type: nauc_recall_at_100_diff1 + value: 2.047346845359303 + - type: nauc_recall_at_100_max + value: 10.839602920439887 + - type: nauc_recall_at_100_std + value: 21.112967913680105 + - type: nauc_recall_at_10_diff1 + value: 24.59299024452016 + - type: nauc_recall_at_10_max + value: -20.50448466493449 + - type: nauc_recall_at_10_std + value: -22.198857077065455 + - type: nauc_recall_at_1_diff1 + value: 33.31026180296999 + - type: nauc_recall_at_1_max + value: -39.24980445669469 + - type: nauc_recall_at_1_std + value: -39.09477812455557 + - type: nauc_recall_at_20_diff1 + value: 20.473876153114187 + - type: nauc_recall_at_20_max + value: -7.977946091168067 + - type: nauc_recall_at_20_std + value: -9.952756164203983 + - type: nauc_recall_at_3_diff1 + value: 38.37782630038568 + - type: nauc_recall_at_3_max + value: -35.69952128519565 + - type: nauc_recall_at_3_std + value: -35.73407225716257 + - type: nauc_recall_at_5_diff1 + value: 36.29194975333239 + - type: nauc_recall_at_5_max + value: -28.71941774004871 + - type: nauc_recall_at_5_std + value: -30.51881398299733 + - type: ndcg_at_1 + value: 73.643 + - type: ndcg_at_10 + value: 72.355 + - type: ndcg_at_100 + value: 67.23400000000001 + - type: ndcg_at_1000 + value: 74.235 + - type: ndcg_at_20 + value: 70.539 + - type: ndcg_at_3 + value: 73.79299999999999 + - type: ndcg_at_5 + value: 73.051 + - type: precision_at_1 + value: 93.023 + - type: precision_at_10 + value: 82.093 + - type: precision_at_100 + value: 39.977000000000004 + - type: precision_at_1000 + value: 7.292999999999999 + - type: precision_at_20 + value: 73.372 + - type: precision_at_3 + value: 86.822 + - type: precision_at_5 + value: 85.116 + - type: recall_at_1 + value: 2.443 + - type: recall_at_10 + value: 17.87 + - type: recall_at_100 + value: 55.361000000000004 + - type: recall_at_1000 + value: 81.257 + - type: recall_at_20 + value: 28.236 + - type: recall_at_3 + value: 6.747 + - type: recall_at_5 + value: 10.299999999999999 + task: + type: Retrieval + - dataset: + config: default + name: MTEB MSMARCO (default) + revision: c5a29a104738b98a9e76336939199e264163d4a0 + split: train + type: mteb/msmarco + metrics: + - type: main_score + value: 40.294000000000004 + - type: map_at_1 + value: 20.827 + - type: map_at_10 + value: 33.211 + - type: map_at_100 + value: 34.467999999999996 + - type: map_at_1000 + value: 34.513 + - type: map_at_20 + value: 33.991 + - type: map_at_3 + value: 29.183999999999997 + - type: map_at_5 + value: 31.472 + - type: mrr_at_1 + value: 21.408958144029395 + - type: mrr_at_10 + value: 33.730745913638195 + - type: mrr_at_100 + value: 34.93236545001326 + - type: mrr_at_1000 + value: 34.97198061072765 + - type: mrr_at_20 + value: 34.48231609505797 + - type: mrr_at_3 + value: 29.782902764215052 + - type: mrr_at_5 + value: 32.035104323458434 + - type: nauc_map_at_1000_diff1 + value: 34.02777987106803 + - type: nauc_map_at_1000_max + value: -1.2815509501626068 + - type: nauc_map_at_1000_std + value: -21.697277337892647 + - type: nauc_map_at_100_diff1 + value: 34.01948446116364 + - type: nauc_map_at_100_max + value: -1.2758003329842975 + - type: nauc_map_at_100_std + value: -21.67054142869089 + - type: nauc_map_at_10_diff1 + value: 33.93401582866424 + - type: nauc_map_at_10_max + value: -1.436899018998692 + - type: nauc_map_at_10_std + value: -22.31805049004397 + - type: nauc_map_at_1_diff1 + value: 37.45478121283546 + - type: nauc_map_at_1_max + value: -1.7292282930395964 + - type: nauc_map_at_1_std + value: -20.09805076172682 + - type: nauc_map_at_20_diff1 + value: 33.971782632277694 + - type: nauc_map_at_20_max + value: -1.3408451363490113 + - type: nauc_map_at_20_std + value: -21.92591100627878 + - type: nauc_map_at_3_diff1 + value: 34.18069178615134 + - type: nauc_map_at_3_max + value: -1.630386214556127 + - type: nauc_map_at_3_std + value: -22.34846958218982 + - type: nauc_map_at_5_diff1 + value: 33.94716003411099 + - type: nauc_map_at_5_max + value: -1.549935182507629 + - type: nauc_map_at_5_std + value: -22.594386694644367 + - type: nauc_mrr_at_1000_diff1 + value: 34.03040430088337 + - type: nauc_mrr_at_1000_max + value: -1.3213937991255904 + - type: nauc_mrr_at_1000_std + value: -21.70389090918072 + - type: nauc_mrr_at_100_diff1 + value: 34.022522339496774 + - type: nauc_mrr_at_100_max + value: -1.3152285911472388 + - type: nauc_mrr_at_100_std + value: -21.676958200881675 + - type: nauc_mrr_at_10_diff1 + value: 33.93355900672435 + - type: nauc_mrr_at_10_max + value: -1.4614678438906665 + - type: nauc_mrr_at_10_std + value: -22.28861947971457 + - type: nauc_mrr_at_1_diff1 + value: 37.546452782182136 + - type: nauc_mrr_at_1_max + value: -1.7294766742589955 + - type: nauc_mrr_at_1_std + value: -20.296209334629502 + - type: nauc_mrr_at_20_diff1 + value: 33.97459099880882 + - type: nauc_mrr_at_20_max + value: -1.3724484493869407 + - type: nauc_mrr_at_20_std + value: -21.90938080921834 + - type: nauc_mrr_at_3_diff1 + value: 34.193028024231275 + - type: nauc_mrr_at_3_max + value: -1.637844162570119 + - type: nauc_mrr_at_3_std + value: -22.391785953371954 + - type: nauc_mrr_at_5_diff1 + value: 33.942270203492505 + - type: nauc_mrr_at_5_max + value: -1.5662427239583012 + - type: nauc_mrr_at_5_std + value: -22.588268045906617 + - type: nauc_ndcg_at_1000_diff1 + value: 33.320660858886434 + - type: nauc_ndcg_at_1000_max + value: -0.5284619160115556 + - type: nauc_ndcg_at_1000_std + value: -20.32856026648269 + - type: nauc_ndcg_at_100_diff1 + value: 33.09849012883688 + - type: nauc_ndcg_at_100_max + value: -0.2789444288826172 + - type: nauc_ndcg_at_100_std + value: -19.26818013053826 + - type: nauc_ndcg_at_10_diff1 + value: 32.72950590630474 + - type: nauc_ndcg_at_10_max + value: -1.1843262559070222 + - type: nauc_ndcg_at_10_std + value: -22.63881505799233 + - type: nauc_ndcg_at_1_diff1 + value: 37.561859267028986 + - type: nauc_ndcg_at_1_max + value: -1.7394271571128594 + - type: nauc_ndcg_at_1_std + value: -20.297582718211768 + - type: nauc_ndcg_at_20_diff1 + value: 32.81773357536739 + - type: nauc_ndcg_at_20_max + value: -0.818814036567139 + - type: nauc_ndcg_at_20_std + value: -21.169463105775417 + - type: nauc_ndcg_at_3_diff1 + value: 33.26148609046833 + - type: nauc_ndcg_at_3_max + value: -1.6039126727937565 + - type: nauc_ndcg_at_3_std + value: -22.964224178873312 + - type: nauc_ndcg_at_5_diff1 + value: 32.819994204462304 + - type: nauc_ndcg_at_5_max + value: -1.4658159671875957 + - type: nauc_ndcg_at_5_std + value: -23.349326219737495 + - type: nauc_precision_at_1000_diff1 + value: -1.6104135238170707 + - type: nauc_precision_at_1000_max + value: 8.363222968624898 + - type: nauc_precision_at_1000_std + value: 10.985242380235336 + - type: nauc_precision_at_100_diff1 + value: 13.990496029241228 + - type: nauc_precision_at_100_max + value: 9.094045257744632 + - type: nauc_precision_at_100_std + value: 12.351355190207318 + - type: nauc_precision_at_10_diff1 + value: 26.921214324266487 + - type: nauc_precision_at_10_max + value: -0.050709399417621606 + - type: nauc_precision_at_10_std + value: -22.106636099886227 + - type: nauc_precision_at_1_diff1 + value: 37.561859267028986 + - type: nauc_precision_at_1_max + value: -1.7394271571128594 + - type: nauc_precision_at_1_std + value: -20.297582718211768 + - type: nauc_precision_at_20_diff1 + value: 24.46377767433309 + - type: nauc_precision_at_20_max + value: 1.927563456163668 + - type: nauc_precision_at_20_std + value: -14.262008534423437 + - type: nauc_precision_at_3_diff1 + value: 30.47654368971038 + - type: nauc_precision_at_3_max + value: -1.4721069715025477 + - type: nauc_precision_at_3_std + value: -24.473419860453287 + - type: nauc_precision_at_5_diff1 + value: 28.84207492028085 + - type: nauc_precision_at_5_max + value: -1.072688996665273 + - type: nauc_precision_at_5_std + value: -24.97422068755472 + - type: nauc_recall_at_1000_diff1 + value: 18.855605147046848 + - type: nauc_recall_at_1000_max + value: 49.50663718661513 + - type: nauc_recall_at_1000_std + value: 68.09344040370827 + - type: nauc_recall_at_100_diff1 + value: 25.72087763068693 + - type: nauc_recall_at_100_max + value: 13.2531664672904 + - type: nauc_recall_at_100_std + value: 18.178666593688614 + - type: nauc_recall_at_10_diff1 + value: 28.61768654600192 + - type: nauc_recall_at_10_max + value: -0.2512367516764246 + - type: nauc_recall_at_10_std + value: -23.054794100612842 + - type: nauc_recall_at_1_diff1 + value: 37.45478121283546 + - type: nauc_recall_at_1_max + value: -1.7292282930395964 + - type: nauc_recall_at_1_std + value: -20.09805076172682 + - type: nauc_recall_at_20_diff1 + value: 27.990638788151053 + - type: nauc_recall_at_20_max + value: 1.766087865322523 + - type: nauc_recall_at_20_std + value: -15.950117081705836 + - type: nauc_recall_at_3_diff1 + value: 30.736564004374976 + - type: nauc_recall_at_3_max + value: -1.5244277586907655 + - type: nauc_recall_at_3_std + value: -24.422939089006718 + - type: nauc_recall_at_5_diff1 + value: 29.564289336037696 + - type: nauc_recall_at_5_max + value: -1.2071216109305303 + - type: nauc_recall_at_5_std + value: -25.241374271913013 + - type: ndcg_at_1 + value: 21.404999999999998 + - type: ndcg_at_10 + value: 40.294000000000004 + - type: ndcg_at_100 + value: 46.337 + - type: ndcg_at_1000 + value: 47.428 + - type: ndcg_at_20 + value: 43.064 + - type: ndcg_at_3 + value: 32.083 + - type: ndcg_at_5 + value: 36.16 + - type: precision_at_1 + value: 21.404999999999998 + - type: precision_at_10 + value: 6.485 + - type: precision_at_100 + value: 0.9490000000000001 + - type: precision_at_1000 + value: 0.104 + - type: precision_at_20 + value: 3.816 + - type: precision_at_3 + value: 13.748 + - type: precision_at_5 + value: 10.31 + - type: recall_at_1 + value: 20.827 + - type: recall_at_10 + value: 62.272000000000006 + - type: recall_at_100 + value: 90.20899999999999 + - type: recall_at_1000 + value: 98.483 + - type: recall_at_20 + value: 73.038 + - type: recall_at_3 + value: 39.883 + - type: recall_at_5 + value: 49.688 + task: + type: Retrieval + - dataset: + config: default + name: MTEB NFCorpus (default) + revision: ec0fa4fe99da2ff19ca1214b7966684033a58814 + split: test + type: mteb/nfcorpus + metrics: + - type: main_score + value: 38.184000000000005 + - type: map_at_1 + value: 6.401 + - type: map_at_10 + value: 14.637 + - type: map_at_100 + value: 18.485 + - type: map_at_1000 + value: 20.025000000000002 + - type: map_at_20 + value: 16.3 + - type: map_at_3 + value: 10.745000000000001 + - type: map_at_5 + value: 12.696 + - type: mrr_at_1 + value: 48.297213622291025 + - type: mrr_at_10 + value: 57.5673251756843 + - type: mrr_at_100 + value: 58.107503418726736 + - type: mrr_at_1000 + value: 58.14401899150543 + - type: mrr_at_20 + value: 57.91440208390433 + - type: mrr_at_3 + value: 55.77915376676987 + - type: mrr_at_5 + value: 56.92466460268319 + - type: nauc_map_at_1000_diff1 + value: 27.606998470664717 + - type: nauc_map_at_1000_max + value: 31.6328019008826 + - type: nauc_map_at_1000_std + value: 17.810662159225608 + - type: nauc_map_at_100_diff1 + value: 29.083885588601984 + - type: nauc_map_at_100_max + value: 30.64819024654504 + - type: nauc_map_at_100_std + value: 14.340669838788134 + - type: nauc_map_at_10_diff1 + value: 32.946874475186 + - type: nauc_map_at_10_max + value: 24.615879188360363 + - type: nauc_map_at_10_std + value: 2.7445636700846694 + - type: nauc_map_at_1_diff1 + value: 44.98678904772625 + - type: nauc_map_at_1_max + value: 13.546030135367564 + - type: nauc_map_at_1_std + value: -9.611686868566363 + - type: nauc_map_at_20_diff1 + value: 30.992002529054773 + - type: nauc_map_at_20_max + value: 27.796270325045814 + - type: nauc_map_at_20_std + value: 7.68173408442734 + - type: nauc_map_at_3_diff1 + value: 38.162397772445075 + - type: nauc_map_at_3_max + value: 17.20746221024842 + - type: nauc_map_at_3_std + value: -5.889837720709133 + - type: nauc_map_at_5_diff1 + value: 36.0872960437883 + - type: nauc_map_at_5_max + value: 20.455525612146143 + - type: nauc_map_at_5_std + value: -2.7411662971514295 + - type: nauc_mrr_at_1000_diff1 + value: 30.79257013179879 + - type: nauc_mrr_at_1000_max + value: 50.555768937733326 + - type: nauc_mrr_at_1000_std + value: 33.749183381655214 + - type: nauc_mrr_at_100_diff1 + value: 30.803504870581605 + - type: nauc_mrr_at_100_max + value: 50.59064210766916 + - type: nauc_mrr_at_100_std + value: 33.79275925748245 + - type: nauc_mrr_at_10_diff1 + value: 30.950058059247947 + - type: nauc_mrr_at_10_max + value: 50.40866010594709 + - type: nauc_mrr_at_10_std + value: 33.68951717948501 + - type: nauc_mrr_at_1_diff1 + value: 31.491330719810335 + - type: nauc_mrr_at_1_max + value: 41.792415954935095 + - type: nauc_mrr_at_1_std + value: 23.047306999315413 + - type: nauc_mrr_at_20_diff1 + value: 30.697653537753162 + - type: nauc_mrr_at_20_max + value: 50.3813529810355 + - type: nauc_mrr_at_20_std + value: 33.66837659146359 + - type: nauc_mrr_at_3_diff1 + value: 31.668156779063818 + - type: nauc_mrr_at_3_max + value: 50.22181337505093 + - type: nauc_mrr_at_3_std + value: 32.14633179870973 + - type: nauc_mrr_at_5_diff1 + value: 31.274630478737077 + - type: nauc_mrr_at_5_max + value: 50.52878386962845 + - type: nauc_mrr_at_5_std + value: 33.43805881295448 + - type: nauc_ndcg_at_1000_diff1 + value: 24.798548344185544 + - type: nauc_ndcg_at_1000_max + value: 48.36701062078609 + - type: nauc_ndcg_at_1000_std + value: 38.34086919539637 + - type: nauc_ndcg_at_100_diff1 + value: 25.427557288368313 + - type: nauc_ndcg_at_100_max + value: 42.06270676662367 + - type: nauc_ndcg_at_100_std + value: 31.748602206925497 + - type: nauc_ndcg_at_10_diff1 + value: 19.692623344846087 + - type: nauc_ndcg_at_10_max + value: 40.232233870099385 + - type: nauc_ndcg_at_10_std + value: 29.97728982863539 + - type: nauc_ndcg_at_1_diff1 + value: 31.779937738240662 + - type: nauc_ndcg_at_1_max + value: 40.34342577419757 + - type: nauc_ndcg_at_1_std + value: 22.950173782876366 + - type: nauc_ndcg_at_20_diff1 + value: 20.747702508210583 + - type: nauc_ndcg_at_20_max + value: 40.16720666498737 + - type: nauc_ndcg_at_20_std + value: 30.903415317477506 + - type: nauc_ndcg_at_3_diff1 + value: 24.42265281145814 + - type: nauc_ndcg_at_3_max + value: 43.24356099664565 + - type: nauc_ndcg_at_3_std + value: 27.632379702409395 + - type: nauc_ndcg_at_5_diff1 + value: 22.070337848806965 + - type: nauc_ndcg_at_5_max + value: 41.64135353616761 + - type: nauc_ndcg_at_5_std + value: 28.07901862839794 + - type: nauc_precision_at_1000_diff1 + value: -14.701266128133256 + - type: nauc_precision_at_1000_max + value: 6.462978714217531 + - type: nauc_precision_at_1000_std + value: 31.814207539261506 + - type: nauc_precision_at_100_diff1 + value: -10.658429998601529 + - type: nauc_precision_at_100_max + value: 19.043079314144624 + - type: nauc_precision_at_100_std + value: 43.633947780936026 + - type: nauc_precision_at_10_diff1 + value: -0.13881704719120078 + - type: nauc_precision_at_10_max + value: 36.9070258018293 + - type: nauc_precision_at_10_std + value: 39.9287945576426 + - type: nauc_precision_at_1_diff1 + value: 31.491330719810335 + - type: nauc_precision_at_1_max + value: 41.792415954935095 + - type: nauc_precision_at_1_std + value: 23.047306999315413 + - type: nauc_precision_at_20_diff1 + value: -4.922519618670497 + - type: nauc_precision_at_20_max + value: 32.57150301394898 + - type: nauc_precision_at_20_std + value: 43.499644765729904 + - type: nauc_precision_at_3_diff1 + value: 14.781686610614608 + - type: nauc_precision_at_3_max + value: 44.57882765811817 + - type: nauc_precision_at_3_std + value: 34.090416737481476 + - type: nauc_precision_at_5_diff1 + value: 7.800725142070681 + - type: nauc_precision_at_5_max + value: 40.41029866706074 + - type: nauc_precision_at_5_std + value: 35.07366907257489 + - type: nauc_recall_at_1000_diff1 + value: 9.319671169112267 + - type: nauc_recall_at_1000_max + value: 22.536801199782687 + - type: nauc_recall_at_1000_std + value: 23.671097200188054 + - type: nauc_recall_at_100_diff1 + value: 17.9627378361286 + - type: nauc_recall_at_100_max + value: 28.03776631284452 + - type: nauc_recall_at_100_std + value: 20.58127171451453 + - type: nauc_recall_at_10_diff1 + value: 27.144388648804746 + - type: nauc_recall_at_10_max + value: 21.92573151789915 + - type: nauc_recall_at_10_std + value: 3.3904974864867077 + - type: nauc_recall_at_1_diff1 + value: 44.98678904772625 + - type: nauc_recall_at_1_max + value: 13.546030135367564 + - type: nauc_recall_at_1_std + value: -9.611686868566363 + - type: nauc_recall_at_20_diff1 + value: 24.28379177579911 + - type: nauc_recall_at_20_max + value: 25.846486376177243 + - type: nauc_recall_at_20_std + value: 9.393542031681196 + - type: nauc_recall_at_3_diff1 + value: 34.45294795679497 + - type: nauc_recall_at_3_max + value: 15.217898442102806 + - type: nauc_recall_at_3_std + value: -6.263148154779078 + - type: nauc_recall_at_5_diff1 + value: 31.265671395262956 + - type: nauc_recall_at_5_max + value: 17.712054796957467 + - type: nauc_recall_at_5_std + value: -2.6905643929982817 + - type: ndcg_at_1 + value: 47.059 + - type: ndcg_at_10 + value: 38.184000000000005 + - type: ndcg_at_100 + value: 34.695 + - type: ndcg_at_1000 + value: 43.336999999999996 + - type: ndcg_at_20 + value: 35.752 + - type: ndcg_at_3 + value: 43.611 + - type: ndcg_at_5 + value: 41.725 + - type: precision_at_1 + value: 48.297000000000004 + - type: precision_at_10 + value: 28.451999999999998 + - type: precision_at_100 + value: 8.796 + - type: precision_at_1000 + value: 2.165 + - type: precision_at_20 + value: 21.115000000000002 + - type: precision_at_3 + value: 41.176 + - type: precision_at_5 + value: 36.594 + - type: recall_at_1 + value: 6.401 + - type: recall_at_10 + value: 18.611 + - type: recall_at_100 + value: 34.575 + - type: recall_at_1000 + value: 66.24600000000001 + - type: recall_at_20 + value: 22.698999999999998 + - type: recall_at_3 + value: 12.074 + - type: recall_at_5 + value: 15.183 + task: + type: Retrieval + - dataset: + config: default + name: MTEB NQ (default) + revision: b774495ed302d8c44a3a7ea25c90dbce03968f31 + split: test + type: mteb/nq + metrics: + - type: main_score + value: 61.43300000000001 + - type: map_at_1 + value: 37.128 + - type: map_at_10 + value: 53.667 + - type: map_at_100 + value: 54.534000000000006 + - type: map_at_1000 + value: 54.55 + - type: map_at_20 + value: 54.266999999999996 + - type: map_at_3 + value: 49.269 + - type: map_at_5 + value: 51.912000000000006 + - type: mrr_at_1 + value: 41.68597914252607 + - type: mrr_at_10 + value: 56.155943644356 + - type: mrr_at_100 + value: 56.739900435896665 + - type: mrr_at_1000 + value: 56.7490673421236 + - type: mrr_at_20 + value: 56.55444451957713 + - type: mrr_at_3 + value: 52.66029354963283 + - type: mrr_at_5 + value: 54.77356122054835 + - type: nauc_map_at_1000_diff1 + value: 38.35357325796708 + - type: nauc_map_at_1000_max + value: 27.684904103629254 + - type: nauc_map_at_1000_std + value: -2.0615074863647473 + - type: nauc_map_at_100_diff1 + value: 38.34938790232624 + - type: nauc_map_at_100_max + value: 27.696112743494766 + - type: nauc_map_at_100_std + value: -2.0427150972305617 + - type: nauc_map_at_10_diff1 + value: 38.15278185727002 + - type: nauc_map_at_10_max + value: 27.715948790085353 + - type: nauc_map_at_10_std + value: -2.328925314870795 + - type: nauc_map_at_1_diff1 + value: 42.29788217295333 + - type: nauc_map_at_1_max + value: 22.76651930977973 + - type: nauc_map_at_1_std + value: -3.768315692325868 + - type: nauc_map_at_20_diff1 + value: 38.302614180998674 + - type: nauc_map_at_20_max + value: 27.78592293953343 + - type: nauc_map_at_20_std + value: -2.0977635859762214 + - type: nauc_map_at_3_diff1 + value: 38.115331660918294 + - type: nauc_map_at_3_max + value: 26.181226890845505 + - type: nauc_map_at_3_std + value: -3.743205465259794 + - type: nauc_map_at_5_diff1 + value: 37.90906363195439 + - type: nauc_map_at_5_max + value: 26.989258393726985 + - type: nauc_map_at_5_std + value: -3.466409193773486 + - type: nauc_mrr_at_1000_diff1 + value: 38.43355997987963 + - type: nauc_mrr_at_1000_max + value: 28.171753738463167 + - type: nauc_mrr_at_1000_std + value: -0.010258951333665323 + - type: nauc_mrr_at_100_diff1 + value: 38.431353692452255 + - type: nauc_mrr_at_100_max + value: 28.179704287828383 + - type: nauc_mrr_at_100_std + value: 0.001576750758498444 + - type: nauc_mrr_at_10_diff1 + value: 38.26843151859599 + - type: nauc_mrr_at_10_max + value: 28.306144736151694 + - type: nauc_mrr_at_10_std + value: -0.0669248703829922 + - type: nauc_mrr_at_1_diff1 + value: 42.123921628197245 + - type: nauc_mrr_at_1_max + value: 24.588261643793565 + - type: nauc_mrr_at_1_std + value: -1.4047148219421608 + - type: nauc_mrr_at_20_diff1 + value: 38.39836187546001 + - type: nauc_mrr_at_20_max + value: 28.295906816963445 + - type: nauc_mrr_at_20_std + value: -0.007618532585832778 + - type: nauc_mrr_at_3_diff1 + value: 38.06690792892592 + - type: nauc_mrr_at_3_max + value: 27.44210364010309 + - type: nauc_mrr_at_3_std + value: -0.7472428440350823 + - type: nauc_mrr_at_5_diff1 + value: 38.0063558642651 + - type: nauc_mrr_at_5_max + value: 27.934465703393656 + - type: nauc_mrr_at_5_std + value: -0.6317390944885618 + - type: nauc_ndcg_at_1000_diff1 + value: 37.88579365541516 + - type: nauc_ndcg_at_1000_max + value: 29.274472432473182 + - type: nauc_ndcg_at_1000_std + value: -0.05915108845712491 + - type: nauc_ndcg_at_100_diff1 + value: 37.828943087002884 + - type: nauc_ndcg_at_100_max + value: 29.699374135251684 + - type: nauc_ndcg_at_100_std + value: 0.5421292608323158 + - type: nauc_ndcg_at_10_diff1 + value: 37.046948662407296 + - type: nauc_ndcg_at_10_max + value: 30.16009404810942 + - type: nauc_ndcg_at_10_std + value: -0.4458044961355911 + - type: nauc_ndcg_at_1_diff1 + value: 42.123921628197245 + - type: nauc_ndcg_at_1_max + value: 24.588261643793565 + - type: nauc_ndcg_at_1_std + value: -1.4047148219421608 + - type: nauc_ndcg_at_20_diff1 + value: 37.54663557752722 + - type: nauc_ndcg_at_20_max + value: 30.413588986848744 + - type: nauc_ndcg_at_20_std + value: 0.25700301995293373 + - type: nauc_ndcg_at_3_diff1 + value: 36.89412167863663 + - type: nauc_ndcg_at_3_max + value: 27.233806790041807 + - type: nauc_ndcg_at_3_std + value: -3.0871708406903 + - type: nauc_ndcg_at_5_diff1 + value: 36.5141342456609 + - type: nauc_ndcg_at_5_max + value: 28.556555633738796 + - type: nauc_ndcg_at_5_std + value: -2.728248192491663 + - type: nauc_precision_at_1000_diff1 + value: -8.137754125603385 + - type: nauc_precision_at_1000_max + value: 5.838400348988766 + - type: nauc_precision_at_1000_std + value: 13.996699568115895 + - type: nauc_precision_at_100_diff1 + value: -4.454681692042504 + - type: nauc_precision_at_100_max + value: 11.976565093822284 + - type: nauc_precision_at_100_std + value: 18.269274901888906 + - type: nauc_precision_at_10_diff1 + value: 9.773575486302922 + - type: nauc_precision_at_10_max + value: 25.790347896491276 + - type: nauc_precision_at_10_std + value: 12.076820923493626 + - type: nauc_precision_at_1_diff1 + value: 42.123921628197245 + - type: nauc_precision_at_1_max + value: 24.588261643793565 + - type: nauc_precision_at_1_std + value: -1.4047148219421608 + - type: nauc_precision_at_20_diff1 + value: 4.496587357358886 + - type: nauc_precision_at_20_max + value: 21.787427995109816 + - type: nauc_precision_at_20_std + value: 15.687199106823549 + - type: nauc_precision_at_3_diff1 + value: 23.713968885402174 + - type: nauc_precision_at_3_max + value: 27.199278010571494 + - type: nauc_precision_at_3_std + value: 1.2478421491292235 + - type: nauc_precision_at_5_diff1 + value: 16.570543406461667 + - type: nauc_precision_at_5_max + value: 26.09436523762944 + - type: nauc_precision_at_5_std + value: 3.4857405443015215 + - type: nauc_recall_at_1000_diff1 + value: 20.14160082354098 + - type: nauc_recall_at_1000_max + value: 89.14103766401765 + - type: nauc_recall_at_1000_std + value: 89.37960845266102 + - type: nauc_recall_at_100_diff1 + value: 31.211692635521487 + - type: nauc_recall_at_100_max + value: 64.21643563588279 + - type: nauc_recall_at_100_std + value: 50.44423051880654 + - type: nauc_recall_at_10_diff1 + value: 29.73508259517405 + - type: nauc_recall_at_10_max + value: 40.29407708917718 + - type: nauc_recall_at_10_std + value: 5.464129983129644 + - type: nauc_recall_at_1_diff1 + value: 42.29788217295333 + - type: nauc_recall_at_1_max + value: 22.76651930977973 + - type: nauc_recall_at_1_std + value: -3.768315692325868 + - type: nauc_recall_at_20_diff1 + value: 31.443876568168154 + - type: nauc_recall_at_20_max + value: 48.7315962115064 + - type: nauc_recall_at_20_std + value: 13.867248011402793 + - type: nauc_recall_at_3_diff1 + value: 31.946580259067396 + - type: nauc_recall_at_3_max + value: 27.990805113588557 + - type: nauc_recall_at_3_std + value: -4.062651237427363 + - type: nauc_recall_at_5_diff1 + value: 29.745482391859234 + - type: nauc_recall_at_5_max + value: 31.59767272329835 + - type: nauc_recall_at_5_std + value: -3.552541111763621 + - type: ndcg_at_1 + value: 41.686 + - type: ndcg_at_10 + value: 61.43300000000001 + - type: ndcg_at_100 + value: 64.82 + - type: ndcg_at_1000 + value: 65.185 + - type: ndcg_at_20 + value: 63.271 + - type: ndcg_at_3 + value: 53.391 + - type: ndcg_at_5 + value: 57.679 + - type: precision_at_1 + value: 41.686 + - type: precision_at_10 + value: 9.838 + - type: precision_at_100 + value: 1.174 + - type: precision_at_1000 + value: 0.121 + - type: precision_at_20 + value: 5.369 + - type: precision_at_3 + value: 24.034 + - type: precision_at_5 + value: 16.929 + - type: recall_at_1 + value: 37.128 + - type: recall_at_10 + value: 82.421 + - type: recall_at_100 + value: 96.676 + - type: recall_at_1000 + value: 99.372 + - type: recall_at_20 + value: 89.11 + - type: recall_at_3 + value: 61.882000000000005 + - type: recall_at_5 + value: 71.637 + task: + type: Retrieval + - dataset: + config: default + name: MTEB QuoraRetrieval (default) + revision: e4e08e0b7dbe3c8700f0daef558ff32256715259 + split: dev + type: mteb/quora + metrics: + - type: main_score + value: 88.947 + - type: map_at_1 + value: 70.974 + - type: map_at_10 + value: 85.357 + - type: map_at_100 + value: 85.904 + - type: map_at_1000 + value: 85.917 + - type: map_at_20 + value: 85.741 + - type: map_at_3 + value: 82.471 + - type: map_at_5 + value: 84.307 + - type: mrr_at_1 + value: 81.72 + - type: mrr_at_10 + value: 87.96561904761907 + - type: mrr_at_100 + value: 88.05981288204102 + - type: mrr_at_1000 + value: 88.0605989923228 + - type: mrr_at_20 + value: 88.03992553842785 + - type: mrr_at_3 + value: 87.1033333333335 + - type: mrr_at_5 + value: 87.69333333333338 + - type: nauc_map_at_1000_diff1 + value: 77.2583095861056 + - type: nauc_map_at_1000_max + value: 29.819766579943856 + - type: nauc_map_at_1000_std + value: -44.22888462511169 + - type: nauc_map_at_100_diff1 + value: 77.26304950694188 + - type: nauc_map_at_100_max + value: 29.766284798482673 + - type: nauc_map_at_100_std + value: -44.310746009053105 + - type: nauc_map_at_10_diff1 + value: 77.44031508365418 + - type: nauc_map_at_10_max + value: 29.37377585409888 + - type: nauc_map_at_10_std + value: -45.92572603956514 + - type: nauc_map_at_1_diff1 + value: 80.78269598421348 + - type: nauc_map_at_1_max + value: 20.375459745079624 + - type: nauc_map_at_1_std + value: -40.40708560773394 + - type: nauc_map_at_20_diff1 + value: 77.34615850639551 + - type: nauc_map_at_20_max + value: 29.62641843400643 + - type: nauc_map_at_20_std + value: -44.93811637470454 + - type: nauc_map_at_3_diff1 + value: 77.63731869100205 + - type: nauc_map_at_3_max + value: 28.37061428460734 + - type: nauc_map_at_3_std + value: -46.90171540251599 + - type: nauc_map_at_5_diff1 + value: 77.6416217351081 + - type: nauc_map_at_5_max + value: 28.94297051651431 + - type: nauc_map_at_5_std + value: -47.26092351421585 + - type: nauc_mrr_at_1000_diff1 + value: 78.06419885079407 + - type: nauc_mrr_at_1000_max + value: 31.678159608697502 + - type: nauc_mrr_at_1000_std + value: -39.57520042310982 + - type: nauc_mrr_at_100_diff1 + value: 78.06493963210038 + - type: nauc_mrr_at_100_max + value: 31.6797189831928 + - type: nauc_mrr_at_100_std + value: -39.576792634210975 + - type: nauc_mrr_at_10_diff1 + value: 78.09395363010262 + - type: nauc_mrr_at_10_max + value: 31.691192281739916 + - type: nauc_mrr_at_10_std + value: -39.74691927943722 + - type: nauc_mrr_at_1_diff1 + value: 78.66350900281675 + - type: nauc_mrr_at_1_max + value: 30.235119997741407 + - type: nauc_mrr_at_1_std + value: -37.67555546868721 + - type: nauc_mrr_at_20_diff1 + value: 78.07234822917289 + - type: nauc_mrr_at_20_max + value: 31.688332584742128 + - type: nauc_mrr_at_20_std + value: -39.577189557335664 + - type: nauc_mrr_at_3_diff1 + value: 77.6348006335469 + - type: nauc_mrr_at_3_max + value: 32.048478875517446 + - type: nauc_mrr_at_3_std + value: -39.46137689861855 + - type: nauc_mrr_at_5_diff1 + value: 77.99686070703672 + - type: nauc_mrr_at_5_max + value: 31.745081527433022 + - type: nauc_mrr_at_5_std + value: -40.05134752020732 + - type: nauc_ndcg_at_1000_diff1 + value: 77.30748561300635 + - type: nauc_ndcg_at_1000_max + value: 31.036498855254475 + - type: nauc_ndcg_at_1000_std + value: -42.016114434384086 + - type: nauc_ndcg_at_100_diff1 + value: 77.32024263250557 + - type: nauc_ndcg_at_100_max + value: 30.84022156601532 + - type: nauc_ndcg_at_100_std + value: -42.439836942464545 + - type: nauc_ndcg_at_10_diff1 + value: 77.5309323684768 + - type: nauc_ndcg_at_10_max + value: 30.45331612382649 + - type: nauc_ndcg_at_10_std + value: -45.57451087917826 + - type: nauc_ndcg_at_1_diff1 + value: 78.66350900281675 + - type: nauc_ndcg_at_1_max + value: 30.235119997741407 + - type: nauc_ndcg_at_1_std + value: -37.67555546868721 + - type: nauc_ndcg_at_20_diff1 + value: 77.51166891171162 + - type: nauc_ndcg_at_20_max + value: 30.661491259365874 + - type: nauc_ndcg_at_20_std + value: -44.08634405360655 + - type: nauc_ndcg_at_3_diff1 + value: 76.30271461099069 + - type: nauc_ndcg_at_3_max + value: 31.006012391984665 + - type: nauc_ndcg_at_3_std + value: -44.7168963927777 + - type: nauc_ndcg_at_5_diff1 + value: 77.22382616758514 + - type: nauc_ndcg_at_5_max + value: 30.447766528635935 + - type: nauc_ndcg_at_5_std + value: -46.70804061911347 + - type: nauc_precision_at_1000_diff1 + value: -43.859567906851446 + - type: nauc_precision_at_1000_max + value: -0.8109139781032491 + - type: nauc_precision_at_1000_std + value: 39.42408660425704 + - type: nauc_precision_at_100_diff1 + value: -43.449801070651425 + - type: nauc_precision_at_100_max + value: -2.0242969976422835 + - type: nauc_precision_at_100_std + value: 36.72249237618423 + - type: nauc_precision_at_10_diff1 + value: -38.21061499361725 + - type: nauc_precision_at_10_max + value: -0.5572626903699998 + - type: nauc_precision_at_10_std + value: 24.710137740124264 + - type: nauc_precision_at_1_diff1 + value: 78.66350900281675 + - type: nauc_precision_at_1_max + value: 30.235119997741407 + - type: nauc_precision_at_1_std + value: -37.67555546868721 + - type: nauc_precision_at_20_diff1 + value: -41.46856876639328 + - type: nauc_precision_at_20_max + value: -1.7948936387960843 + - type: nauc_precision_at_20_std + value: 30.73511099189952 + - type: nauc_precision_at_3_diff1 + value: -20.72983464260141 + - type: nauc_precision_at_3_max + value: 9.29173133391085 + - type: nauc_precision_at_3_std + value: 8.014964817204172 + - type: nauc_precision_at_5_diff1 + value: -31.862430757438364 + - type: nauc_precision_at_5_max + value: 3.0172810360114917 + - type: nauc_precision_at_5_std + value: 16.6050176953488 + - type: nauc_recall_at_1000_diff1 + value: 87.23210214323363 + - type: nauc_recall_at_1000_max + value: 97.27596464931685 + - type: nauc_recall_at_1000_std + value: 80.86009597636799 + - type: nauc_recall_at_100_diff1 + value: 73.96238195012937 + - type: nauc_recall_at_100_max + value: 23.69214011088809 + - type: nauc_recall_at_100_std + value: -70.0836554037542 + - type: nauc_recall_at_10_diff1 + value: 75.27112739694192 + - type: nauc_recall_at_10_max + value: 27.328681315886648 + - type: nauc_recall_at_10_std + value: -68.13668056983711 + - type: nauc_recall_at_1_diff1 + value: 80.78269598421348 + - type: nauc_recall_at_1_max + value: 20.375459745079624 + - type: nauc_recall_at_1_std + value: -40.40708560773394 + - type: nauc_recall_at_20_diff1 + value: 77.02460817214431 + - type: nauc_recall_at_20_max + value: 27.83014811353238 + - type: nauc_recall_at_20_std + value: -71.19576480985623 + - type: nauc_recall_at_3_diff1 + value: 72.81020455567739 + - type: nauc_recall_at_3_max + value: 28.000021508004576 + - type: nauc_recall_at_3_std + value: -53.52349625767309 + - type: nauc_recall_at_5_diff1 + value: 73.16164425572008 + - type: nauc_recall_at_5_max + value: 26.932602565117513 + - type: nauc_recall_at_5_std + value: -62.81566446222357 + - type: ndcg_at_1 + value: 81.72 + - type: ndcg_at_10 + value: 88.947 + - type: ndcg_at_100 + value: 90.02 + - type: ndcg_at_1000 + value: 90.09899999999999 + - type: ndcg_at_20 + value: 89.579 + - type: ndcg_at_3 + value: 86.311 + - type: ndcg_at_5 + value: 87.79299999999999 + - type: precision_at_1 + value: 81.72 + - type: precision_at_10 + value: 13.450000000000001 + - type: precision_at_100 + value: 1.493 + - type: precision_at_1000 + value: 0.152 + - type: precision_at_20 + value: 7.087000000000001 + - type: precision_at_3 + value: 37.987 + - type: precision_at_5 + value: 24.895999999999997 + - type: recall_at_1 + value: 70.974 + - type: recall_at_10 + value: 96.045 + - type: recall_at_100 + value: 99.653 + - type: recall_at_1000 + value: 99.99 + - type: recall_at_20 + value: 98.042 + - type: recall_at_3 + value: 88.408 + - type: recall_at_5 + value: 92.682 + task: + type: Retrieval + - dataset: + config: default + name: MTEB QuoraRetrieval (default) + revision: e4e08e0b7dbe3c8700f0daef558ff32256715259 + split: test + type: mteb/quora + metrics: + - type: main_score + value: 89.042 + - type: map_at_1 + value: 71.18100000000001 + - type: map_at_10 + value: 85.384 + - type: map_at_100 + value: 85.99 + - type: map_at_1000 + value: 86.005 + - type: map_at_20 + value: 85.78 + - type: map_at_3 + value: 82.42699999999999 + - type: map_at_5 + value: 84.33699999999999 + - type: mrr_at_1 + value: 81.89999999999999 + - type: mrr_at_10 + value: 88.17330158730135 + - type: mrr_at_100 + value: 88.25738167262922 + - type: mrr_at_1000 + value: 88.2579183769215 + - type: mrr_at_20 + value: 88.2364860045782 + - type: mrr_at_3 + value: 87.28499999999978 + - type: mrr_at_5 + value: 87.92399999999971 + - type: nauc_map_at_1000_diff1 + value: 76.94224070599522 + - type: nauc_map_at_1000_max + value: 32.86265599982142 + - type: nauc_map_at_1000_std + value: -47.874048772730724 + - type: nauc_map_at_100_diff1 + value: 76.94812573138398 + - type: nauc_map_at_100_max + value: 32.83478858017187 + - type: nauc_map_at_100_std + value: -47.92536084823849 + - type: nauc_map_at_10_diff1 + value: 77.11861821982768 + - type: nauc_map_at_10_max + value: 32.61122251065569 + - type: nauc_map_at_10_std + value: -49.73062037843575 + - type: nauc_map_at_1_diff1 + value: 80.68464416663663 + - type: nauc_map_at_1_max + value: 25.130331560643484 + - type: nauc_map_at_1_std + value: -42.69376370993886 + - type: nauc_map_at_20_diff1 + value: 77.00835848634523 + - type: nauc_map_at_20_max + value: 32.791295950502 + - type: nauc_map_at_20_std + value: -48.647455509031076 + - type: nauc_map_at_3_diff1 + value: 77.62367982394356 + - type: nauc_map_at_3_max + value: 30.56463506638987 + - type: nauc_map_at_3_std + value: -51.3079740029453 + - type: nauc_map_at_5_diff1 + value: 77.36397766805766 + - type: nauc_map_at_5_max + value: 31.603837598707464 + - type: nauc_map_at_5_std + value: -51.250702167521766 + - type: nauc_mrr_at_1000_diff1 + value: 77.6745443611078 + - type: nauc_mrr_at_1000_max + value: 34.81372951189249 + - type: nauc_mrr_at_1000_std + value: -44.685627588340324 + - type: nauc_mrr_at_100_diff1 + value: 77.67442482950652 + - type: nauc_mrr_at_100_max + value: 34.81563217265893 + - type: nauc_mrr_at_100_std + value: -44.685788307308236 + - type: nauc_mrr_at_10_diff1 + value: 77.66351536493457 + - type: nauc_mrr_at_10_max + value: 34.811940902898854 + - type: nauc_mrr_at_10_std + value: -44.913863287383634 + - type: nauc_mrr_at_1_diff1 + value: 78.69027072340837 + - type: nauc_mrr_at_1_max + value: 34.535343075211884 + - type: nauc_mrr_at_1_std + value: -42.05371183106761 + - type: nauc_mrr_at_20_diff1 + value: 77.67317131050916 + - type: nauc_mrr_at_20_max + value: 34.82600643348109 + - type: nauc_mrr_at_20_std + value: -44.719973370227514 + - type: nauc_mrr_at_3_diff1 + value: 77.3979916119859 + - type: nauc_mrr_at_3_max + value: 34.89565554203138 + - type: nauc_mrr_at_3_std + value: -45.10206872930435 + - type: nauc_mrr_at_5_diff1 + value: 77.58369940603552 + - type: nauc_mrr_at_5_max + value: 34.942688637821526 + - type: nauc_mrr_at_5_std + value: -44.940237640780396 + - type: nauc_ndcg_at_1000_diff1 + value: 76.80425318237687 + - type: nauc_ndcg_at_1000_max + value: 33.9609099849118 + - type: nauc_ndcg_at_1000_std + value: -46.1082948244873 + - type: nauc_ndcg_at_100_diff1 + value: 76.81715602849619 + - type: nauc_ndcg_at_100_max + value: 33.91745038088129 + - type: nauc_ndcg_at_100_std + value: -46.25734034012133 + - type: nauc_ndcg_at_10_diff1 + value: 76.72463349884477 + - type: nauc_ndcg_at_10_max + value: 33.60846362779925 + - type: nauc_ndcg_at_10_std + value: -49.37907317686101 + - type: nauc_ndcg_at_1_diff1 + value: 78.65315086044242 + - type: nauc_ndcg_at_1_max + value: 34.44342184935891 + - type: nauc_ndcg_at_1_std + value: -42.06221534316784 + - type: nauc_ndcg_at_20_diff1 + value: 76.7677763470474 + - type: nauc_ndcg_at_20_max + value: 33.78207501816115 + - type: nauc_ndcg_at_20_std + value: -48.03584530160146 + - type: nauc_ndcg_at_3_diff1 + value: 76.26316572734727 + - type: nauc_ndcg_at_3_max + value: 32.7994864609015 + - type: nauc_ndcg_at_3_std + value: -49.15009821852673 + - type: nauc_ndcg_at_5_diff1 + value: 76.5708013213933 + - type: nauc_ndcg_at_5_max + value: 32.784295545465014 + - type: nauc_ndcg_at_5_std + value: -50.22649993066655 + - type: nauc_precision_at_1000_diff1 + value: -44.17641808403434 + - type: nauc_precision_at_1000_max + value: -6.445241484858761 + - type: nauc_precision_at_1000_std + value: 39.380183824975745 + - type: nauc_precision_at_100_diff1 + value: -43.85712333054463 + - type: nauc_precision_at_100_max + value: -6.683686433731103 + - type: nauc_precision_at_100_std + value: 38.1951647498668 + - type: nauc_precision_at_10_diff1 + value: -39.53342587310036 + - type: nauc_precision_at_10_max + value: -2.4287657595731864 + - type: nauc_precision_at_10_std + value: 26.787117372424024 + - type: nauc_precision_at_1_diff1 + value: 78.65315086044242 + - type: nauc_precision_at_1_max + value: 34.44342184935891 + - type: nauc_precision_at_1_std + value: -42.06221534316784 + - type: nauc_precision_at_20_diff1 + value: -42.03094246162483 + - type: nauc_precision_at_20_max + value: -4.506291694734023 + - type: nauc_precision_at_20_std + value: 32.374171897083244 + - type: nauc_precision_at_3_diff1 + value: -20.35984936538199 + - type: nauc_precision_at_3_max + value: 6.128482693131924 + - type: nauc_precision_at_3_std + value: 4.819390672397723 + - type: nauc_precision_at_5_diff1 + value: -32.7587696375733 + - type: nauc_precision_at_5_max + value: 0.6102015658624735 + - type: nauc_precision_at_5_std + value: 16.60913173065126 + - type: nauc_recall_at_1000_diff1 + value: 36.737410642439635 + - type: nauc_recall_at_1000_max + value: -9.39853631116378 + - type: nauc_recall_at_1000_std + value: -0.09207269376465006 + - type: nauc_recall_at_100_diff1 + value: 71.77351345762469 + - type: nauc_recall_at_100_max + value: 38.713535826157326 + - type: nauc_recall_at_100_std + value: -41.989934733197934 + - type: nauc_recall_at_10_diff1 + value: 72.15368607528913 + - type: nauc_recall_at_10_max + value: 31.273908673270263 + - type: nauc_recall_at_10_std + value: -70.80735695744565 + - type: nauc_recall_at_1_diff1 + value: 80.68464416663663 + - type: nauc_recall_at_1_max + value: 25.130331560643484 + - type: nauc_recall_at_1_std + value: -42.69376370993886 + - type: nauc_recall_at_20_diff1 + value: 71.91924866714969 + - type: nauc_recall_at_20_max + value: 32.23809490306376 + - type: nauc_recall_at_20_std + value: -67.85286306800333 + - type: nauc_recall_at_3_diff1 + value: 73.12438187286197 + - type: nauc_recall_at_3_max + value: 28.43745129274971 + - type: nauc_recall_at_3_std + value: -57.890245562569774 + - type: nauc_recall_at_5_diff1 + value: 71.68504720806837 + - type: nauc_recall_at_5_max + value: 28.583877872188424 + - type: nauc_recall_at_5_std + value: -64.27885805779695 + - type: ndcg_at_1 + value: 81.92 + - type: ndcg_at_10 + value: 89.042 + - type: ndcg_at_100 + value: 90.141 + - type: ndcg_at_1000 + value: 90.223 + - type: ndcg_at_20 + value: 89.62599999999999 + - type: ndcg_at_3 + value: 86.299 + - type: ndcg_at_5 + value: 87.90599999999999 + - type: precision_at_1 + value: 81.92 + - type: precision_at_10 + value: 13.526 + - type: precision_at_100 + value: 1.533 + - type: precision_at_1000 + value: 0.157 + - type: precision_at_20 + value: 7.147 + - type: precision_at_3 + value: 37.78 + - type: precision_at_5 + value: 24.898 + - type: recall_at_1 + value: 71.18100000000001 + - type: recall_at_10 + value: 95.979 + - type: recall_at_100 + value: 99.661 + - type: recall_at_1000 + value: 99.992 + - type: recall_at_20 + value: 97.824 + - type: recall_at_3 + value: 88.15899999999999 + - type: recall_at_5 + value: 92.659 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SCIDOCS (default) + revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88 + split: test + type: mteb/scidocs + metrics: + - type: main_score + value: 22.438 + - type: map_at_1 + value: 5.063 + - type: map_at_10 + value: 13.666 + - type: map_at_100 + value: 15.925 + - type: map_at_1000 + value: 16.274 + - type: map_at_20 + value: 14.752 + - type: map_at_3 + value: 9.472999999999999 + - type: map_at_5 + value: 11.675 + - type: mrr_at_1 + value: 25.0 + - type: mrr_at_10 + value: 36.94829365079364 + - type: mrr_at_100 + value: 37.98577637584872 + - type: mrr_at_1000 + value: 38.03079561913509 + - type: mrr_at_20 + value: 37.57252403384367 + - type: mrr_at_3 + value: 33.61666666666666 + - type: mrr_at_5 + value: 35.45666666666662 + - type: nauc_map_at_1000_diff1 + value: 17.3896009045565 + - type: nauc_map_at_1000_max + value: 31.781531552415753 + - type: nauc_map_at_1000_std + value: 19.31247312170496 + - type: nauc_map_at_100_diff1 + value: 17.446238364297383 + - type: nauc_map_at_100_max + value: 31.792529718985836 + - type: nauc_map_at_100_std + value: 19.191529999286434 + - type: nauc_map_at_10_diff1 + value: 17.406307807685565 + - type: nauc_map_at_10_max + value: 30.413445853940935 + - type: nauc_map_at_10_std + value: 14.805927078891182 + - type: nauc_map_at_1_diff1 + value: 26.051674096410654 + - type: nauc_map_at_1_max + value: 23.440663550809504 + - type: nauc_map_at_1_std + value: 6.590054787382868 + - type: nauc_map_at_20_diff1 + value: 17.751717311269044 + - type: nauc_map_at_20_max + value: 31.351894154400924 + - type: nauc_map_at_20_std + value: 17.174153441128816 + - type: nauc_map_at_3_diff1 + value: 20.74346539368042 + - type: nauc_map_at_3_max + value: 27.71724867431406 + - type: nauc_map_at_3_std + value: 8.717384918632796 + - type: nauc_map_at_5_diff1 + value: 19.147573001561188 + - type: nauc_map_at_5_max + value: 29.071995825111095 + - type: nauc_map_at_5_std + value: 11.605397908212584 + - type: nauc_mrr_at_1000_diff1 + value: 24.098006216623737 + - type: nauc_mrr_at_1000_max + value: 26.987579085590042 + - type: nauc_mrr_at_1000_std + value: 12.525556686707558 + - type: nauc_mrr_at_100_diff1 + value: 24.101590206402893 + - type: nauc_mrr_at_100_max + value: 26.998508387559205 + - type: nauc_mrr_at_100_std + value: 12.568669861338808 + - type: nauc_mrr_at_10_diff1 + value: 23.971321759573005 + - type: nauc_mrr_at_10_max + value: 27.04564716411525 + - type: nauc_mrr_at_10_std + value: 12.33777066444399 + - type: nauc_mrr_at_1_diff1 + value: 26.16035307098197 + - type: nauc_mrr_at_1_max + value: 23.60956233909525 + - type: nauc_mrr_at_1_std + value: 6.931518940787054 + - type: nauc_mrr_at_20_diff1 + value: 24.010728303576894 + - type: nauc_mrr_at_20_max + value: 26.96756641712243 + - type: nauc_mrr_at_20_std + value: 12.58761062502782 + - type: nauc_mrr_at_3_diff1 + value: 23.868165796319076 + - type: nauc_mrr_at_3_max + value: 26.34212626099733 + - type: nauc_mrr_at_3_std + value: 11.957402573210297 + - type: nauc_mrr_at_5_diff1 + value: 24.01914162756954 + - type: nauc_mrr_at_5_max + value: 27.075660163094017 + - type: nauc_mrr_at_5_std + value: 11.788065358577441 + - type: nauc_ndcg_at_1000_diff1 + value: 17.63137634115007 + - type: nauc_ndcg_at_1000_max + value: 32.547134337631 + - type: nauc_ndcg_at_1000_std + value: 25.40429199550974 + - type: nauc_ndcg_at_100_diff1 + value: 18.377008979505 + - type: nauc_ndcg_at_100_max + value: 33.00682583446948 + - type: nauc_ndcg_at_100_std + value: 25.578370398312423 + - type: nauc_ndcg_at_10_diff1 + value: 18.371134616100967 + - type: nauc_ndcg_at_10_max + value: 30.54838561855297 + - type: nauc_ndcg_at_10_std + value: 16.696156820844745 + - type: nauc_ndcg_at_1_diff1 + value: 26.16035307098197 + - type: nauc_ndcg_at_1_max + value: 23.60956233909525 + - type: nauc_ndcg_at_1_std + value: 6.931518940787054 + - type: nauc_ndcg_at_20_diff1 + value: 18.90584624310486 + - type: nauc_ndcg_at_20_max + value: 31.88696431445188 + - type: nauc_ndcg_at_20_std + value: 20.447001793088845 + - type: nauc_ndcg_at_3_diff1 + value: 20.89191347568384 + - type: nauc_ndcg_at_3_max + value: 27.74036662565143 + - type: nauc_ndcg_at_3_std + value: 11.014097860476618 + - type: nauc_ndcg_at_5_diff1 + value: 19.909289197975376 + - type: nauc_ndcg_at_5_max + value: 29.368346805220796 + - type: nauc_ndcg_at_5_std + value: 13.27410846006284 + - type: nauc_precision_at_1000_diff1 + value: 2.4282988019269602 + - type: nauc_precision_at_1000_max + value: 24.655150729725367 + - type: nauc_precision_at_1000_std + value: 37.82258020418101 + - type: nauc_precision_at_100_diff1 + value: 10.606286515256018 + - type: nauc_precision_at_100_max + value: 30.2260852734373 + - type: nauc_precision_at_100_std + value: 37.29779532533911 + - type: nauc_precision_at_10_diff1 + value: 13.038228914088684 + - type: nauc_precision_at_10_max + value: 30.616916008558164 + - type: nauc_precision_at_10_std + value: 21.115461015714722 + - type: nauc_precision_at_1_diff1 + value: 26.16035307098197 + - type: nauc_precision_at_1_max + value: 23.60956233909525 + - type: nauc_precision_at_1_std + value: 6.931518940787054 + - type: nauc_precision_at_20_diff1 + value: 13.511620251161816 + - type: nauc_precision_at_20_max + value: 31.295653113322242 + - type: nauc_precision_at_20_std + value: 27.370771969849724 + - type: nauc_precision_at_3_diff1 + value: 18.249687633398455 + - type: nauc_precision_at_3_max + value: 29.014962438110903 + - type: nauc_precision_at_3_std + value: 12.930497262702634 + - type: nauc_precision_at_5_diff1 + value: 16.286276043543673 + - type: nauc_precision_at_5_max + value: 30.195622711423436 + - type: nauc_precision_at_5_std + value: 15.95614204033211 + - type: nauc_recall_at_1000_diff1 + value: 1.9878640319149925 + - type: nauc_recall_at_1000_max + value: 24.019194368065033 + - type: nauc_recall_at_1000_std + value: 39.574541597654196 + - type: nauc_recall_at_100_diff1 + value: 10.305886144318624 + - type: nauc_recall_at_100_max + value: 29.8010281910016 + - type: nauc_recall_at_100_std + value: 37.6689114837801 + - type: nauc_recall_at_10_diff1 + value: 12.780292447702408 + - type: nauc_recall_at_10_max + value: 30.054876102286876 + - type: nauc_recall_at_10_std + value: 20.841914628110775 + - type: nauc_recall_at_1_diff1 + value: 26.051674096410654 + - type: nauc_recall_at_1_max + value: 23.440663550809504 + - type: nauc_recall_at_1_std + value: 6.590054787382868 + - type: nauc_recall_at_20_diff1 + value: 13.275330452383052 + - type: nauc_recall_at_20_max + value: 30.76355251755915 + - type: nauc_recall_at_20_std + value: 27.246020024010036 + - type: nauc_recall_at_3_diff1 + value: 18.156285596934158 + - type: nauc_recall_at_3_max + value: 28.8270289641924 + - type: nauc_recall_at_3_std + value: 12.726004717430769 + - type: nauc_recall_at_5_diff1 + value: 15.991944272752887 + - type: nauc_recall_at_5_max + value: 29.738367880630108 + - type: nauc_recall_at_5_std + value: 15.723376486711063 + - type: ndcg_at_1 + value: 25.0 + - type: ndcg_at_10 + value: 22.438 + - type: ndcg_at_100 + value: 30.968 + - type: ndcg_at_1000 + value: 36.738 + - type: ndcg_at_20 + value: 25.35 + - type: ndcg_at_3 + value: 21.057000000000002 + - type: ndcg_at_5 + value: 18.676000000000002 + - type: precision_at_1 + value: 25.0 + - type: precision_at_10 + value: 11.73 + - type: precision_at_100 + value: 2.405 + - type: precision_at_1000 + value: 0.379 + - type: precision_at_20 + value: 7.5600000000000005 + - type: precision_at_3 + value: 19.8 + - type: precision_at_5 + value: 16.64 + - type: recall_at_1 + value: 5.063 + - type: recall_at_10 + value: 23.767 + - type: recall_at_100 + value: 48.809999999999995 + - type: recall_at_1000 + value: 76.89 + - type: recall_at_20 + value: 30.659999999999997 + - type: recall_at_3 + value: 12.028 + - type: recall_at_5 + value: 16.872 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SciFact (default) + revision: 0228b52cf27578f30900b9e5271d331663a030d7 + split: test + type: mteb/scifact + metrics: + - type: main_score + value: 74.97500000000001 + - type: map_at_1 + value: 60.428000000000004 + - type: map_at_10 + value: 70.459 + - type: map_at_100 + value: 70.976 + - type: map_at_1000 + value: 70.987 + - type: map_at_20 + value: 70.804 + - type: map_at_3 + value: 67.948 + - type: map_at_5 + value: 69.17 + - type: mrr_at_1 + value: 63.66666666666667 + - type: mrr_at_10 + value: 71.7009259259259 + - type: mrr_at_100 + value: 72.1286347931845 + - type: mrr_at_1000 + value: 72.13903486018125 + - type: mrr_at_20 + value: 71.990719757825 + - type: mrr_at_3 + value: 70.0 + - type: mrr_at_5 + value: 70.73333333333332 + - type: nauc_map_at_1000_diff1 + value: 74.49493500371021 + - type: nauc_map_at_1000_max + value: 55.38706216742691 + - type: nauc_map_at_1000_std + value: 7.742922659604426 + - type: nauc_map_at_100_diff1 + value: 74.49193389442027 + - type: nauc_map_at_100_max + value: 55.39820712283397 + - type: nauc_map_at_100_std + value: 7.757866978741148 + - type: nauc_map_at_10_diff1 + value: 74.43239483777734 + - type: nauc_map_at_10_max + value: 55.795105662164424 + - type: nauc_map_at_10_std + value: 7.804970476433032 + - type: nauc_map_at_1_diff1 + value: 77.5441224700081 + - type: nauc_map_at_1_max + value: 49.54683902231174 + - type: nauc_map_at_1_std + value: 0.38667943989648074 + - type: nauc_map_at_20_diff1 + value: 74.47524111514866 + - type: nauc_map_at_20_max + value: 55.500938968925965 + - type: nauc_map_at_20_std + value: 7.817688160909167 + - type: nauc_map_at_3_diff1 + value: 74.41338223429253 + - type: nauc_map_at_3_max + value: 51.32254254023355 + - type: nauc_map_at_3_std + value: 1.9686045613525895 + - type: nauc_map_at_5_diff1 + value: 74.03022514117326 + - type: nauc_map_at_5_max + value: 53.23390857614976 + - type: nauc_map_at_5_std + value: 5.61303023140507 + - type: nauc_mrr_at_1000_diff1 + value: 74.98002817226967 + - type: nauc_mrr_at_1000_max + value: 57.67230766607398 + - type: nauc_mrr_at_1000_std + value: 10.424021164655617 + - type: nauc_mrr_at_100_diff1 + value: 74.97673431923158 + - type: nauc_mrr_at_100_max + value: 57.682736127840606 + - type: nauc_mrr_at_100_std + value: 10.438173855425077 + - type: nauc_mrr_at_10_diff1 + value: 74.93967247650161 + - type: nauc_mrr_at_10_max + value: 58.10416634907251 + - type: nauc_mrr_at_10_std + value: 10.730911849882084 + - type: nauc_mrr_at_1_diff1 + value: 77.94301973207536 + - type: nauc_mrr_at_1_max + value: 56.91045731239722 + - type: nauc_mrr_at_1_std + value: 9.167931543290486 + - type: nauc_mrr_at_20_diff1 + value: 74.94957270321532 + - type: nauc_mrr_at_20_max + value: 57.80563495985277 + - type: nauc_mrr_at_20_std + value: 10.555180814511047 + - type: nauc_mrr_at_3_diff1 + value: 75.22551243537083 + - type: nauc_mrr_at_3_max + value: 56.51697218645902 + - type: nauc_mrr_at_3_std + value: 7.992152535099033 + - type: nauc_mrr_at_5_diff1 + value: 74.70178554480918 + - type: nauc_mrr_at_5_max + value: 56.93334236411708 + - type: nauc_mrr_at_5_std + value: 9.576330817674801 + - type: nauc_ndcg_at_1000_diff1 + value: 74.06140886824686 + - type: nauc_ndcg_at_1000_max + value: 57.59127448868717 + - type: nauc_ndcg_at_1000_std + value: 10.834746659988394 + - type: nauc_ndcg_at_100_diff1 + value: 73.95745500366705 + - type: nauc_ndcg_at_100_max + value: 57.91639635748248 + - type: nauc_ndcg_at_100_std + value: 11.372619369212634 + - type: nauc_ndcg_at_10_diff1 + value: 73.56801609109546 + - type: nauc_ndcg_at_10_max + value: 59.68246464463658 + - type: nauc_ndcg_at_10_std + value: 12.172882653515906 + - type: nauc_ndcg_at_1_diff1 + value: 77.94301973207536 + - type: nauc_ndcg_at_1_max + value: 56.91045731239722 + - type: nauc_ndcg_at_1_std + value: 9.167931543290486 + - type: nauc_ndcg_at_20_diff1 + value: 73.6735665321717 + - type: nauc_ndcg_at_20_max + value: 58.749860352253094 + - type: nauc_ndcg_at_20_std + value: 12.329040862446139 + - type: nauc_ndcg_at_3_diff1 + value: 73.5457200101784 + - type: nauc_ndcg_at_3_max + value: 53.87973443142293 + - type: nauc_ndcg_at_3_std + value: 4.220097142536938 + - type: nauc_ndcg_at_5_diff1 + value: 72.7978750157529 + - type: nauc_ndcg_at_5_max + value: 55.03545084016862 + - type: nauc_ndcg_at_5_std + value: 8.159538619361104 + - type: nauc_precision_at_1000_diff1 + value: -21.107079131498136 + - type: nauc_precision_at_1000_max + value: 23.397956387791865 + - type: nauc_precision_at_1000_std + value: 51.81338839992782 + - type: nauc_precision_at_100_diff1 + value: -13.058901421967953 + - type: nauc_precision_at_100_max + value: 28.4551512169639 + - type: nauc_precision_at_100_std + value: 51.49819900261797 + - type: nauc_precision_at_10_diff1 + value: 12.117676343383437 + - type: nauc_precision_at_10_max + value: 49.3854818133732 + - type: nauc_precision_at_10_std + value: 49.40052065507124 + - type: nauc_precision_at_1_diff1 + value: 77.94301973207536 + - type: nauc_precision_at_1_max + value: 56.91045731239722 + - type: nauc_precision_at_1_std + value: 9.167931543290486 + - type: nauc_precision_at_20_diff1 + value: 2.3491501377485653 + - type: nauc_precision_at_20_max + value: 40.91051563625563 + - type: nauc_precision_at_20_std + value: 52.163560678130125 + - type: nauc_precision_at_3_diff1 + value: 41.51381221432537 + - type: nauc_precision_at_3_max + value: 47.78934416940529 + - type: nauc_precision_at_3_std + value: 20.63710491238864 + - type: nauc_precision_at_5_diff1 + value: 28.07004279384857 + - type: nauc_precision_at_5_max + value: 46.410911675311134 + - type: nauc_precision_at_5_std + value: 35.09297203026323 + - type: nauc_recall_at_1000_diff1 + value: 55.4154995331476 + - type: nauc_recall_at_1000_max + value: 100.0 + - type: nauc_recall_at_1000_std + value: 72.2222222222207 + - type: nauc_recall_at_100_diff1 + value: 64.62807345160296 + - type: nauc_recall_at_100_max + value: 83.60825811806248 + - type: nauc_recall_at_100_std + value: 53.65701836290058 + - type: nauc_recall_at_10_diff1 + value: 66.87506727422773 + - type: nauc_recall_at_10_max + value: 77.28591418627656 + - type: nauc_recall_at_10_std + value: 31.001245534272666 + - type: nauc_recall_at_1_diff1 + value: 77.5441224700081 + - type: nauc_recall_at_1_max + value: 49.54683902231174 + - type: nauc_recall_at_1_std + value: 0.38667943989648074 + - type: nauc_recall_at_20_diff1 + value: 65.00198504598687 + - type: nauc_recall_at_20_max + value: 76.98118617525755 + - type: nauc_recall_at_20_std + value: 40.88275730239587 + - type: nauc_recall_at_3_diff1 + value: 68.94113666263492 + - type: nauc_recall_at_3_max + value: 51.549294352038736 + - type: nauc_recall_at_3_std + value: -0.7295825993918809 + - type: nauc_recall_at_5_diff1 + value: 65.91499162888101 + - type: nauc_recall_at_5_max + value: 56.089008758487346 + - type: nauc_recall_at_5_std + value: 12.439742316334371 + - type: ndcg_at_1 + value: 63.666999999999994 + - type: ndcg_at_10 + value: 74.97500000000001 + - type: ndcg_at_100 + value: 77.132 + - type: ndcg_at_1000 + value: 77.422 + - type: ndcg_at_20 + value: 76.065 + - type: ndcg_at_3 + value: 70.814 + - type: ndcg_at_5 + value: 72.346 + - type: precision_at_1 + value: 63.666999999999994 + - type: precision_at_10 + value: 9.9 + - type: precision_at_100 + value: 1.0999999999999999 + - type: precision_at_1000 + value: 0.11199999999999999 + - type: precision_at_20 + value: 5.183 + - type: precision_at_3 + value: 27.667 + - type: precision_at_5 + value: 17.8 + - type: recall_at_1 + value: 60.428000000000004 + - type: recall_at_10 + value: 87.533 + - type: recall_at_100 + value: 97.0 + - type: recall_at_1000 + value: 99.333 + - type: recall_at_20 + value: 91.533 + - type: recall_at_3 + value: 76.017 + - type: recall_at_5 + value: 80.011 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SciFact (default) + revision: 0228b52cf27578f30900b9e5271d331663a030d7 + split: train + type: mteb/scifact + metrics: + - type: main_score + value: 78.062 + - type: map_at_1 + value: 64.24 + - type: map_at_10 + value: 73.923 + - type: map_at_100 + value: 74.337 + - type: map_at_1000 + value: 74.35000000000001 + - type: map_at_20 + value: 74.188 + - type: map_at_3 + value: 71.307 + - type: map_at_5 + value: 72.926 + - type: mrr_at_1 + value: 67.49072929542646 + - type: mrr_at_10 + value: 75.1880138128593 + - type: mrr_at_100 + value: 75.48796055172588 + - type: mrr_at_1000 + value: 75.49858394191004 + - type: mrr_at_20 + value: 75.3483873956106 + - type: mrr_at_3 + value: 73.52698805109189 + - type: mrr_at_5 + value: 74.57148743304491 + - type: nauc_map_at_1000_diff1 + value: 72.704936020183 + - type: nauc_map_at_1000_max + value: 54.25394358790662 + - type: nauc_map_at_1000_std + value: 13.156859241275972 + - type: nauc_map_at_100_diff1 + value: 72.69463393786388 + - type: nauc_map_at_100_max + value: 54.267995153498624 + - type: nauc_map_at_100_std + value: 13.176372961544022 + - type: nauc_map_at_10_diff1 + value: 72.56547255677417 + - type: nauc_map_at_10_max + value: 54.13224272095242 + - type: nauc_map_at_10_std + value: 12.737271686006663 + - type: nauc_map_at_1_diff1 + value: 74.8291465411949 + - type: nauc_map_at_1_max + value: 46.151834291555275 + - type: nauc_map_at_1_std + value: 6.031949825020963 + - type: nauc_map_at_20_diff1 + value: 72.63735079172274 + - type: nauc_map_at_20_max + value: 54.18641620867146 + - type: nauc_map_at_20_std + value: 13.189598766924204 + - type: nauc_map_at_3_diff1 + value: 73.6672498055705 + - type: nauc_map_at_3_max + value: 51.49532971827955 + - type: nauc_map_at_3_std + value: 8.989297153170508 + - type: nauc_map_at_5_diff1 + value: 72.72108768492257 + - type: nauc_map_at_5_max + value: 52.93123375743044 + - type: nauc_map_at_5_std + value: 10.706382204328833 + - type: nauc_mrr_at_1000_diff1 + value: 72.35177939549008 + - type: nauc_mrr_at_1000_max + value: 55.72330584461652 + - type: nauc_mrr_at_1000_std + value: 16.285823507669992 + - type: nauc_mrr_at_100_diff1 + value: 72.34036221786981 + - type: nauc_mrr_at_100_max + value: 55.7326283458132 + - type: nauc_mrr_at_100_std + value: 16.299182234650438 + - type: nauc_mrr_at_10_diff1 + value: 72.17103067754233 + - type: nauc_mrr_at_10_max + value: 55.71478856890253 + - type: nauc_mrr_at_10_std + value: 16.348706749764222 + - type: nauc_mrr_at_1_diff1 + value: 74.22009810416182 + - type: nauc_mrr_at_1_max + value: 52.866494006642576 + - type: nauc_mrr_at_1_std + value: 12.919496136122405 + - type: nauc_mrr_at_20_diff1 + value: 72.2625755808264 + - type: nauc_mrr_at_20_max + value: 55.64526652001681 + - type: nauc_mrr_at_20_std + value: 16.362968600732863 + - type: nauc_mrr_at_3_diff1 + value: 73.05245441977691 + - type: nauc_mrr_at_3_max + value: 56.230679352704996 + - type: nauc_mrr_at_3_std + value: 15.865956698877289 + - type: nauc_mrr_at_5_diff1 + value: 72.38153532728295 + - type: nauc_mrr_at_5_max + value: 55.4753491067377 + - type: nauc_mrr_at_5_std + value: 15.587419908618461 + - type: nauc_ndcg_at_1000_diff1 + value: 71.8539632541432 + - type: nauc_ndcg_at_1000_max + value: 55.79719159954046 + - type: nauc_ndcg_at_1000_std + value: 16.151083310206456 + - type: nauc_ndcg_at_100_diff1 + value: 71.5114198914329 + - type: nauc_ndcg_at_100_max + value: 56.16106745245034 + - type: nauc_ndcg_at_100_std + value: 16.713945939778302 + - type: nauc_ndcg_at_10_diff1 + value: 70.86463880660648 + - type: nauc_ndcg_at_10_max + value: 55.84050982919621 + - type: nauc_ndcg_at_10_std + value: 16.4034482797237 + - type: nauc_ndcg_at_1_diff1 + value: 74.22009810416182 + - type: nauc_ndcg_at_1_max + value: 52.866494006642576 + - type: nauc_ndcg_at_1_std + value: 12.919496136122405 + - type: nauc_ndcg_at_20_diff1 + value: 71.128145573512 + - type: nauc_ndcg_at_20_max + value: 55.78331123482868 + - type: nauc_ndcg_at_20_std + value: 17.35039598104976 + - type: nauc_ndcg_at_3_diff1 + value: 72.81016213001276 + - type: nauc_ndcg_at_3_max + value: 53.665879366774796 + - type: nauc_ndcg_at_3_std + value: 11.776394857979009 + - type: nauc_ndcg_at_5_diff1 + value: 71.41544523318102 + - type: nauc_ndcg_at_5_max + value: 53.97771739060286 + - type: nauc_ndcg_at_5_std + value: 12.614626718533234 + - type: nauc_precision_at_1000_diff1 + value: -31.471423338557663 + - type: nauc_precision_at_1000_max + value: 18.000249574871916 + - type: nauc_precision_at_1000_std + value: 39.19692872985424 + - type: nauc_precision_at_100_diff1 + value: -23.86904452473403 + - type: nauc_precision_at_100_max + value: 25.240899302231234 + - type: nauc_precision_at_100_std + value: 42.056018672943566 + - type: nauc_precision_at_10_diff1 + value: 0.4752190460312635 + - type: nauc_precision_at_10_max + value: 38.815380362627735 + - type: nauc_precision_at_10_std + value: 39.400357666709446 + - type: nauc_precision_at_1_diff1 + value: 74.22009810416182 + - type: nauc_precision_at_1_max + value: 52.866494006642576 + - type: nauc_precision_at_1_std + value: 12.919496136122405 + - type: nauc_precision_at_20_diff1 + value: -7.018570976551137 + - type: nauc_precision_at_20_max + value: 32.65180618931654 + - type: nauc_precision_at_20_std + value: 42.21835162061081 + - type: nauc_precision_at_3_diff1 + value: 38.04845187399018 + - type: nauc_precision_at_3_max + value: 51.15303813839884 + - type: nauc_precision_at_3_std + value: 24.81720425894165 + - type: nauc_precision_at_5_diff1 + value: 16.968325235777122 + - type: nauc_precision_at_5_max + value: 44.17970232579579 + - type: nauc_precision_at_5_std + value: 30.440676310667914 + - type: nauc_recall_at_1000_diff1 + value: 63.8080653413978 + - type: nauc_recall_at_1000_max + value: 100.0 + - type: nauc_recall_at_1000_std + value: 100.0 + - type: nauc_recall_at_100_diff1 + value: 52.74274243000091 + - type: nauc_recall_at_100_max + value: 77.89706586648443 + - type: nauc_recall_at_100_std + value: 53.59588178879462 + - type: nauc_recall_at_10_diff1 + value: 60.32462525476192 + - type: nauc_recall_at_10_max + value: 59.36778228420133 + - type: nauc_recall_at_10_std + value: 28.626643650555554 + - type: nauc_recall_at_1_diff1 + value: 74.8291465411949 + - type: nauc_recall_at_1_max + value: 46.151834291555275 + - type: nauc_recall_at_1_std + value: 6.031949825020963 + - type: nauc_recall_at_20_diff1 + value: 60.187154368758954 + - type: nauc_recall_at_20_max + value: 60.20725625748765 + - type: nauc_recall_at_20_std + value: 39.377322568538794 + - type: nauc_recall_at_3_diff1 + value: 70.56428275743455 + - type: nauc_recall_at_3_max + value: 52.195799388933665 + - type: nauc_recall_at_3_std + value: 9.652140170622282 + - type: nauc_recall_at_5_diff1 + value: 64.60941882295688 + - type: nauc_recall_at_5_max + value: 51.887676330901925 + - type: nauc_recall_at_5_std + value: 11.7171373609305 + - type: ndcg_at_1 + value: 67.491 + - type: ndcg_at_10 + value: 78.062 + - type: ndcg_at_100 + value: 79.851 + - type: ndcg_at_1000 + value: 80.17 + - type: ndcg_at_20 + value: 78.85 + - type: ndcg_at_3 + value: 73.831 + - type: ndcg_at_5 + value: 76.142 + - type: precision_at_1 + value: 67.491 + - type: precision_at_10 + value: 10.087 + - type: precision_at_100 + value: 1.106 + - type: precision_at_1000 + value: 0.11299999999999999 + - type: precision_at_20 + value: 5.235 + - type: precision_at_3 + value: 28.719 + - type: precision_at_5 + value: 18.714 + - type: recall_at_1 + value: 64.24 + - type: recall_at_10 + value: 89.20700000000001 + - type: recall_at_100 + value: 97.342 + - type: recall_at_1000 + value: 99.753 + - type: recall_at_20 + value: 92.147 + - type: recall_at_3 + value: 78.41799999999999 + - type: recall_at_5 + value: 83.83 + task: + type: Retrieval + - dataset: + config: default + name: MTEB TRECCOVID (default) + revision: bb9466bac8153a0349341eb1b22e06409e78ef4e + split: test + type: mteb/trec-covid + metrics: + - type: main_score + value: 84.724 + - type: map_at_1 + value: 0.243 + - type: map_at_10 + value: 2.2089999999999996 + - type: map_at_100 + value: 13.372 + - type: map_at_1000 + value: 32.083 + - type: map_at_20 + value: 3.893 + - type: map_at_3 + value: 0.719 + - type: map_at_5 + value: 1.1860000000000002 + - type: mrr_at_1 + value: 94.0 + - type: mrr_at_10 + value: 97.0 + - type: mrr_at_100 + value: 97.0 + - type: mrr_at_1000 + value: 97.0 + - type: mrr_at_20 + value: 97.0 + - type: mrr_at_3 + value: 97.0 + - type: mrr_at_5 + value: 97.0 + - type: nauc_map_at_1000_diff1 + value: -22.408703052625835 + - type: nauc_map_at_1000_max + value: 53.302695352035755 + - type: nauc_map_at_1000_std + value: 78.67866818015943 + - type: nauc_map_at_100_diff1 + value: -6.245766059343929 + - type: nauc_map_at_100_max + value: 27.694365832005907 + - type: nauc_map_at_100_std + value: 52.09571817392694 + - type: nauc_map_at_10_diff1 + value: 30.73952826767339 + - type: nauc_map_at_10_max + value: 3.003001619774156 + - type: nauc_map_at_10_std + value: 3.3901778142194092 + - type: nauc_map_at_1_diff1 + value: 33.494599532067284 + - type: nauc_map_at_1_max + value: -1.5973770957299893 + - type: nauc_map_at_1_std + value: -0.7201049676889548 + - type: nauc_map_at_20_diff1 + value: 18.37398220019707 + - type: nauc_map_at_20_max + value: 11.190427313812439 + - type: nauc_map_at_20_std + value: 17.8716632215046 + - type: nauc_map_at_3_diff1 + value: 26.61747655824615 + - type: nauc_map_at_3_max + value: 0.007332013543592022 + - type: nauc_map_at_3_std + value: 0.6854775555995046 + - type: nauc_map_at_5_diff1 + value: 31.151578388840413 + - type: nauc_map_at_5_max + value: 0.6231581595979546 + - type: nauc_map_at_5_std + value: -0.6002965206725245 + - type: nauc_mrr_at_1000_diff1 + value: 11.624649859943707 + - type: nauc_mrr_at_1000_max + value: 22.128851540616388 + - type: nauc_mrr_at_1000_std + value: 66.61998132586402 + - type: nauc_mrr_at_100_diff1 + value: 11.624649859943707 + - type: nauc_mrr_at_100_max + value: 22.128851540616388 + - type: nauc_mrr_at_100_std + value: 66.61998132586402 + - type: nauc_mrr_at_10_diff1 + value: 11.624649859943707 + - type: nauc_mrr_at_10_max + value: 22.128851540616388 + - type: nauc_mrr_at_10_std + value: 66.61998132586402 + - type: nauc_mrr_at_1_diff1 + value: 11.624649859944117 + - type: nauc_mrr_at_1_max + value: 22.12885154061634 + - type: nauc_mrr_at_1_std + value: 66.61998132586365 + - type: nauc_mrr_at_20_diff1 + value: 11.624649859943707 + - type: nauc_mrr_at_20_max + value: 22.128851540616388 + - type: nauc_mrr_at_20_std + value: 66.61998132586402 + - type: nauc_mrr_at_3_diff1 + value: 11.624649859943707 + - type: nauc_mrr_at_3_max + value: 22.128851540616388 + - type: nauc_mrr_at_3_std + value: 66.61998132586402 + - type: nauc_mrr_at_5_diff1 + value: 11.624649859943707 + - type: nauc_mrr_at_5_max + value: 22.128851540616388 + - type: nauc_mrr_at_5_std + value: 66.61998132586402 + - type: nauc_ndcg_at_1000_diff1 + value: -16.634440227947607 + - type: nauc_ndcg_at_1000_max + value: 52.02407283868307 + - type: nauc_ndcg_at_1000_std + value: 75.04864144471006 + - type: nauc_ndcg_at_100_diff1 + value: -39.907639941509295 + - type: nauc_ndcg_at_100_max + value: 47.48611610046854 + - type: nauc_ndcg_at_100_std + value: 76.64908580965817 + - type: nauc_ndcg_at_10_diff1 + value: -22.144869126347576 + - type: nauc_ndcg_at_10_max + value: 39.32109886166166 + - type: nauc_ndcg_at_10_std + value: 60.429249763186476 + - type: nauc_ndcg_at_1_diff1 + value: 3.763183125599249 + - type: nauc_ndcg_at_1_max + value: 23.053691275167807 + - type: nauc_ndcg_at_1_std + value: 47.61265580057534 + - type: nauc_ndcg_at_20_diff1 + value: -33.888360753754284 + - type: nauc_ndcg_at_20_max + value: 42.545826213560886 + - type: nauc_ndcg_at_20_std + value: 67.81771052556327 + - type: nauc_ndcg_at_3_diff1 + value: -31.243868114117358 + - type: nauc_ndcg_at_3_max + value: 41.09132875179433 + - type: nauc_ndcg_at_3_std + value: 65.67663873391709 + - type: nauc_ndcg_at_5_diff1 + value: -26.976609750751955 + - type: nauc_ndcg_at_5_max + value: 35.005828383626884 + - type: nauc_ndcg_at_5_std + value: 59.41865420459437 + - type: nauc_precision_at_1000_diff1 + value: -32.40893581977419 + - type: nauc_precision_at_1000_max + value: 44.787097245654515 + - type: nauc_precision_at_1000_std + value: 45.92800088788868 + - type: nauc_precision_at_100_diff1 + value: -40.645397158960535 + - type: nauc_precision_at_100_max + value: 48.56851812763327 + - type: nauc_precision_at_100_std + value: 74.9665015749215 + - type: nauc_precision_at_10_diff1 + value: -4.547782341068627 + - type: nauc_precision_at_10_max + value: 40.35265647122632 + - type: nauc_precision_at_10_std + value: 53.30715856487296 + - type: nauc_precision_at_1_diff1 + value: 11.624649859944117 + - type: nauc_precision_at_1_max + value: 22.12885154061634 + - type: nauc_precision_at_1_std + value: 66.61998132586365 + - type: nauc_precision_at_20_diff1 + value: -29.626954018858626 + - type: nauc_precision_at_20_max + value: 43.29723780310212 + - type: nauc_precision_at_20_std + value: 67.38164309677705 + - type: nauc_precision_at_3_diff1 + value: -38.40659340659336 + - type: nauc_precision_at_3_max + value: 50.18981018980998 + - type: nauc_precision_at_3_std + value: 69.53546453546438 + - type: nauc_precision_at_5_diff1 + value: -4.730807427247311 + - type: nauc_precision_at_5_max + value: 43.62153540582012 + - type: nauc_precision_at_5_std + value: 52.12916428420928 + - type: nauc_recall_at_1000_diff1 + value: -4.046307822916301 + - type: nauc_recall_at_1000_max + value: 45.08728754223066 + - type: nauc_recall_at_1000_std + value: 61.95773371253187 + - type: nauc_recall_at_100_diff1 + value: 2.7756446498351073 + - type: nauc_recall_at_100_max + value: 18.057275017004137 + - type: nauc_recall_at_100_std + value: 37.937338250515076 + - type: nauc_recall_at_10_diff1 + value: 33.41888630205771 + - type: nauc_recall_at_10_max + value: -0.1417030905304976 + - type: nauc_recall_at_10_std + value: -1.1529503277229434 + - type: nauc_recall_at_1_diff1 + value: 33.494599532067284 + - type: nauc_recall_at_1_max + value: -1.5973770957299893 + - type: nauc_recall_at_1_std + value: -0.7201049676889548 + - type: nauc_recall_at_20_diff1 + value: 22.608423310994123 + - type: nauc_recall_at_20_max + value: 6.1708789558734 + - type: nauc_recall_at_20_std + value: 10.204449131302491 + - type: nauc_recall_at_3_diff1 + value: 27.296473899945827 + - type: nauc_recall_at_3_max + value: -2.8531911237138616 + - type: nauc_recall_at_3_std + value: -3.727362878844563 + - type: nauc_recall_at_5_diff1 + value: 33.28834761941098 + - type: nauc_recall_at_5_max + value: -2.8936383088858064 + - type: nauc_recall_at_5_std + value: -5.8975808961221245 + - type: ndcg_at_1 + value: 90.0 + - type: ndcg_at_10 + value: 84.724 + - type: ndcg_at_100 + value: 64.754 + - type: ndcg_at_1000 + value: 57.318999999999996 + - type: ndcg_at_20 + value: 80.24900000000001 + - type: ndcg_at_3 + value: 88.592 + - type: ndcg_at_5 + value: 87.94800000000001 + - type: precision_at_1 + value: 94.0 + - type: precision_at_10 + value: 88.8 + - type: precision_at_100 + value: 66.82000000000001 + - type: precision_at_1000 + value: 25.197999999999997 + - type: precision_at_20 + value: 83.2 + - type: precision_at_3 + value: 93.333 + - type: precision_at_5 + value: 93.2 + - type: recall_at_1 + value: 0.243 + - type: recall_at_10 + value: 2.318 + - type: recall_at_100 + value: 16.089000000000002 + - type: recall_at_1000 + value: 53.632999999999996 + - type: recall_at_20 + value: 4.2299999999999995 + - type: recall_at_3 + value: 0.743 + - type: recall_at_5 + value: 1.233 + task: + type: Retrieval + - dataset: + config: default + name: MTEB Touche2020 (default) + revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f + split: test + type: mteb/touche2020 + metrics: + - type: main_score + value: 32.917 + - type: map_at_1 + value: 3.383 + - type: map_at_10 + value: 14.066 + - type: map_at_100 + value: 20.66 + - type: map_at_1000 + value: 22.284000000000002 + - type: map_at_20 + value: 16.653000000000002 + - type: map_at_3 + value: 7.661999999999999 + - type: map_at_5 + value: 10.173 + - type: mrr_at_1 + value: 42.857142857142854 + - type: mrr_at_10 + value: 59.71655328798185 + - type: mrr_at_100 + value: 60.194024496245376 + - type: mrr_at_1000 + value: 60.194024496245376 + - type: mrr_at_20 + value: 59.964152327597695 + - type: mrr_at_3 + value: 56.4625850340136 + - type: mrr_at_5 + value: 58.8095238095238 + - type: nauc_map_at_1000_diff1 + value: 12.579510790257872 + - type: nauc_map_at_1000_max + value: -4.264942252703021 + - type: nauc_map_at_1000_std + value: -2.438273508875829 + - type: nauc_map_at_100_diff1 + value: 13.003922436075754 + - type: nauc_map_at_100_max + value: -5.3301687094103105 + - type: nauc_map_at_100_std + value: -5.318225960144131 + - type: nauc_map_at_10_diff1 + value: 10.693503493056797 + - type: nauc_map_at_10_max + value: -4.32660381229821 + - type: nauc_map_at_10_std + value: -21.5903282626883 + - type: nauc_map_at_1_diff1 + value: 5.397462315448758 + - type: nauc_map_at_1_max + value: -20.4324316949736 + - type: nauc_map_at_1_std + value: -21.84417049994903 + - type: nauc_map_at_20_diff1 + value: 13.557335341043464 + - type: nauc_map_at_20_max + value: -5.489171183784985 + - type: nauc_map_at_20_std + value: -17.692719893249055 + - type: nauc_map_at_3_diff1 + value: 3.3472459083131962 + - type: nauc_map_at_3_max + value: -18.108103266914675 + - type: nauc_map_at_3_std + value: -25.92201988690831 + - type: nauc_map_at_5_diff1 + value: 9.368231305307056 + - type: nauc_map_at_5_max + value: -9.15825678300707 + - type: nauc_map_at_5_std + value: -27.398604293720474 + - type: nauc_mrr_at_1000_diff1 + value: -6.149664958666229 + - type: nauc_mrr_at_1000_max + value: -33.30488479921579 + - type: nauc_mrr_at_1000_std + value: -12.827341694953992 + - type: nauc_mrr_at_100_diff1 + value: -6.149664958666229 + - type: nauc_mrr_at_100_max + value: -33.30488479921579 + - type: nauc_mrr_at_100_std + value: -12.827341694953992 + - type: nauc_mrr_at_10_diff1 + value: -7.821173168559035 + - type: nauc_mrr_at_10_max + value: -34.627605757871386 + - type: nauc_mrr_at_10_std + value: -11.839825079324873 + - type: nauc_mrr_at_1_diff1 + value: 1.1039706185822074 + - type: nauc_mrr_at_1_max + value: -25.32875826505669 + - type: nauc_mrr_at_1_std + value: -15.520024220257683 + - type: nauc_mrr_at_20_diff1 + value: -6.627782202862055 + - type: nauc_mrr_at_20_max + value: -33.91986756503583 + - type: nauc_mrr_at_20_std + value: -12.77542710186697 + - type: nauc_mrr_at_3_diff1 + value: -5.684443689873644 + - type: nauc_mrr_at_3_max + value: -32.31420774810655 + - type: nauc_mrr_at_3_std + value: -16.620509048596986 + - type: nauc_mrr_at_5_diff1 + value: -5.8341673661904245 + - type: nauc_mrr_at_5_max + value: -31.742534982165733 + - type: nauc_mrr_at_5_std + value: -12.338506564576873 + - type: nauc_ndcg_at_1000_diff1 + value: 6.873644758353055 + - type: nauc_ndcg_at_1000_max + value: -10.190425519023206 + - type: nauc_ndcg_at_1000_std + value: 19.779751696545933 + - type: nauc_ndcg_at_100_diff1 + value: 8.844192480674973 + - type: nauc_ndcg_at_100_max + value: -15.48908041052085 + - type: nauc_ndcg_at_100_std + value: 14.216198715508671 + - type: nauc_ndcg_at_10_diff1 + value: 8.692174921685304 + - type: nauc_ndcg_at_10_max + value: -14.44537192758476 + - type: nauc_ndcg_at_10_std + value: -12.380881855102194 + - type: nauc_ndcg_at_1_diff1 + value: 1.5840183579660068 + - type: nauc_ndcg_at_1_max + value: -24.26693497249263 + - type: nauc_ndcg_at_1_std + value: -14.773679317054524 + - type: nauc_ndcg_at_20_diff1 + value: 9.38975780162556 + - type: nauc_ndcg_at_20_max + value: -17.25024380139134 + - type: nauc_ndcg_at_20_std + value: -12.070137131016175 + - type: nauc_ndcg_at_3_diff1 + value: 1.0981734851532068 + - type: nauc_ndcg_at_3_max + value: -18.668259534384973 + - type: nauc_ndcg_at_3_std + value: -15.895758732090936 + - type: nauc_ndcg_at_5_diff1 + value: 10.66125285870417 + - type: nauc_ndcg_at_5_max + value: -12.554242167548388 + - type: nauc_ndcg_at_5_std + value: -16.186943323915116 + - type: nauc_precision_at_1000_diff1 + value: -10.51346076213006 + - type: nauc_precision_at_1000_max + value: 32.714946388398 + - type: nauc_precision_at_1000_std + value: 38.901329108718585 + - type: nauc_precision_at_100_diff1 + value: 1.9694881127973165 + - type: nauc_precision_at_100_max + value: 4.920837162244592 + - type: nauc_precision_at_100_std + value: 68.50058610152277 + - type: nauc_precision_at_10_diff1 + value: 7.990315555541806 + - type: nauc_precision_at_10_max + value: -10.404436976147956 + - type: nauc_precision_at_10_std + value: -3.0053809521608046 + - type: nauc_precision_at_1_diff1 + value: 1.1039706185822074 + - type: nauc_precision_at_1_max + value: -25.32875826505669 + - type: nauc_precision_at_1_std + value: -15.520024220257683 + - type: nauc_precision_at_20_diff1 + value: 12.709085196921944 + - type: nauc_precision_at_20_max + value: -8.126579722574737 + - type: nauc_precision_at_20_std + value: 8.554621584480888 + - type: nauc_precision_at_3_diff1 + value: -0.5660657946405515 + - type: nauc_precision_at_3_max + value: -18.2180922088033 + - type: nauc_precision_at_3_std + value: -19.74781894851289 + - type: nauc_precision_at_5_diff1 + value: 11.960508196663335 + - type: nauc_precision_at_5_max + value: -7.505470678471681 + - type: nauc_precision_at_5_std + value: -19.679551739925007 + - type: nauc_recall_at_1000_diff1 + value: 2.0894293955487906 + - type: nauc_recall_at_1000_max + value: 2.565223878252418 + - type: nauc_recall_at_1000_std + value: 72.80850465698563 + - type: nauc_recall_at_100_diff1 + value: 8.370601331640701 + - type: nauc_recall_at_100_max + value: -13.813786821955048 + - type: nauc_recall_at_100_std + value: 33.05653636954439 + - type: nauc_recall_at_10_diff1 + value: 8.055635577653009 + - type: nauc_recall_at_10_max + value: -12.159484753499648 + - type: nauc_recall_at_10_std + value: -18.3886058528649 + - type: nauc_recall_at_1_diff1 + value: 5.397462315448758 + - type: nauc_recall_at_1_max + value: -20.4324316949736 + - type: nauc_recall_at_1_std + value: -21.84417049994903 + - type: nauc_recall_at_20_diff1 + value: 11.648146655416976 + - type: nauc_recall_at_20_max + value: -14.51749970488514 + - type: nauc_recall_at_20_std + value: -12.087956357152999 + - type: nauc_recall_at_3_diff1 + value: 0.7233105449826043 + - type: nauc_recall_at_3_max + value: -22.734765025405245 + - type: nauc_recall_at_3_std + value: -28.94891737849546 + - type: nauc_recall_at_5_diff1 + value: 9.153063794746425 + - type: nauc_recall_at_5_max + value: -12.070313664412732 + - type: nauc_recall_at_5_std + value: -28.58815211049064 + - type: ndcg_at_1 + value: 39.796 + - type: ndcg_at_10 + value: 32.917 + - type: ndcg_at_100 + value: 43.009 + - type: ndcg_at_1000 + value: 54.128 + - type: ndcg_at_20 + value: 32.012 + - type: ndcg_at_3 + value: 39.913 + - type: ndcg_at_5 + value: 35.55 + - type: precision_at_1 + value: 42.857 + - type: precision_at_10 + value: 28.366999999999997 + - type: precision_at_100 + value: 8.388 + - type: precision_at_1000 + value: 1.5859999999999999 + - type: precision_at_20 + value: 19.694 + - type: precision_at_3 + value: 42.857 + - type: precision_at_5 + value: 35.510000000000005 + - type: recall_at_1 + value: 3.383 + - type: recall_at_10 + value: 20.212 + - type: recall_at_100 + value: 51.67699999999999 + - type: recall_at_1000 + value: 85.461 + - type: recall_at_20 + value: 27.192 + - type: recall_at_3 + value: 8.981 + - type: recall_at_5 + value: 12.887 + task: + type: Retrieval +tags: +- mteb license: mit --- +
+ Acknowledgement | + Combination of Embedding Models | + Usage | + Citation | + License +
+ + +## Acknowledgement + +First of all, we want to acknowledge the original creators of the [Snowflake/snowflake-arctic-embed-l](https://huggingface.co/Snowflake/snowflake-arctic-embed-l) and [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) models which are used to create this model. Our model is just a combination of these two models, and we have not made any changes to the original models. + +Furthermore, we want to acknowledge the team of Marqo, who has worked on the idea of combining two models through concatenation in parallel to ourselves. Their initial effort allowed to re-use existing pieces of code, in particular the [modeling script](https://huggingface.co/PaDaS-Lab/arctic-l-bge-small/blob/main/modeling_arctic_l_bge_small.py) for bringing the combined model to HuggingFace. + +## Combination of Embedding Models + +### Overview + +Embedding models have become increasingly powerful and applicable across various use cases. However, the next significant challenge lies in enhancing their efficiency in terms of resource consumption. Our goal is to experiment with combining two embedding models to achieve better performance with fewer resources. + +### Key Insights + +1. **Diversity Matters**: Initial findings suggest that combining models with differing characteristics can complement each other, resulting in improved outcomes. To design an effective combination, the diversity of the models—evaluated by factors like MTEB performance, architecture, and training data—is crucial. +2. **Combination Technique**: + - We combine the embeddings of two models using the most straightforward approach: concatenation. + - Prior to concatenation, we normalize the embeddings to ensure they are on the same scale. This step is vital for achieving coherent and meaningful results. + +### Implementation + +We combined the following models: + +- **[Snowflake/snowflake-arctic-embed-l](https://huggingface.co/Snowflake/snowflake-arctic-embed-l)** +- **[BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)** + +#### Model Details + +- **Output Embedding Dimensions**: 1408 (1024 + 384) +- **Total Parameters**: 367M (334M + 33M) + + + +### Future Directions + +While the results are promising, we acknowledge the complexity of model combinations and the importance of focusing on more than leaderboard rankings. The simplicity of concatenating embeddings yielding tangible gains emphasizes the potential for further exploration in this area. + +We look forward to conducting additional experiments and engaging in discussions to deepen our understanding of effective model combinations. + +## Usage + +Please refer to our first concatenated model, [Arctic M (v1.5) & BGE small](https://huggingface.co/PaDaS-Lab/arctic-m-bge-small), for more details on how to use this model. + +## Citation + +```bibtex +@misc{https://doi.org/10.48550/arxiv.2407.08275, + doi = {10.48550/ARXIV.2407.08275}, + url = {https://arxiv.org/abs/2407.08275}, + author = {Caspari, Laura and Dastidar, Kanishka Ghosh and Zerhoudi, Saber and Mitrovic, Jelena and Granitzer, Michael}, + title = {Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systems}, + year = {2024}, + copyright = {Creative Commons Attribution 4.0 International} +} +``` + +## License + +Notice that Arctic L is licensed under [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) and BGE Small (en; v1.5) is licensed under [MIT](https://opensource.org/licenses/MIT) license. Please refer to the licenses of the original models for more details.