Evaluating on training dataset
Hello, noticed that evaluation is done on the data that is provided for training. I think i'll result as highly performing models if overfitted on a large size of training data. Was wondering if there is a private dataset for benchmarks on a second step ?
Thanks for organising, I love playing with the HF space :)
Best,
Clément
During the webinar the organizers mentioned that there is a private set that the models will be evaluated on once submission closes. I do think that they should have used a private set to make the leaderboard more informative, but I guess it's just to get a rough idea.
Hello ! Yes we also just corrected it to be sure everyone was using the same test dataset and minimize the risk for overfitting.
The public dataset has two sub datasets train and test and the public leaderboards used the public test set.
And for the final eval we will use a private test set :)
And yes the leaderboard is more informative