"Uncheatable" LLMs Evaluation - LatestEval

Humans receive new test questions every exam, but LLMs? They've been evaluated with the same benchmarks for too long. Why not assess LLMs with fresh test just like we test our students? In this project, we introduce LatestEval, which automatically constructs language model benchmarks using the latest materials (e.g., arXiv, BBC, GitHub, etc.) to prevent "cheating" and data contamination.

News!!

• 15 Dec, 2023 - This project was accpeted by the main track of AAAI 2024 🥳! Check out the paper here: 👉 Dynamic Test Construction with Latest Materials.

Key Features

1. We maintain a QA benchmark that updates every half month using the latest online resources (created in the past half month). This approach aims to avoid 1) LLMs being trained on the test set (cheating); and 2) the unintentional inclusion of test questions in the training dataset (data contamination).
2. We analyzed real Human-AI conversations to ensure the automated benchmark aligns well with real-life applications (see paper for more detail).