LLM Olympiad: Why Model Evaluation Needs a Sealed Exam

This paper argues for a "sealed exam" paradigm in evaluating large language models (LLMs) to ensure fair and robust assessment of their true capabilities. It highlights the limitations of current evaluation methodologies, where models can inadvertently learn from test sets, leading to inflated performance metrics that do not reflect real-world generalization. The proposed LLM Olympiad concept advocates for novel, unseen evaluation challenges to promote genuine progress in LLM development and deployment.