Community Evals: Because We’re Done Trusting Black-Box Leaderboards Over the Community скачать в хорошем качестве

Community Evals: Because We’re Done Trusting Black-Box Leaderboards Over the Community

To address the discrepancy between saturated benchmark metrics and actual model reliability, Hugging Face has introduced "Community Evals," a decentralized framework designed to democratize and transparently report AI performance. This system enables benchmark dataset repositories to function as dynamic leaderboards that aggregate evaluation scores directly from model repositories, where results are stored in standardized YAML files adhering to Inspect AI specifications. By permitting the broader community to submit evaluation results via pull requests and maintaining a Git-based history of these contributions, the initiative establishes a verifiable and reproducible ecosystem that captures both model author and independent community data. While this open approach does not immediately resolve issues such as test-set contamination or the plateauing of scores on established tests like GSM8K, it aims to illuminate the "who, how, and when" of evaluations, fostering a more rigorous environment for developing and tracking the next generation of model capabilities. https://huggingface.co/blog/community...