MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark

Recent advancements in large language models (LLMs) have showcasedsignificant improvements in mathematics. However, traditional math benchmarkslike GSM8k offer a unidimensional perspective, falling short in providing aholistic assessment of the LLMs' math capabilities. To address this gap, weintroduce MathBench, a new benchmark that rigorously assesses the mathematicalcapabilities of large language models. MathBench spans a wide range ofmathematical disciplines, offering a detailed evaluation of both theoreticalunderstanding and practical problem-solving skills. The benchmark progressesthrough five distinct stages, from basic arithmetic to college mathematics, andis structured to evaluate models at various depths of knowledge. Each stageincludes theoretical questions and application problems, allowing us to measurea model's mathematical proficiency and its ability to apply concepts inpractical scenarios. MathBench aims to enhance the evaluation of LLMs'mathematical abilities, providing a nuanced view of their knowledgeunderstanding levels and problem solving skills in a bilingual context. Theproject is released at https://github.com/open-compass/MathBench .