MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

We introduce MMMU: a new benchmark designed to evaluate multimodal models onmassive multi-discipline tasks demanding college-level subject knowledge anddeliberate reasoning. MMMU includes 11.5K meticulously collected multimodalquestions from college exams, quizzes, and textbooks, covering six coredisciplines: Art Design, Business, Science, Health Medicine, Humanities Social Science, and Tech Engineering. These questions span 30 subjects and183 subfields, comprising 30 highly heterogeneous image types, such as charts,diagrams, maps, tables, music sheets, and chemical structures. Unlike existingbenchmarks, MMMU focuses on advanced perception and reasoning withdomain-specific knowledge, challenging models to perform tasks akin to thosefaced by experts. The evaluation of 14 open-source LMMs as well as theproprietary GPT-4V(ision) and Gemini highlights the substantial challengesposed by MMMU. Even the advanced GPT-4V and Gemini Ultra only achieveaccuracies of 56improvement. We believe MMMU will stimulate the community to buildnext-generation multimodal foundation models towards expert artificial generalintelligence.