II-Bench: an Image Implication Understanding Benchmark for Multimodal Large Language Models

The rapid advancements in the development of multimodal large language models(MLLMs) have consistently led to new breakthroughs on various benchmarks. Inresponse, numerous challenging and comprehensive benchmarks have been proposedto more accurately assess the capabilities of MLLMs. However, there is a dearthof exploration of the higher-order perceptual capabilities of MLLMs. To fillthis gap, we propose the Image Implication understanding Benchmark, II-Bench,which aims to evaluate the model's higher-order perception of images. Throughextensive experiments on II-Bench across multiple MLLMs, we have madesignificant findings. Initially, a substantial gap is observed between theperformance of MLLMs and humans on II-Bench. The pinnacle accuracy of MLLMsattains 74.898suggesting limitations in their ability to understand high-level semantics andcapture image details. Finally, it is observed that most models exhibitenhanced accuracy when image sentiment polarity hints are incorporated into theprompts. This observation underscores a notable deficiency in their inherentunderstanding of image sentiment. We believe that II-Bench will inspire thecommunity to develop the next generation of MLLMs, advancing the journeytowards expert artificial general intelligence (AGI). II-Bench is publiclyavailable at https://huggingface.co/datasets/m-a-p/II-Bench.