AIR-Bench: Benchmarking Large Audio-Language Models Via Generative Comprehension

Recently, instruction-following audio-language models have received broadattention for human-audio interaction. However, the absence of benchmarkscapable of evaluating audio-centric interaction capabilities has impededadvancements in this field. Previous models primarily focus on assessingdifferent fundamental tasks, such as Automatic Speech Recognition (ASR), andlack an assessment of the open-ended generative capabilities centered aroundaudio. Thus, it is challenging to track the progression in the LargeAudio-Language Models (LALMs) domain and to provide guidance for futureimprovement. In this paper, we introduce AIR-Bench (AudioInstRuction Benchmark), the first benchmark designedto evaluate the ability of LALMs to understand various types of audio signals(including human speech, natural sounds, and music), and furthermore, tointeract with humans in the textual format. AIR-Bench encompasses twodimensions: foundation and chat benchmarks. The formerconsists of 19 tasks with approximately 19k single-choice questions, intendingto inspect the basic single-task ability of LALMs. The latter one contains 2kinstances of open-ended question-and-answer data, directly assessing thecomprehension of the model on complex audio and its capacity to followinstructions. Both benchmarks require the model to generate hypothesesdirectly. We design a unified framework that leverages advanced languagemodels, such as GPT-4, to evaluate the scores of generated hypotheses given themeta-information of the audio. Experimental results demonstrate a high level ofconsistency between GPT-4-based evaluation and human evaluation. By revealingthe limitations of existing LALMs through evaluation results, AIR-Bench canprovide insights into the direction of future research.