MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformers

Recent advances in pre-trained vision transformers have shown promise inparameter-efficient audio-visual learning without audio pre-training. However,few studies have investigated effective methods for aligning multimodalfeatures in parameter-efficient audio-visual transformers. In this paper, wepropose MA-AVT, a new parameter-efficient audio-visual transformer employingdeep modality alignment for corresponding multimodal semantic features.Specifically, we introduce joint unimodal and multimodal token learning foraligning the two modalities with a frozen modality-shared transformer. Thisallows the model to learn separate representations for each modality, whilealso attending to the cross-modal relationships between them. In addition,unlike prior work that only aligns coarse features from the output of unimodalencoders, we introduce blockwise contrastive learning to aligncoarse-to-fine-grain hierarchical features throughout the encoding phase.Furthermore, to suppress the background features in each modality fromforeground matched audio-visual features, we introduce a robust discriminativeforeground mining scheme. Through extensive experiments on benchmark AVE,VGGSound, and CREMA-D datasets, we achieve considerable performanceimprovements over SOTA methods.