CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning Via Counterfactual Examples

We propose CounterCurate, a framework to comprehensively improve thevisio-linguistic compositional reasoning capability for both contrastive andgenerative multimodal models. In particular, we identify two under-exploredcritical problems: the neglect of the physically grounded reasoning (countingand position understanding) and the potential of using highly capable text andimage generation models for semantic counterfactual fine-tuning. Our workpioneers an approach that addresses these gaps. We first spotlight thenear-chance performance of multimodal models like CLIP and LLaVA in physicallygrounded compositional reasoning. We then apply simple data augmentation usinga grounded image generation model, GLIGEN, to generate finetuning data,resulting in significant performance improvements: +33LLaVA, respectively, on our newly curated Flickr30k-Positions benchmark.Moreover, we exploit the capabilities of high-performing text generation andimage generation models, specifically GPT-4V and DALLE-3, to curate challengingsemantic counterfactuals, thereby further enhancing compositional reasoningcapabilities on benchmarks such as SugarCrepe, where CounterCurate outperformsGPT-4V.