MAFIA: Multi-Adapter Fused Inclusive LanguAge Models

Pretrained Language Models (PLMs) are widely used in NLP for various tasks.Recent studies have identified various biases that such models exhibit and haveproposed methods to correct these biases. However, most of the works address alimited set of bias dimensions independently such as gender, race, or religion.Moreover, the methods typically involve finetuning the full model to maintainthe performance on the downstream task. In this work, we aim to modularlydebias a pretrained language model across multiple dimensions. Previous worksextensively explored debiasing PLMs using limited US-centric counterfactualdata augmentation (CDA). We use structured knowledge and a large generativemodel to build a diverse CDA across multiple bias dimensions in asemi-automated way. We highlight how existing debiasing methods do not considerinteractions between multiple societal biases and propose a debiasing modelthat exploits the synergy amongst various societal biases and enablesmulti-bias debiasing simultaneously. An extensive evaluation on multiple tasksand languages demonstrates the efficacy of our approach.