Towards Few-Shot Adaptation of Foundation Models Via Multitask Finetuning

Foundation models have emerged as a powerful tool for many AI problems.Despite the tremendous success of foundation models, effective adaptation tonew tasks, particularly those with limited labels, remains an open question andlacks theoretical understanding. An emerging solution with recent success invision and NLP involves finetuning a foundation model on a selection ofrelevant tasks, before its adaptation to a target task with limited labeledsamples. In this paper, we study the theoretical justification of thismultitask finetuning approach. Our theoretical analysis reveals that with adiverse set of related tasks, this multitask finetuning leads to reduced errorin the target task, in comparison to directly adapting the same pretrainedmodel. We quantify the relationship between finetuning tasks and target tasksby diversity and consistency metrics, and further propose a practical taskselection algorithm. We substantiate our theoretical claims with extensiveempirical evidence. Further, we present results affirming our task selectionalgorithm adeptly chooses related finetuning tasks, providing advantages to themodel performance on target tasks. We believe our study shed new light on theeffective adaptation of foundation models to new tasks that lack abundantlabels. Our code is available athttps://github.com/OliverXUZY/Foudation-Model_Multitask.