When Scaling Meets LLM Finetuning: the Effect of Data, Model and Finetuning Method

While large language models (LLMs) often adopt finetuning to unlock theircapabilities for downstream applications, our understanding on the inductivebiases (especially the scaling properties) of different finetuning methods isstill limited. To fill this gap, we conduct systematic experiments studyingwhether and how different scaling factors, including LLM model size,pretraining data size, new finetuning parameter size and finetuning data size,affect the finetuning performance. We consider two types of finetuning –full-model tuning (FMT) and parameter efficient tuning (PET, including prompttuning and LoRA), and explore their scaling behaviors in the data-limitedregime where the LLM model size substantially outweighs the finetuning datasize. Based on two sets of pretrained bilingual LLMs from 1B to 16B andexperiments on bilingual machine translation and multilingual summarizationbenchmarks, we find that 1) LLM finetuning follows a powerbased multiplicativejoint scaling law between finetuning data size and each other scaling factor;2) LLM finetuning benefits more from LLM model scaling than pretraining datascaling, and PET parameter scaling is generally ineffective; and 3) the optimalfinetuning method is highly task- and finetuning data-dependent. We hope ourfindings could shed light on understanding, selecting and developing LLMfinetuning methods.