Automatic Pair Construction for Contrastive Post-training

Alignment serves as an important step to steer large language models (LLMs)towards human preferences. In this paper, we propose an automatic way toconstruct contrastive data for LLM, using preference pairs from multiple modelsof varying strengths (e.g., InstructGPT, ChatGPT and GPT-4). We compare thecontrastive techniques of SLiC and DPO to SFT baselines and find that DPOprovides a step-function improvement even after continuing SFT saturates. Wealso explore a data curriculum learning scheme for contrastive post-training,which starts by learning from "easier" pairs and transitioning to "harder"ones, which further improves alignment. Finally, we scale up our experiments totrain with more data and larger models like Orca. Remarkably, our automaticcontrastive post-training further improves the performance of Orca, already astate-of-the-art instruction learning model tuned with GPT-4 outputs, tooutperform ChatGPT.