Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

This paper studies post-training large language models (LLMs) usingpreference feedback from a powerful oracle to help a model iteratively improveover itself. The typical approach for post-training LLMs involves ReinforcementLearning from Human Feedback (RLHF), which traditionally separates rewardlearning and subsequent policy optimization. However, such a rewardmaximization approach is limited by the nature of "point-wise" rewards (such asBradley-Terry model), which fails to express complex intransitive or cyclicpreference relations. While advances on RLHF show reward learning and policyoptimization can be merged into a single contrastive objective for stability,they yet still remain tethered to the reward maximization framework. Recently,a new wave of research sidesteps the reward maximization presumptions in favorof directly optimizing over "pair-wise" or general preferences. In this paper,we introduce Direct Nash Optimization (DNO), a provable and scalable algorithmthat marries the simplicity and stability of contrastive learning withtheoretical generality from optimizing general preferences. Because DNO is abatched on-policy algorithm using a regression-based objective, itsimplementation is straightforward and efficient. Moreover, DNO enjoys monotonicimprovement across iterations that help it improve even over a strong teacher(such as GPT-4). In our experiments, a resulting 7B parameter Orca-2.5 modelaligned by DNO achieves the state-of-the-art win-rate against GPT-4-Turbo of33gain of 26far more parameters, including Mistral Large, Self-Rewarding LM (70Bparameters), and older versions of GPT-4.