Training Large Language Models for Reasoning Through Reverse Curriculum Reinforcement Learning

In this paper, we propose R^3: Learning Reasoning through ReverseCurriculum Reinforcement Learning (RL), a novel method that employs onlyoutcome supervision to achieve the benefits of process supervision for largelanguage models. The core challenge in applying RL to complex reasoning is toidentify a sequence of actions that result in positive rewards and provideappropriate supervision for optimization. Outcome supervision provides sparserewards for final results without identifying error locations, whereas processsupervision offers step-wise rewards but requires extensive manual annotation.R^3 overcomes these limitations by learning from correct demonstrations.Specifically, R^3 progressively slides the start state of reasoning from ademonstration's end to its beginning, facilitating easier model exploration atall stages. Thus, R^3 establishes a step-wise curriculum, allowing outcomesupervision to offer step-level signals and precisely pinpoint errors. UsingLlama2-7B, our method surpasses RL baseline on eight reasoning tasks by 4.1points on average. Notebaly, in program-based reasoning on GSM8K, it exceedsthe baseline by 4.2 points across three backbone models, and without anyextra data, Codellama-7B + R^3 performs comparable to larger models orclosed-source models.