Orca-Math: Unlocking the Potential of SLMs in Grade School Math

Mathematical word problem-solving has long been recognized as a complex taskfor small language models (SLMs). A recent study hypothesized that the smallestmodel size, needed to achieve over 80billion parameters. To reach this level of performance with smaller models,researcher often train SLMs to generate Python code or use tools to help avoidcalculation errors. Additionally, they employ ensembling, where outputs of upto 100 model runs are combined to arrive at a more accurate result. Resultselection is done using consensus, majority vote or a separate a verifier modelused in conjunction with the SLM. Ensembling provides a substantial boost inaccuracy but at a significant cost increase with multiple calls to the model(e.g., Phi-GSM uses top-48 to boost the performance from 68.2 to 81.5). In this work, we present Orca-Math, a 7-billion-parameter SLM based on theMistral-7B, which achieves 86.81calls or the use of verifiers, code execution or any other external tools. Ourapproach has the following key elements: (1) A high quality synthetic datasetof 200K math problems created using a multi-agent setup where agentscollaborate to create the data, (2) An iterative learning techniques thatenables the SLM to practice solving problems, receive feedback on its solutionsand learn from preference pairs incorporating the SLM solutions and thefeedback. When trained with Supervised Fine-Tuning alone, Orca-Math achieves81.50achieves 86.81larger models such as LLAMA-2-70B, WizardMath-70B, Gemini-Pro, ChatGPT-3.5. Italso significantly outperforms other smaller models while using much smallerdata (hundreds of thousands vs. millions of problems).