JetMoE: Reaching Llama2 Performance with 0.1M Dollars

Large Language Models (LLMs) have achieved remarkable results, but theirincreasing resource demand has become a major obstacle to the development ofpowerful and accessible super-human intelligence. This report introducesJetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokensfrom carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite itslow cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8Boutperforming the Llama2-7B model and JetMoE-8B-Chat surpassing theLlama2-13B-Chat model. These results suggest that LLM training can be much morecost-effective than generally thought. JetMoE-8B is based on an efficientSparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attentionand feedforward experts. Both layers are sparsely activated, allowing JetMoE-8Bto have 8B parameters while only activating 2B for each input token, reducinginference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8Bis highly open and academia-friendly, using only public datasets and trainingcode. All training parameters and data mixtures have been detailed in thisreport to facilitate future efforts in the development of open foundationmodels. This transparency aims to encourage collaboration and furtheradvancements in the field of accessible and efficient LLMs. The model weightsare publicly available at https://github.com/myshell-ai/JetMoE.