Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models

This report introduces , a Korean adaptation oflarge language models that exhibit remarkable capabilities across English andKorean text understanding. Building on recent highly capable butEnglish-centric LLMs, such as SOLAR-10.7B and Phi-2, where non-English textsare inefficiently processed with English-centric tokenizers, we present anefficient and effective vocabulary expansion (EEVE) method, which encompassesparameter freezing and subword initialization. In contrast to previous effortsthat believe new embeddings require trillions of training tokens, we show thatour method can significantly boost non-English proficiency within just 2billion tokens. Surpassing most instruction-tuned LLMs on the Open Ko-LLMLeaderboard, as of January 2024, our model ranks as the leading Korean pre-trained model in the open-source community,according to Hugging Face's leaderboard. We open-source our models onHuggingface to empower the open research community in various languages.