SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages

Exploring and quantifying semantic relatedness is central to representinglanguage and holds significant implications across various NLP tasks. Whileearlier NLP research primarily focused on semantic similarity, often within theEnglish language context, we instead investigate the broader phenomenon ofsemantic relatedness. In this paper, we present SemRel, a new semanticrelatedness dataset collection annotated by native speakers across 13languages: Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi,Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic,Spanish, and Telugu. These languages originate from five distinctlanguage families and are predominantly spoken in Africa and Asia – regionscharacterised by a relatively limited availability of NLP resources. Eachinstance in the SemRel datasets is a sentence pair associated with a score thatrepresents the degree of semantic textual relatedness between the twosentences. The scores are obtained using a comparative annotation framework. Wedescribe the data collection and annotation processes, challenges when buildingthe datasets, baseline experiments, and their impact and utility in NLP.