SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages

We present the first shared task on Semantic Textual Relatedness (STR). Whileearlier shared tasks primarily focused on semantic similarity, we insteadinvestigate the broader phenomenon of semantic relatedness across 14 languages:Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian,Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Punjabi,Spanish, and Telugu. These languages originate from five distinct languagefamilies and are predominantly spoken in Africa and Asia – regionscharacterised by the relatively limited availability of NLP resources. Eachinstance in the datasets is a sentence pair associated with a score thatrepresents the degree of semantic textual relatedness between the twosentences. Participating systems were asked to rank sentence pairs by theircloseness in meaning (i.e., their degree of semantic relatedness) in the 14languages in three main tracks: (a) supervised, (b) unsupervised, and (c)crosslingual. The task attracted 163 participants. We received 70 submissionsin total (across all tasks) from 51 different teams, and 38 system descriptionpapers. We report on the best-performing systems as well as the most common andthe most effective approaches for the three different tracks.