DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures

Generative models are increasingly being used in various applications, suchas text generation, commonsense reasoning, and question-answering. To beeffective globally, these models must be aware of and account for localsocio-cultural contexts, making it necessary to have benchmarks to evaluate themodels for their cultural familiarity. Since the training data for LLMs isweb-based and the Web is limited in its representation of information, it doesnot capture knowledge present within communities that are not on the Web. Thus,these models exacerbate the inequities, semantic misalignment, and stereotypesfrom the Web. There has been a growing call for community-centeredparticipatory research methods in NLP. In this work, we respond to this call byusing participatory research methods to introduce DOSA, the firstcommunity-generated Dataset of 615 SocialArtifacts, by engaging with 260 participants from 19 differentIndian geographic subcultures. We use a gamified framework that relies oncollective sensemaking to collect the names and descriptions of these artifactssuch that the descriptions semantically align with the shared sensibilities ofthe individuals from those cultures. Next, we benchmark four popular LLMs andfind that they show significant variation across regional sub-cultures in theirability to infer the artifacts.