HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision

Large Vision Language Models (VLMs) are now the de facto state-of-the-art fora number of tasks including visual question answering, recognising objects, andspatial referral. In this work, we propose the HOI-Ref task for egocentricimages that aims to understand interactions between hands and objects usingVLMs. To enable HOI-Ref, we curate the HOI-QA dataset that consists of 3.9Mquestion-answer pairs for training and evaluating VLMs. HOI-QA includesquestions relating to locating hands, objects, and critically theirinteractions (e.g. referring to the object being manipulated by the hand). Wetrain the first VLM for HOI-Ref on this dataset and call it VLM4HOI. Ourresults demonstrate that VLMs trained for referral on third person images failto recognise and refer hands and objects in egocentric images. When fine-tunedon our egocentric HOI-QA dataset, performance improves by 27.9hands and objects, and by 26.7