Read to Play (R2-Play): Decision Transformer with Multimodal Game Instruction

Developing a generalist agent is a longstanding objective in artificialintelligence. Previous efforts utilizing extensive offline datasets fromvarious tasks demonstrate remarkable performance in multitasking scenarioswithin Reinforcement Learning. However, these works encounter challenges inextending their capabilities to new tasks. Recent approaches integrate textualguidance or visual trajectory into decision networks to provide task-specificcontextual cues, representing a promising direction. However, it is observedthat relying solely on textual guidance or visual trajectory is insufficientfor accurately conveying the contextual information of tasks. This paperexplores enhanced forms of task guidance for agents, enabling them tocomprehend gameplay instructions, thereby facilitating a "read-to-play"capability. Drawing inspiration from the success of multimodal instructiontuning in visual tasks, we treat the visual-based RL task as a long-horizonvision task and construct a set of multimodal game instructions to incorporateinstruction tuning into a decision transformer. Experimental resultsdemonstrate that incorporating multimodal game instructions significantlyenhances the decision transformer's multitasking and generalizationcapabilities.