Detours for Navigating Instructional Videos

We introduce the video detours problem for navigating instructional videos.Given a source video and a natural language query asking to alter the how-tovideo's current path of execution in a certain way, the goal is to find arelated ”detour video” that satisfies the requested alteration. To addressthis challenge, we propose VidDetours, a novel video-language approach thatlearns to retrieve the targeted temporal segments from a large repository ofhow-to's using video-and-text conditioned queries. Furthermore, we devise alanguage-based pipeline that exploits how-to video narration text to createweakly supervised training data. We demonstrate our idea applied to the domainof how-to cooking videos, where a user can detour from their current recipe tofind steps with alternate ingredients, tools, and techniques. Validating on aground truth annotated dataset of 16K samples, we show our model's significantimprovements over best available methods for video retrieval and questionanswering, with recall rates exceeding the state of the art by 35