Beyond-Voice: Towards Continuous 3D Hand Pose Tracking on Commercial Home Assistant Devices

The surging popularity of home assistants and their voice user interface(VUI) have made them an ideal central control hub for smart home devices.However, current form factors heavily rely on VUI, which poses accessibilityand usability issues; some latest ones are equipped with additional cameras anddisplays, which are costly and raise privacy concerns. These concerns jointlymotivate Beyond-Voice, a novel high-fidelity acoustic sensing system thatallows commodity home assistant devices to track and reconstruct hand posescontinuously. It transforms the home assistant into an active sonar systemusing its existing onboard microphones and speakers. We feed a high-resolutionrange profile to the deep learning model that can analyze the motions ofmultiple body parts and predict the 3D positions of 21 finger joints, bringingthe granularity for acoustic hand tracking to the next level. It operatesacross different environments and users without the need for personalizedtraining data. A user study with 11 participants in 3 different environmentsshows that Beyond-Voice can track joints with an average mean absolute error of16.47mm without any training data provided by the testing subject.