A Multimodal Neural Network for Contact State Recognition During Probe Implantation into Skull Holes.

Brain-machine interfaces (BMIs) have attracted wide attention, where invasive BMIs can obtain higher-quality signals compared to non-invasive ones. In invasive BMIs, flexible electrodes for acquiring signals are usually implanted with the assistance of probes. However, due to the orientation error, the probe may collide with the skull wall during implantation. Unlike typical insertion problems, it is difficult to model the interaction forces due to the low stiffness of the probe. To avoid physical modeling, previous approaches leverage force sensor data to identify contact states, thus can adjust the orientation. However, solely relying on the force sensor is insufficient to accurately distinguish the contact states of probes. Therefore, we propose the multimodal Contact State Recognition Network (multimodal CSRNet) that incorporates both binocular RGB images and force sensor data as input. Notably, our paper is the first to investigate the problem of contact state recognition during probe implantation into skull holes. Besides, experiment results show that the proposed multimodal CSRNet relatively enhances the performance by 28.8% and 61.1% than its image-based and force-based counterparts. By performing few-shot transfer learning on unseen holes, it can achieve an accuracy of 89.9% with only 90 samples in about 43s training time.