Folding Attention: Memory and Power Optimization for On-Device Transformer-Based Streaming Speech Recognition.

Transformer-based models excel in speech recognition. Existing efforts tooptimize Transformer inference, typically for long-context applications, centeron simplifying attention score calculations. However, streaming speechrecognition models usually process a limited number of tokens each time, makingattention score calculation less of a bottleneck. Instead, the bottleneck liesin the linear projection layers of multi-head attention and feedforwardnetworks, constituting a substantial portion of the model size and contributingsignificantly to computation, memory, and power usage. To address this bottleneck, we propose folding attention, a techniquetargeting these linear layers, significantly reducing model size and improvingmemory and power efficiency. Experiments on on-device Transformer-basedstreaming speech recognition models show that folding attention reduces modelsize (and corresponding memory consumption) by up to 24by up to 23