Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentum

Federated Learning (FL) has emerged as the state-of-the-art approach forlearning from decentralized data in privacy-constrained scenarios. However,system and statistical challenges hinder real-world applications, which demandefficient learning from edge devices and robustness to heterogeneity. Despitesignificant research efforts, existing approaches (i) are not sufficientlyrobust, (ii) do not perform well in large-scale scenarios, and (iii) are notcommunication efficient. In this work, we propose a novel GeneralizedHeavy-Ball Momentum (GHBM), motivating its principled application to counteractthe effects of statistical heterogeneity in FL. Then, we present FedHBM as anadaptive, communication-efficient by-design instance of GHBM. Extensiveexperimentation on vision and language tasks, in both controlled and realisticlarge-scale scenarios, provides compelling evidence of substantial andconsistent performance gains over the state of the art.