Achieving Linear Speedup in Asynchronous Federated Learning with Heterogeneous Clients

Federated learning (FL) is an emerging distributed training paradigm thataims to learn a common global model without exchanging or transferring the datathat are stored locally at different clients. The Federated Averaging(FedAvg)-based algorithms have gained substantial popularity in FL to reducethe communication overhead, where each client conducts multiple localizediterations before communicating with a central server. In this paper, we focuson FL where the clients have diverse computation and/or communicationcapabilities. Under this circumstance, FedAvg can be less efficient since itrequires all clients that participate in the global aggregation in a round toinitiate iterations from the latest global model, and thus the synchronizationamong fast clients and straggler clients can severely slow down the overalltraining process. To address this issue, we propose an efficient asynchronousfederated learning (AFL) framework called Delayed Federated Averaging(DeFedAvg). In DeFedAvg, the clients are allowed to perform local training withdifferent stale global models at their own paces. Theoretical analysesdemonstrate that DeFedAvg achieves asymptotic convergence rates that are on parwith the results of FedAvg for solving nonconvex problems. More importantly,DeFedAvg is the first AFL algorithm that provably achieves the desirable linearspeedup property, which indicates its high scalability. Additionally, we carryout extensive numerical experiments using real datasets to validate theefficiency and scalability of our approach when training deep neural networks.