Federated Optimization with Doubly Regularized Drift Correction

Federated learning is a distributed optimization paradigm that allowstraining machine learning models across decentralized devices while keeping thedata localized. The standard method, FedAvg, suffers from client drift whichcan hamper performance and increase communication costs over centralizedmethods. Previous works proposed various strategies to mitigate drift, yet nonehave shown uniformly improved communication-computation trade-offs over vanillagradient descent. In this work, we revisit DANE, an established method in distributedoptimization. We show that (i) DANE can achieve the desired communicationreduction under Hessian similarity constraints. Furthermore, (ii) we present anextension, DANE+, which supports arbitrary inexact local solvers and has morefreedom to choose how to aggregate the local updates. We propose (iii) a novelmethod, FedRed, which has improved local computational complexity and retainsthe same communication complexity compared to DANE/DANE+. This is achieved byusing doubly regularized drift correction.