Accelerating Heterogeneous Federated Learning with Closed-form Classifiers

Federated Learning (FL) methods often struggle in highly statisticallyheterogeneous settings. Indeed, non-IID data distributions cause client driftand biased local solutions, particularly pronounced in the final classificationlayer, negatively impacting convergence speed and accuracy. To address thisissue, we introduce Federated Recursive Ridge Regression (Fed3R). Our methodfits a Ridge Regression classifier computed in closed form leveragingpre-trained features. Fed3R is immune to statistical heterogeneity and isinvariant to the sampling order of the clients. Therefore, it provesparticularly effective in cross-device scenarios. Furthermore, it is fast andefficient in terms of communication and computation costs, requiring up to twoorders of magnitude fewer resources than the competitors. Finally, we proposeto leverage the Fed3R parameters as an initialization for a softmax classifierand subsequently fine-tune the model using any FL algorithm (Fed3R withFine-Tuning, Fed3R+FT). Our findings also indicate that maintaining a fixedclassifier aids in stabilizing the training and learning more discriminativefeatures in cross-device settings. Official website: https://fed-3r.github.io/.