Abstract:
Federated Learning (FL) is a distributed Machine Learning (ML) paradigm that enables multiple local devices, that is, clients, and a central server to collaboratively train a ML model using data stored locally on the clients without transferring the data. Common challenges in FL include data privacy preservation, communication efficiency, and convergence guarantees. This presentation introduces two new FL algorithms: Accelerating Federated Learning with One-Step Anderson Acceleration (FedOSAA) and Federated Dynamical Low-Rank Training (FeDLRT). FedOSAA leverages a widely used acceleration scheme, Anderson acceleration, to improve the convergence rate of standard FL schemes while incurring minimal additional communication cost. FeDLRT adopts a dynamical low-rank training scheme that constrains model training to low-rank factors of the model parameters, resulting in reductions in both communication and memory costs during the training process. By incorporating a variance-reduction scheme, convergence guarantees are established for both algorithms, even when the data distribution is heterogeneous, that is, non–independent and identically distributed, among the local clients. The advantages of these two FL algorithms are demonstrated through numerical experiments on ML problems ranging from logistic regression to the training of transformer models in a FL setting.
Speaker’s Bio:
Paul Laiu is a Staff Mathematician in the Multiscale Methods and Dynamics Group at the Oak Ridge National Laboratory. He received his Ph.D. degree in Electrical and Computer Engineering from University of Maryland College Park in 2016. Paul’s research interest includes scientific machine learning, surrogate modeling, iterative solvers, and numerical schemes for various partial differential equations. His work focuses on the design, development, and analysis of mathematical tools that accelerate the simulation and learning of multiscale systems, with applications in astrophysics, computational fluid dynamics, cybersecurity, fusion, and rarefied gas.