An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Data
J. Zhang, Y. Fan, Y. Xu, K. Yuan
Learn with Optimization, Optimize with Learning.

MachinE Learning and OptimizatioN (MELON) Lab works at the intersection of optimization and machine learning. Our research spans three interconnected areas: optimization theory and methods, efficient large language model (LLM) training, and AI for optimization. These directions inform and reinforce one another: theory guides the design of new algorithms; advances in training enable more capable and reliable LLMs; and LLMs, in turn, offer new tools for discovering, designing, and analyzing optimization methods. Our aim is to create a self-reinforcing cycle that advances both optimization and machine learning.
Our research develops distributed, robust, and scalable optimization methods for modern machine learning and scientific computing. We focus on computational, memory, and communication efficiency, with rigorous convergence guarantees for high-dimensional, large-scale problems.
Our research develops memory- and compute-efficient methods for LLM pre-training and post-training, emphasizing communication efficiency, robustness, and fault tolerance. We also design optimizers guided by the geometry of the loss landscape to improve training efficiency and stability at scale.
Our research uses AI to advance optimization theory and characterize the fundamental limits of algorithmic performance. We also harness AI to discover and design optimization algorithms that approach these limits in practice.
J. Zhang, Y. Fan, Y. Xu, K. Yuan
S. Zhu, R. Hu, M. Wang, M. Sun, X. Wang, K. Yuan, Z. Wen
B Kong, X Huang, Y Xu, Y Liang, B Wang, K Yuan