This study considers a federated learning setup where cost-sensitive and strategic agents train a learning model with a server. During each round, each agent samples a minibatch of training data and sends his gradient update. As an increasing function of his minibatch size choice, the agent incurs a cost associated with the data collection, gradient computation and communication. The agents have the freedom to choose their minibatch size and may even opt out from training. To reduce his cost, an agent may diminish his minibatch size, which may also cause an increase in the noise level of the gradient update. The server can offer rewards to compensate the agents for their costs and to incentivize their participation but she lacks the capability of validating the true minibatch sizes of the agents. To tackle this challenge, the proposed reward mechanism evaluates the quality of each agent's gradient according to the its distance to a reference which is constructed from the gradients provided by other agents. It is shown that the proposed reward mechanism has a cooperative Nash equilibrium in which the agents determine the minibatch size choices according to the requests of the server.
Zhejiang University/University of Illinois at Urbana-Champaign Institute(浙江大学-伊利诺伊大学厄巴纳香槟校区联合学院);Hong Kong Metropolitan University(香港都会大学);Singapore University of Technology and Design(新加坡科技设计大学);Princeton University(普林斯顿大学)
First-ish Order Methods: Hessian-aware Scalings of Gradient Descent
School of Mathematics and Physics(数学与物理学院);University of Queensland(昆士兰大学);ARC Training Centre for Information Resilience(信息韧性培训中心);Computer Sciences Department(计算机科学系);University of Wisconsin-Madison(威斯康星大学麦迪逊分校)
Statistical Inference for Differentially Private Stochastic Gradient Descent
Center for Data Science, Zhejiang University(浙江大学数据科学中心);Department of Statistics, Rutgers University(罗格斯大学统计系);Faculty of Business and Economics, The University of Hong Kong(香港大学商学院)
DOME: Improving Signal-to-Noise in Stochastic Gradient Descent via Sharp-Direction Subspace Filtering
International Laboratory on Learning Systems (ILLS), Montreal, Canada;Mila – Quebec AI Institute, Montreal, Canada;McGill University, Montreal, Canada;Inria, INSA Lyon, CITI, UR3720, 69621 Villeurbanne, France;CNRS, INSA Lyon, LIRIS, UMR 5205, 69621 Villeurbanne, France
An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants
Department of Computer Science, George Mason University(乔治·马歇尔大学计算机科学系);Center for Cosmology and Particle Physics, New York University(纽约大学宇宙与粒子物理中心);Center for Computational Mathematics, Flatiron Institute(Flatiron研究所计算数学中心)
Precise gradient descent training dynamics for finite-width multi-layer neural networks
Department of Statistics, Rutgers University, Piscataway, NJ 08854, USA.(统计学系,罗格斯大学);Graduate School of Arts and Sciences, The University of Tokyo, Meguro, Tokyo 153-8902, Japan. RIKEN Center for Advanced Intelligence Project, Chuo, Tokyo 103-0027, Japan.(艺术与科学研究生院,东京大学)
Convex Relaxation for Solving Large-Margin Classifiers in Hyperbolic Space
John A. Paulson School of Engineering and Applied Sciences, Harvard University(哈佛大学约翰·A·保尔森工程与应用科学学院);Center for Brain Science, Harvard University(哈佛大学脑科学中心);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(哈佛大学肯普ner自然与人工智能研究所)
LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates
Center for Cognitive Interaction Technology, Universität Bielefeld, Germany(认知交互技术中心,比勒菲尔德大学,德国);Bielefeld University(比勒菲尔德大学);Department of Computer Science, Royal Holloway, University of London, United Kingdom(计算机科学系,伦敦大学皇家霍洛威学院,英国)
The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions
School of Computer Science, McGill University, Montreal, Canada(麦吉尔大学计算机科学学院);Mila -- Quebec AI Institute, Montreal, Canada(魁北克人工智能研究所);University of Toronto, Vector Canada(多伦多大学);Vector Institute, Toronto, Canada(向量研究所);McGill University(麦吉尔大学);Mila -- Quebec AI Institute(魁北克人工智能研究所);University of Toronto(多伦多大学);Vector Institute(向量研究所)
Efficient Over-parameterized Matrix Sensing from Noisy Measurements via Alternating Preconditioned Gradient Descent
State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences(机器人国家重点实验室,沈阳自动化研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);State University of New York at Buffalo(纽约州立大学布法罗分校);School of Management, Xi’an Jiaotong University(西安交通大学管理学院)
How do simple rotations affect the implicit bias of Adam?
Committee on Computational and Applied Mathematics, University of Chicago(计算与应用数学委员会,芝加哥大学);Department of Electrical & Computer Engineering, University of Washington(电气与计算机工程系,华盛顿大学);NSF-Simons National Institute for Theory and Mathematics in Biology(NSF-西蒙斯国家理论与生物数学研究所);Department of Statistics, Department of Computer Science, and Data Science Institute, University of Chicago(统计系、计算机科学系和数据科学研究所,芝加哥大学)
D$^2$: Decentralized Training over Decentralized Data
Department of Computer Science, University of Rochester(罗切斯特大学计算机科学系);Department of Computational Mathematics, Science and Engineering, Michigan State University(密歇根州立大学计算数学、科学与工程系);Department of Computer Science, ETH Zurich(苏黎世联邦理工学院计算机科学系);Tencent AI Lab(腾讯AI实验室)
Explaining Strategic Decisions in Multi-Agent Reinforcement Learning for Aerial Combat Tactics
Dalle Molle Institute for Artificial Intelligence (IDSIA) - SUPSI/USI(达勒莫尔人工智能研究所(IDSIA)- 瑞士南部应用科学与艺术大学/瑞士意大利语区大学);armasuisse Science and Technology(瑞士联邦国防采购局科学与技术)