论文标题
分布式平均方法用于随机二阶优化
Distributed Averaging Methods for Randomized Second Order Optimization
论文作者
论文摘要
我们考虑分布式优化问题，其中形成Hessian在计算上具有挑战性，并且交流是一个重要的瓶颈。我们开发无偏见的参数平均方法，用于采用黑森的抽样和草图随机二阶优化。现有作品不会考虑估计器的偏见，这将其应用限制为大规模并行计算。我们为正规化参数和步进大小提供了封闭式公式，这些公式可将牛顿方向的偏差最小化。我们还扩展了二阶平均方法的框架，以引入一个无偏的分布式优化框架，用于具有不同工人资源的异质计算系统。此外，我们通过在无服务器计算平台上执行的大规模实验来证明我们的理论发现的含义。
We consider distributed optimization problems where forming the Hessian is computationally challenging and communication is a significant bottleneck. We develop unbiased parameter averaging methods for randomized second order optimization that employ sampling and sketching of the Hessian. Existing works do not take the bias of the estimators into consideration, which limits their application to massively parallel computation. We provide closed-form formulas for regularization parameters and step sizes that provably minimize the bias for sketched Newton directions. We also extend the framework of second order averaging methods to introduce an unbiased distributed optimization framework for heterogeneous computing systems with varying worker resources. Additionally, we demonstrate the implications of our theoretical findings via large scale experiments performed on a serverless computing platform.
