论文标题
关于自适应随机优化方法的趋势校正变体
On the Trend-corrected Variant of Adaptive Stochastic Optimization Methods
论文作者
论文摘要
Adam型优化器作为具有指数移动平均方案的一类自适应力矩估计方法，已成功地用于深度学习的许多应用中。由于具有较高计算效率的大规模稀疏数据集的能力，此类方法具有吸引力。在本文中，我们为Adam类型方法提供了一个新的框架，并在使用自适应步长和梯度更新参数时，并提供趋势信息。算法中的附加术语有望在复杂成本表面上有效运动，因此损失会更快地融合。我们从经验上展示了添加趋势组件的重要性，其中我们的框架在经典模型上不断使用多个现实世界数据集在经典模型上不断优于常规的Adam和Amsgrad方法。
Adam-type optimizers, as a class of adaptive moment estimation methods with the exponential moving average scheme, have been successfully used in many applications of deep learning. Such methods are appealing due to the capability on large-scale sparse datasets with high computational efficiency. In this paper, we present a new framework for Adam-type methods with the trend information when updating the parameters with the adaptive step size and gradients. The additional terms in the algorithm promise an efficient movement on the complex cost surface, and thus the loss would converge more rapidly. We show empirically the importance of adding the trend component, where our framework outperforms the conventional Adam and AMSGrad methods constantly on the classical models with several real-world datasets.
