论文标题
对角线性网络中的增量学习
Incremental Learning in Diagonal Linear Networks
论文作者
论文摘要
对角线线性网络（DLN）是人工神经网络的玩具。它们由线性回归的二次再现化，诱导稀疏的隐式正则化。在本文中，我们描述了DLN的梯度流的轨迹，在小初始化的极限下。我们表明，增量学习有效地在限制下进行：坐标是连续激活的，而迭代是最小化的损耗的最小化器，仅在有源坐标上得到支持。这表明DLN的稀疏隐式正则化随时间而减小。由于技术原因，这项工作仅限于具有反相关特征的隔离式制度。
Diagonal linear networks (DLNs) are a toy simplification of artificial neural networks; they consist in a quadratic reparametrization of linear regression inducing a sparse implicit regularization. In this paper, we describe the trajectory of the gradient flow of DLNs in the limit of small initialization. We show that incremental learning is effectively performed in the limit: coordinates are successively activated, while the iterate is the minimizer of the loss constrained to have support on the active coordinates only. This shows that the sparse implicit regularization of DLNs decreases with time. This work is restricted to the underparametrized regime with anti-correlated features for technical reasons.
