论文标题
最陡峭的下降神经结构优化：通过签名的神经分裂逃脱局部最佳
Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting
论文作者
论文摘要
开发高效和有原则的神经体系结构优化方法是现代深度学习的关键挑战。最近，Liu等人[19]提出了一种陡峭的下降（S2D）方法，该方法通过以最陡峭的下降方式将神经元分成多个副本，共同优化基于逐渐增长的网络结构的神经参数和架构。但是，当所有神经元变得“分裂稳定”时，S2D遭受了局部最优问题，这是一个类似于参数优化的局部稳定性的概念。在这项工作中，我们开发了解决当地最佳问题的分裂下降框架的重大且令人惊讶的扩展。这个想法是要观察到原始的S2D不必要地仅限于将神经元分成阳性的加权副本。通过简单地允许在分裂过程中允许正权重和负重，我们可以消除S2D中分裂稳定性的外观，从而逃脱局部Optima以获得更好的性能。通过合并签名的分裂，我们在理论上和经验上都显着扩展了分裂最陡峭下降的优化能力。我们验证了我们的方法，例如CIFAR-100，ImageNet和ModelNet40等各种具有挑战性的基准，在这些基准测试中，我们在这些基准测试中均超过了S2D和其他有关学习准确和节能的神经网络的高级方法。
Developing efficient and principled neural architecture optimization methods is a critical challenge of modern deep learning. Recently, Liu et al.[19] proposed a splitting steepest descent (S2D) method that jointly optimizes the neural parameters and architectures based on progressively growing network structures by splitting neurons into multiple copies in a steepest descent fashion. However, S2D suffers from a local optimality issue when all the neurons become "splitting stable", a concept akin to local stability in parametric optimization. In this work, we develop a significant and surprising extension of the splitting descent framework that addresses the local optimality issue. The idea is to observe that the original S2D is unnecessarily restricted to splitting neurons into positive weighted copies. By simply allowing both positive and negative weights during splitting, we can eliminate the appearance of splitting stability in S2D and hence escape the local optima to obtain better performance. By incorporating signed splittings, we significantly extend the optimization power of splitting steepest descent both theoretically and empirically. We verify our method on various challenging benchmarks such as CIFAR-100, ImageNet and ModelNet40, on which we outperform S2D and other advanced methods on learning accurate and energy-efficient neural networks.
