论文标题
嘈杂的可区分架构搜索
Noisy Differentiable Architecture Search
论文作者
论文摘要
简单是最终的复杂性。可区分的建筑搜索（飞镖）现在已成为神经体系结构搜索的主流范式之一。但是，由于跳过连接的聚集，它在很大程度上遭受了众所周知的性能崩溃问题。人们认为它从加速信息流的残留结构中受益。为了削弱这种影响，我们建议注入无偏的随机噪声来阻碍流动。我们将这种新颖的方法命名为Noisydarts。实际上，网络优化器应在每个训练步骤中感知这一困难，并避免过度旋转，尤其是在跳过连接上。从长远来看，由于我们在期望方面对梯度没有任何偏见，因此它仍然很可能会收敛到正确的解决方案区域。我们还证明，注入的噪声在平滑损失景观方面起作用，这使优化更加容易。我们的方法具有极致的简单性，并充当新的强基线。我们在各种搜索空间，数据集和任务上执行广泛的实验，在此实验中，我们可以在此实现最新的结果。我们的代码可从https://github.com/xiaomi-automl/noisydarts获得。
Simplicity is the ultimate sophistication. Differentiable Architecture Search (DARTS) has now become one of the mainstream paradigms of neural architecture search. However, it largely suffers from the well-known performance collapse issue due to the aggregation of skip connections. It is thought to have overly benefited from the residual structure which accelerates the information flow. To weaken this impact, we propose to inject unbiased random noise to impede the flow. We name this novel approach NoisyDARTS. In effect, a network optimizer should perceive this difficulty at each training step and refrain from overshooting, especially on skip connections. In the long run, since we add no bias to the gradient in terms of expectation, it is still likely to converge to the right solution area. We also prove that the injected noise plays a role in smoothing the loss landscape, which makes the optimization easier. Our method features extreme simplicity and acts as a new strong baseline. We perform extensive experiments across various search spaces, datasets, and tasks, where we robustly achieve state-of-the-art results. Our code is available at https://github.com/xiaomi-automl/NoisyDARTS.
