论文标题
Treelike神经网络存储能力的激活功能依赖性
Activation function dependence of the storage capacity of treelike neural networks
论文作者
论文摘要
人工神经网络的表达力至关重要取决于其激活函数的非线性。尽管已经提出了各种非线性激活功能用于人工神经网络，但尚未出现对它们在确定网络表达能力中的作用的详细理解。在这里，我们研究激活功能如何影响Treelike两层网络的存储能力。我们将无限宽度限制能力的界限或差异与激活函数的平滑度联系起来，从而阐明了先前研究的特殊情况之间的关系。我们的结果表明，非线性既可以增加容量并降低分类的鲁棒性，并为具有多种常用的激活功能的网络的能力提供简单的估计。此外，它们为树枝状神经元中树突状尖峰的功能益处产生了一个假设。
The expressive power of artificial neural networks crucially depends on the nonlinearity of their activation functions. Though a wide variety of nonlinear activation functions have been proposed for use in artificial neural networks, a detailed understanding of their role in determining the expressive power of a network has not emerged. Here, we study how activation functions affect the storage capacity of treelike two-layer networks. We relate the boundedness or divergence of the capacity in the infinite-width limit to the smoothness of the activation function, elucidating the relationship between previously studied special cases. Our results show that nonlinearity can both increase capacity and decrease the robustness of classification, and provide simple estimates for the capacity of networks with several commonly used activation functions. Furthermore, they generate a hypothesis for the functional benefit of dendritic spikes in branched neurons.
