论文标题
具有余弦和relu $^k $激活功能的浅神经网络的高阶近似率
High-Order Approximation Rates for Shallow Neural Networks with Cosine and ReLU$^k$ Activation Functions
论文作者
论文摘要
我们研究具有整流线性单元的功能的浅神经网络的近似特性。具体而言，我们考虑了近似函数$ f $的近似值对尺寸的依赖性以及光谱中的光滑度。我们表明，随着$ f $ $ f $的平滑性指数$ s $ s的增加，具有relu $^k $激活功能的浅神经网络获得了提高的近似率，最佳可能的$ o（n^{ - （k+1）} \ log（n）$ l^2 $，独立于尺寸$ d $。该结果的重要性是，激活函数relu $^k $与维度无关，而对于经典方法，多项式近似的程度或所使用的小波的平滑度必须增加，以利用尺寸依赖于$ f $的尺寸的平滑度。此外，我们得出了在光谱巴伦空间上具有余弦激活函数的浅神经网络的近似速率。最后，我们证明下限表明在给定假设下达到的近似率是最佳的。
We study the approximation properties of shallow neural networks with an activation function which is a power of the rectified linear unit. Specifically, we consider the dependence of the approximation rate on the dimension and the smoothness in the spectral Barron space of the underlying function $f$ to be approximated. We show that as the smoothness index $s$ of $f$ increases, shallow neural networks with ReLU$^k$ activation function obtain an improved approximation rate up to a best possible rate of $O(n^{-(k+1)}\log(n))$ in $L^2$, independent of the dimension $d$. The significance of this result is that the activation function ReLU$^k$ is fixed independent of the dimension, while for classical methods the degree of polynomial approximation or the smoothness of the wavelets used would have to increase in order to take advantage of the dimension dependent smoothness of $f$. In addition, we derive improved approximation rates for shallow neural networks with cosine activation function on the spectral Barron space. Finally, we prove lower bounds showing that the approximation rates attained are optimal under the given assumptions.
