论文标题
神经内核无切线
Neural Kernels Without Tangents
论文作者
论文摘要
我们研究了神经网络与内核空间中简单的构建块之间的联系。特别是，使用良好的特征空间工具，例如直接总和，平均和力矩提升，我们提出了一个代数，用于从功能袋中创建“构图”内核。我们表明，这些操作对应于“神经切线内核（NTK）”的许多基础。在实验上，我们表明神经网络架构与相关内核之间的测试误差存在相关性。我们仅使用3x3卷积，2x2平均合并，relu和SGD和MSE损失进行优化，在CIFAR10上获得96％的精度，并且其相应的组成核的精度为90％。我们还使用我们的构造来研究小型数据集制度中神经网络，NTK和组成核的相对性能。特别是，我们发现组成核的表现优于NTK和神经网络的表现优于这两种内核方法。
We investigate the connections between neural networks and simple building blocks in kernel space. In particular, using well established feature space tools such as direct sum, averaging, and moment lifting, we present an algebra for creating "compositional" kernels from bags of features. We show that these operations correspond to many of the building blocks of "neural tangent kernels (NTK)". Experimentally, we show that there is a correlation in test error between neural network architectures and the associated kernels. We construct a simple neural network architecture using only 3x3 convolutions, 2x2 average pooling, ReLU, and optimized with SGD and MSE loss that achieves 96% accuracy on CIFAR10, and whose corresponding compositional kernel achieves 90% accuracy. We also use our constructions to investigate the relative performance of neural networks, NTKs, and compositional kernels in the small dataset regime. In particular, we find that compositional kernels outperform NTKs and neural networks outperform both kernel methods.
