论文标题
探索卷积神经网络低位培训的潜力
Exploring the Potential of Low-bit Training of Convolutional Neural Networks
论文作者
论文摘要
在这项工作中，我们为卷积神经网络提出了一个低位训练框架，该框架围绕新型的多级缩放（MLS）张量格式构建。我们的框架着重于通过将所有卷积操作数量化为低宽度格式来减少卷积操作的能源消耗。具体而言，我们提出了MLS张量格式，其中可以大大降低元素的位宽度。然后，我们描述了动态量化和低位张量卷积算术，以有效利用MLS张量格式。实验表明，与以前的低位训练框架相比，我们的框架在准确性和位宽度之间取得了更高的权衡。对于培训CIFAR-10的各种型号，使用1位Mantissa和2位指数足以使准确性损失保持在$ 1 \％$之内。在诸如Imagenet之类的较大数据集上，使用4位Mantissa和2位指数足以将准确损失保持在$ 1 \％$之内。通过对计算单元的能源消耗模拟，我们可以估计，使用我们的框架的各种型号可以达到$ 8.3 \ sim10.2 \ times $和$ 1.9 \ sim2.3 \ sim2.3 \ times $ $ $ $ $ $比分别进行全面精确的培训和8位浮动点的培训。
In this work, we propose a low-bit training framework for convolutional neural networks, which is built around a novel multi-level scaling (MLS) tensor format. Our framework focuses on reducing the energy consumption of convolution operations by quantizing all the convolution operands to low bit-width format. Specifically, we propose the MLS tensor format, in which the element-wise bit-width can be largely reduced. Then, we describe the dynamic quantization and the low-bit tensor convolution arithmetic to leverage the MLS tensor format efficiently. Experiments show that our framework achieves a superior trade-off between the accuracy and the bit-width than previous low-bit training frameworks. For training a variety of models on CIFAR-10, using 1-bit mantissa and 2-bit exponent is adequate to keep the accuracy loss within $1\%$. And on larger datasets like ImageNet, using 4-bit mantissa and 2-bit exponent is adequate to keep the accuracy loss within $1\%$. Through the energy consumption simulation of the computing units, we can estimate that training a variety of models with our framework could achieve $8.3\sim10.2\times$ and $1.9\sim2.3\times$ higher energy efficiency than training with full-precision and 8-bit floating-point arithmetic, respectively.
