论文标题
Squeezebert：计算机视觉可以教给NLP有效的神经网络？
SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
论文作者
论文摘要
人类每天读并写几亿封信。此外，由于大型数据集，大型计算系统以及更好的神经网络模型的可用性，自然语言处理（NLP）技术在理解，校对和组织这些信息方面取得了长足的进步。因此，有很大的机会在众多应用程序中部署NLP，以帮助网络用户，社交网络和业务。特别是，我们将智能手机和其他移动设备视为用于大规模部署NLP模型的关键平台。但是，当今高度精确的NLP神经网络模型（例如Bert和Roberta）在计算上非常昂贵，Bert-Base需要1.7秒来对像素3智能手机上的文本片段进行分类。在这项工作中，我们观察到诸如分组卷积之类的方法为计算机视觉网络带来了重大的加速，但是其中许多技术尚未被NLP神经网络设计师采用。我们演示了如何用分组的卷积替换自发动层中的多个操作，并且我们在一种名为Squeezebert的新型网络体系结构中使用此技术，该技术比像素3上的Bert-Base快4.3倍，同时在胶水测试集中实现竞争精度。 Squeezebert代码将发布。
Humans read and write hundreds of billions of messages every day. Further, due to the availability of large datasets, large computing systems, and better neural network models, natural language processing (NLP) technology has made significant strides in understanding, proofreading, and organizing these messages. Thus, there is a significant opportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. In particular, we consider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today's highly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, with BERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. In this work, we observe that methods such as grouped convolutions have yielded significant speedups for computer vision networks, but many of these techniques have not been adopted by NLP neural network designers. We demonstrate how to replace several operations in self-attention layers with grouped convolutions, and we use this technique in a novel network architecture called SqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test set. The SqueezeBERT code will be released.
