论文标题
评估复发性神经网络的记忆能力
Assessing the Memory Ability of Recurrent Neural Networks
论文作者
论文摘要
众所周知，经常性的神经网络（RNN）可以记住，在其隐藏的层中，通过正在处理的序列（例如，句子）表示的语义信息的一部分。已经设计了不同类型的复发单元，以使RNN能够在更长的时间内记住信息。但是，不同复发单元的记忆能力在理论上和经验上仍然不清楚，因此限制了更有效和可解释的RNN的发展。为了解决该问题，在本文中，我们识别和分析影响RNN记忆能力的内部和外部因素，并提出语义欧几里得空间来表示由序列表达的语义。基于语义欧几里得空间，定义了一系列评估指标来衡量不同复发单元的记忆能力并分析其局限性。这些评估指标还提供了有用的指导，可以在训练过程中为不同的RNN选择合适的序列长度。
It is known that Recurrent Neural Networks (RNNs) can remember, in their hidden layers, part of the semantic information expressed by a sequence (e.g., a sentence) that is being processed. Different types of recurrent units have been designed to enable RNNs to remember information over longer time spans. However, the memory abilities of different recurrent units are still theoretically and empirically unclear, thus limiting the development of more effective and explainable RNNs. To tackle the problem, in this paper, we identify and analyze the internal and external factors that affect the memory ability of RNNs, and propose a Semantic Euclidean Space to represent the semantics expressed by a sequence. Based on the Semantic Euclidean Space, a series of evaluation indicators are defined to measure the memory abilities of different recurrent units and analyze their limitations. These evaluation indicators also provide a useful guidance to select suitable sequence lengths for different RNNs during training.
