论文标题
没有单词嵌入模型是完美的：评估媒体中社会偏见的表示准确性
No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media
论文作者
论文摘要
新闻文章既形成又反映了整个政治领域的公众舆论。因此，分析它们的社会偏见可以提供有价值的见解，例如社会和媒体的普遍刻板印象，这些刻板印象经常被接受各自数据培训的NLP模型采用。最近的工作依赖于嵌入偏见措施（例如Weat）的单词。但是，嵌入的几个表示问题可能会损害措施的准确性，包括低资源设置和令牌频率差异。在这项工作中，我们研究了哪种嵌入算法最适合准确测量在美国在线新闻文章中已知的社会偏见类型。为了涵盖美国的全部政治偏见，我们收集了500k文章，并就预期的社会偏见进行了审查心理学文献。然后，我们使用WEAT量化社会偏见，并嵌入说明上述问题的算法。我们比较新闻文章中的算法训练的模型如何代表预期的社会偏见。我们的结果表明，量化偏见的标准方法与心理学的知识并不十分吻合。虽然提出的算法减少了〜间隙，但它们仍然不完全匹配文献。
News articles both shape and reflect public opinion across the political spectrum. Analyzing them for social bias can thus provide valuable insights, such as prevailing stereotypes in society and the media, which are often adopted by NLP models trained on respective data. Recent work has relied on word embedding bias measures, such as WEAT. However, several representation issues of embeddings can harm the measures' accuracy, including low-resource settings and token frequency differences. In this work, we study what kind of embedding algorithm serves best to accurately measure types of social bias known to exist in US online news articles. To cover the whole spectrum of political bias in the US, we collect 500k articles and review psychology literature with respect to expected social bias. We then quantify social bias using WEAT along with embedding algorithms that account for the aforementioned issues. We compare how models trained with the algorithms on news articles represent the expected social bias. Our results suggest that the standard way to quantify bias does not align well with knowledge from psychology. While the proposed algorithms reduce the~gap, they still do not fully match the literature.
