论文标题
通过语法多样性提示进行稳健的NLG偏差评估
Towards Robust NLG Bias Evaluation with Syntactically-diverse Prompts
论文作者
论文摘要
我们提出了一种可用于评估自然语言产生（NLG）系统偏见的强大方法。先前的作品使用固定的手工制作的前缀模板，并提及各种人口组，以促使模型生成偏置分析的连续性。这些固定的前缀模板本身可能在样式或语言结构方面是特定的，这可能导致不可靠的公平结论，这些结论不能代表音调变化的提示的一般趋势。为了研究这个问题，我们用不同的句法结构来解释提示，并使用这些提示来评估NLG系统中的人口偏见。我们的结果表明，总体偏见趋势相似，但与过去的作品相比，某些句法结构导致了矛盾的结论。我们表明我们的方法论更强大，并且某些句法结构促使更多的有毒内容，而另一些句法结构可能会引起偏见的产生。这表明不依赖固定的句法结构并使用不变提示的重要性。引入句法多样性提示可以实现更强大的NLG（偏见）评估。
We present a robust methodology for evaluating biases in natural language generation(NLG) systems. Previous works use fixed hand-crafted prefix templates with mentions of various demographic groups to prompt models to generate continuations for bias analysis. These fixed prefix templates could themselves be specific in terms of styles or linguistic structures, which may lead to unreliable fairness conclusions that are not representative of the general trends from tone varying prompts. To study this problem, we paraphrase the prompts with different syntactic structures and use these to evaluate demographic bias in NLG systems. Our results suggest similar overall bias trends but some syntactic structures lead to contradictory conclusions compared to past works. We show that our methodology is more robust and that some syntactic structures prompt more toxic content while others could prompt less biased generation. This suggests the importance of not relying on a fixed syntactic structure and using tone-invariant prompts. Introducing syntactically-diverse prompts can achieve more robust NLG (bias) evaluation.
