arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.13888cs.CLcs.AIcs.LG
收藏

效用在用户眼中:对NLP排行榜的批判

Utility is in the Eye of the User: A Critique of NLP Leaderboards

Stanford University(斯坦福大学)

Kawin Ethayarajh, Dan Jurafsky

摘要:

GLUE等基准通过激励创建更准确的模型,帮助推动了NLP的进步。虽然这种排行榜模式非常成功,但历史上对基于性能的评估的关注是以牺牲NLP社区在模型中重视的其他品质为代价的,例如紧凑性,公平性和能源效率。在这份意见书中,我们通过微观经济理论的视角,研究了排行榜激励的内容与实践中有用的内容之间的差异。我们将排行榜和NLP从业者都视为消费者,他们从模型中获得的好处是模型对他们的效用。有了这个框架,我们正式确定了排行榜--以它们目前的形式--是如何成为整个NLP社区的糟糕代理的。例如,效率极低的模式对从业者的效用较低,但对排行榜的效用却不高,因为只有从业者必须承担这一成本。为了让从业者更好地估计模型对他们的效用,我们提倡排行榜更加透明,例如报告实际关注的统计数据(例如,模型大小、能量效率和推理延迟)。

英文摘要:

Benchmarks such as GLUE have helped drive advances in NLP by incentivizing the creation of more accurate models. While this leaderboard paradigm has been remarkably successful, a historical focus on performance-based evaluation has been at the expense of other qualities that the NLP community values in models, such as compactness, fairness, and energy efficiency. In this opinion paper, we study the divergence between what is incentivized by leaderboards and what is useful in practice through the lens of microeconomic theory. We frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. With this framing, we formalize how leaderboards -- in their current form -- can be poor proxies for the NLP community at large. For example, a highly inefficient model would provide less utility to practitioners but not to a leaderboard, since it is a cost that only the former must bear. To allow practitioners to better estimate a model's utility to them, we advocate for more transparency on leaderboards, such as the reporting of statistics that are of practical concern (e.g., model size, energy efficiency, and inference latency).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑