再见视角API:NLP、CSS和LLM评估中的测量基础设施教训
Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructure
AI总结:
视角API的关闭使NLP、CSS和LLM评估领域失去自动化毒性测量的基准,本文探讨其依赖问题及对研究的影响,呼吁建立独立、可验证的测量基础设施。
摘要:
视角API在2026年底关闭,使其成为NLP、CSS和LLM评估中自动化毒性测量的现实标准。我们记录了社区对这一单一专有工具的依赖,并讨论这种依赖导致的知识问题,影响并可能持续影响集体研究。视角模型定期更新但无版本控制或披露,其注释结构反映了单一企业对争议概念的操作化,其评分同时作为评估目标和标准。其关闭留下不可更新的基准、不可复现的结果,最终使领域有风险通过转向闭源LLM继续这些问题。我们利用视角的宣布终止机会,呼吁建立独立、有效、可适应和可复现的毒性及仇恨言论测量基础设施,本文提出了技术与治理要求。
英文摘要:
Perspective API closes at the end of 2026, removing the de facto standard for toxicity measurement and exposing researchers' dependence on a tool they did not control. Drawing on this case, we argue that a research field must build and govern its own measurement infrastructure rather than borrow it. Surveying 241 papers that use or study Perspective, we show what depending on it cost the research community: claims reaching past what the tool could support, results that shifted when its model was silently retrained, and disparities researchers could measure but not explain. These failures were amplified throughout the LLM lifecycle, where Perspective supplied the labels, filtered the corpora, and graded the systems trained on each, rewarding errors rather than catching them. To keep the instrument open to study after shutdown, we release Perspective scores for 5.9 million text snippets from 77 datasets. We further specify ten requirements for measurement infrastructure a field owns, and argue that what blocks such infrastructure is not technical capability but the value the field places on infrastructure work.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。