arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2611.06384cs.CL
收藏

主观文本评估的类型化决策模型中的呈现敏感性

Presentation Sensitivity in Typed Decision Models for Subjective Text Evaluation

IIITH(印度信息技术学院海得拉巴分校);LTRC(语言技术研究中心)

Jatin Agrawal, Radhika Mamidi

AI总结:

该研究针对Jev、Clef Flash、Perplexity Decider三类类型化决策模型,在6个含9000项标注的数据集上,发现呈现方式会显著影响其评估结果,强调需验证评估流程的决策与评分解释以确保可靠性。

摘要:

类型化决策模型可为低成本自动化评估提供结构化选择、评分与概率。我们对Jev、Clef Flash和Perplexity Decider在原生选择与评分界面下的呈现敏感性开展受控审计。在6个包含9000项经多重标注的幽默、有用性及摘要类条目的数据集上,我们收集了702000条判断,这些判断来自对位置、标识符、作者声明、背书、决策框架及评分 rubric(评分准则)的干预,同时保留被评估文本。Jev在摘要任务中,第二位的概率优势达6.11个百分点;反转读者背书会使Perplexity Decider在40.40%的配对中改变所选幽默文本;拒绝框架使Clef Flash的摘要与人类偏好的一致性降低23.36个百分点。评分格式也会改变评分水平,却常保留排名。这些发现表明,可重复性、呈现鲁棒性及人类对齐是评估工具的不同属性,因此可靠使用需验证评估流程所依赖的决策与评分解释。

英文摘要:

Typed decision models offer structured choices, scores, and probabilities for low-cost automated evaluation. We present a controlled audit of presentation sensitivity in Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces. Across six datasets with 9,000 multiply annotated humor, helpfulness, and summarization items, we collect 702,000 judgments under interventions that vary positions, identifiers, authorship claims, endorsements, decision framing, and rating rubrics while preserving the evaluated texts. Jev's second-position probability advantage reaches 6.11 percentage points on summaries; reversing reader endorsements changes Perplexity's selected humor text on 40.40% of pairs; and rejection framing reduces Clef's summary agreement with human preferences by 23.36 points. Rating formats also shift score levels while often preserving rankings. Across these findings, repeatability, presentation robustness, and human alignment emerge as distinct properties of an evaluator. Reliable use therefore requires validating the decisions and score interpretations on which an evaluation pipeline depends.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑