arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.09142cs.LGcs.CL
收藏

EvasionBench: 一个大规模基准用于检测盈余电话会议问答中的管理层回避

EvasionBench: A Large-Scale Benchmark for Detecting Managerial Evasion in Earnings Call Q&A

Hong Kong University of Science and Technology(香港科学与技术大学);University of Macau(澳门大学)

Shijian Ma, Yan Lin, Yi Yang

AI总结:

EvasionBench通过多模型共识框架和40亿参数分类器,首次大规模评估管理层在盈余电话会议中回避性回应的检测能力。

摘要:

我们提出了EvasionBench,一个用于检测企业盈余电话会议问答环节中回避性回应的综合性基准。从S&P Capital IQ的 transcripts 中提取了2270万个问答对,我们构建了一个严格筛选的数据集,并引入了三级回避分类法:直接、中间和完全回避。我们的标注流程采用多模型共识(MMC)框架,结合双前沿LLM标注和三审多数投票机制处理模糊情况,实现了人类标注者之间Cohen's Kappa为0.835的共识。我们发布:(1)一个平衡的84,000训练集,(2)一个1,000个金标准评估集,带有专家人工标签,以及(3)[Eva-4B],一个经过Qwen3-4B微调的40亿参数分类器,实现了84.9%的宏F1值,优于Claude 4.5、GPT-5.2和Gemini 3 Flash。我们的消融研究证明了多模型共识标注优于单模型标注。EvasionBench通过提供首个专门针对管理层沟通回避的大型基准,填补了金融NLP中的关键空白。

英文摘要:

We present EvasionBench, a comprehensive benchmark for detecting evasive responses in corporate earnings call question-and-answer sessions. Drawing from 22.7 million Q&A pairs extracted from S&P Capital IQ transcripts, we construct a rigorously filtered dataset and introduce a three-level evasion taxonomy: direct, intermediate, and fully evasive. Our annotation pipeline employs a Multi-Model Consensus (MMC) framework, combining dual frontier LLM annotation with a three-judge majority voting mechanism for ambiguous cases, achieving a Cohen's Kappa of 0.835 on human inter-annotator agreement. We release: (1) a balanced 84K training set, (2) a 1K gold-standard evaluation set with expert human labels, and (3) [Eva-4B], a 4-billion parameter classifier fine-tuned from Qwen3-4B that achieves 84.9% Macro-F1, outperforming Claude 4.5, GPT-5.2, and Gemini 3 Flash. Our ablation studies demonstrate the effectiveness of multi-model consensus labeling over single-model annotation. EvasionBench fills a critical gap in financial NLP by providing the first large-scale benchmark specifically targeting managerial communication evasion.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑