arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.22221cs.IRcs.AIcs.CLcs.CY
收藏

评估中文事实搜索与AI答案中的可靠性不对称性

Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answers

Department of Electronics, Information and Bioengineering, Politecnico di Milano(电子、信息与生物工程系,米兰理工学院);University of Science and Technology of China(中国科学技术大学)

Geng Liu, Li Feng, Mengxiao Zhu, Francesco Pierri

AI总结:

通过构建基于真实中文搜索日志的查询事实核查数据集,比较传统搜索引擎、大型语言模型和搜索集成AI概览在中文是非问题上的准确性、回答频率、极性差距及区域信息需求差异,揭示可靠性不仅取决于回答正确性,还受回答频率、否定主张处理和信息需求暴露风险影响。

摘要:

搜索引擎和AI驱动的系统越来越多地成为获取事实信息的媒介,但在现实信息寻求场景中,其可靠性仍难以评估。我们通过从真实中文搜索日志构建基于查询的事实核查数据集,并比较传统搜索引擎、独立大型语言模型和搜索集成AI概览等九种系统,在中文网络生态中研究这一问题。聚焦于中文事实性是非问题,我们根据证据推导的基准事实评估系统是否提供正确、错误或不确定的判断。我们发现,当系统给出明确答案时,准确率相似(73.2%至78.9%),但给出明确答案的频率差异显著:搜索引擎对超过83%的查询给出明确答案,而Qwen-Max则不到一半。我们还发现一致的极性差距:所有系统在标记为“是”的查询上表现优于标记为“否”的查询。我们利用百度指数数据识别健康相关搜索关注度较高的中国省份,这可能表明更大的错误信息暴露风险。总体而言,我们的结果表明,可靠性不仅取决于系统回答时的正确性,还取决于回答频率、如何处理否定主张以及信息需求可能增加暴露风险的地方。

英文摘要:

Search engines and AI-powered systems increasingly mediate access to factual information, yet their reliability remains difficult to evaluate in realistic information-seeking settings. We study this problem in the Chinese web ecosystem by constructing a query-based fact-checking dataset from real Chinese search logs and comparing nine systems across traditional search engines, standalone large language models, and search-integrated AI Overviews. Focusing on factual Chinese-language factual Yes/No questions, we evaluate whether systems provide correct, incorrect, or uncertain decisions against evidence-derived ground truth. We find that systems are similarly accurate when they provide definitive answers, but differ sharply in how often they do so. Conditional accuracy ranges from 73.2% to 78.9%, yet search engines answer definitively on over 83% of queries, while Qwen-Max does so on fewer than half. We also find a consistent polarity gap: all systems perform better on yes-labeled queries than on no-labeled queries. We also use Baidu Index data to identify Chinese provinces with higher health-related search attention, which may indicate greater potential exposure to misinformation. Overall, our results show that reliability depends not only on whether systems are correct when they answer, but also on how often they answer, how they handle negative claims, and where information demand may increase exposure risks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑