arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.26106cs.CL
收藏

LLM基准-气候变化中的用户需求错位

LLM Benchmark-User Need Misalignment for Climate Change

Oucheng Liu, Lexing Xie, Jing Jiang

AI总结:

研究揭示现有LLM基准与实际用户需求存在显著不匹配,提出主动知识行为框架和主题-意图-形式分类法,为基准设计、RAG系统开发和LLM训练提供指导。

摘要:

气候变化是塑造公共决策和政策讨论的重大社会科学问题。随着大型语言模型(LLMs)越来越多地作为获取气候知识的接口,现有基准是否反映用户需求对评估LLM在现实中的表现至关重要。我们提出了一种主动知识行为框架,捕捉人类-人类和人类-AI的知识寻求和提供行为。我们进一步开发了主题-意图-形式分类法,并将其应用于分析代表不同知识行为的气候相关数据。我们的结果揭示了现有基准与现实用户需求之间存在显著不匹配,而人类与LLMs之间的知识交互模式与人类-人类互动模式相似。这些发现为基准设计、RAG系统开发和LLM训练提供了可操作的指导。代码可在https://github.com/OuchengLiu/LLM-Misalign-Climate-Change获取。

英文摘要:

Climate change is a major socio-scientific issue shapes public decision-making and policy discussions. As large language models (LLMs) increasingly serve as an interface for accessing climate knowledge, whether existing benchmarks reflect user needs is critical for evaluating LLM in real-world settings. We propose a Proactive Knowledge Behaviors Framework that captures the different human-human and human-AI knowledge seeking and provision behaviors. We further develop a Topic-Intent-Form taxonomy and apply it to analyze climate-related data representing different knowledge behaviors. Our results reveal a substantial mismatch between current benchmarks and real-world user needs, while knowledge interaction patterns between humans and LLMs closely resemble those in human-human interactions. These findings provide actionable guidance for benchmark design, RAG system development, and LLM training. Code is available at https://github.com/OuchengLiu/LLM-Misalign-Climate-Change.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑