arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.41124cs.AI
收藏

自信的傻瓜还是谨慎的探索者?大型语言模型何时超越静态内部记忆进行搜索的行为研究

Confident Fools or Careful Searchers? A Behavioural Study of When LLMs Search Beyond Their Static Internal Memory

Rensselaer Polytechnic Institute(伦斯勒理工学院)

Nipun D. Pathirage, Kwan Srijomkwan

AI总结:

本研究通过含400道题的基准测试,分析五款商用LLM的搜索决策,发现不同模型对OK/IK问题的搜索表现差异大,搜索部分受时间线索触发,相关分数衡量产品搜索触发情况而非严格元认知。

摘要:

智能大型语言模型(LLM)助手必须反复决定是从内部记忆中回答问题,还是调用网络搜索等工具。不必要的搜索会增加延迟和成本,而遗漏搜索则会产生自信但无依据的答案。现有研究大多测量口头表达的置信度或训练新的自适应检索方法,很少审计已部署产品实际做出的搜索决策。我们引入了一个包含400道选择题的基准,该基准结合了广为人知的内部知识(IK)问题(应直接回答)和外部知识(OK)问题(涉及模型训练截止日期后发布的新闻事实,据我们所知,应触发搜索,若无工具则弃权(不执行))。每个题目提供弃权选项,以及两个动作级分数NAS和KIS,分别奖励对OK问题进行搜索和对IK问题保持克制。在五个商用LLM及其厂商提供的网络搜索工具中,GPT-5 mini和Grok 4 Fast对超过90%的OK问题进行搜索,但仅对1%的IK问题进行搜索;而Mistral Small仅对20%的OK问题进行搜索,常选择弃权而非使用工具;某产品的搜索指标被证明无法使用。搜索部分由明确的时间线索触发:在无时间线索的OK问题上,这两个选择性模型的NAS从约0.72降至0.57-0.58。因此,这些分数衡量的是特定产品的搜索触发情况,而非严格意义上的元认知。

英文摘要:

Agentic LLM assistants must repeatedly decide whether to answer from memory or to call a tool such as web search. Unnecessary searches cost latency and money; missing ones yield confident but unsupported answers. Prior work mostly measures verbalised confidence or trains new adaptive-retrieval methods, and rarely audits the search decisions that deployed products actually make. We introduce a benchmark of 400 multiple-choice questions that combines widely known inside-knowledge (IK) questions, which should be answered directly, with outside-knowledge (OK) questions about facts from news published after the models' training cut-offs (to our knowledge), which should trigger search or, without tools, abstention. Each item offers an abstain option, and two action-level scores, NAS and KIS, reward searching on OK questions and restraint on IK questions. Across five commercial LLMs with and without their vendors' web search, GPT-5 mini and Grok 4 Fast search on over 90% of OK but only 1% of IK questions, whereas Mistral Small searches on just 20% of OK questions, often abstaining instead of using its tool; one product's search indicator proved unusable. Searching is partly triggered by explicit temporal cues: on uncued OK questions, the NAS of the two selective models falls from about 0.72 to 0.57--0.58. The scores thus measure the search triggering of specific products, not metacognition in a strict sense.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑