arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.20652cs.HCcs.AIcs.CY
收藏

当LLM无法帮助:在营养领域的LLM实证评估

When LLMs Can't Help: Real-World Evaluation of LLMs in Nutrition

Charles University, Faculty of Mathematics and Physics(查尔斯大学数学与物理学院);Saarland University(萨尔兰州大学);TU Darmstadt(图宾根大学);University of Aberdeen(阿伯丁大学)

Karen Jia-Hui Li, Simone Balloccu, Ondrej Dusek, Ehud Reiter

AI总结:

本文通过实证研究揭示了LLM在营养领域应用中的实际效果与内在评估之间的差距,指出需采用跨学科方法改进LLM的应用。

摘要:

随着大型语言模型(LLM)在聊天机器人形式中的信任度增加,这种信任往往受到其缺乏外在评估的限制。这在营养领域尤为明显,因为在该领域,随机对照试验(RCT)是金标准,专家要求其以支持基于证据的部署。LLM在该领域已显示出有希望的结果,但这些结果仅限于内在设置。我们通过运行第一个涉及LLM的营养RCT来填补这一空白。我们增强了一个基于规则的聊天机器人,加入了两个基于LLM的功能:(1)消息改写以实现对话的多样性与参与度,以及(2)通过微调模型进行营养咨询。在为期七周的RCT(n=81)中,我们比较了有无LLM集成的聊天机器人变体。我们测量了饮食结果、情绪福祉和参与度的影响。尽管我们的基于LLM的功能在内在评估中表现良好,但我们发现它们在现实世界部署中并未带来一致的好处。这些结果突显了内在评估与现实世界影响之间的关键差距,强调了需要跨学科、以人类为中心的方法的重要性。

英文摘要:

The increasing trust in large language models (LLMs), especially in the form of chatbots, is often undermined by the lack of their extrinsic evaluation. This holds particularly true in nutrition, where randomised controlled trials (RCTs) are the gold standard, and experts demand them for evidence-based deployment. LLMs have shown promising results in this field, but these are limited to intrinsic setups. We address this gap by running the first RCT involving LLMs for nutrition. We augment a rule-based chatbot with two LLM-based features: (1) message rephrasing for conversational variety and engagement, and (2) nutritional counselling through a fine-tuned model. In our seven-week RCT (n=81), we compare chatbot variants with and without LLM integration. We measure effects on dietary outcome, emotional well-being, and engagement. Despite our LLM-based features performing well in intrinsic evaluation, we find that they did not yield consistent benefits in real-world deployment. These results highlight critical gaps between intrinsic evaluations and real-world impact, emphasising the need for interdisciplinary, human-centred approaches.\footnote{We provide all of our code and results at: \\ \href{https://github.com/saeshyra/diet-chatbot-trial}{https://github.com/saeshyra/diet-chatbot-trial}}

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑