arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.10652cs.CLcs.AIcs.CY
收藏

大型语言模型能否模拟人类回答?一项关于供暖相关选择中陈述性偏好实验的案例研究

Can Large Language Models Simulate Human Responses? A Case Study of Stated Preference Experiments in the Context of Heating-related Choices

Imperial College London(伦敦帝国理工学院)

Han Wang, Jacek Pawlak, Aruna Sivakumar

AI总结:

本研究通过供暖相关陈述性偏好实验系统评估了多个大型语言模型模拟人类选择的能力,发现推理模型DeepSeek-R1准确率最高(77%),并揭示了先前选择是最有效输入因素及系统性偏差的存在。

摘要:

陈述性偏好(SP)调查是研究个体在假设性乃至未来情境中如何进行权衡的关键方法。在能源领域,这包括关键的脱碳赋能情境,如低碳技术、分布式可再生能源发电和需求侧响应[1,2]。然而,这类调查往往成本高昂、耗时,并且可能受到受访者疲劳和伦理约束的影响。大型语言模型(LLMs)在生成类人文本回答方面展现出卓越能力,促使人们对其在调查研究中的应用日益关注。本研究探讨了使用LLMs模拟能源相关SP调查中的消费者选择,并探索将其整合到数据分析工作流程中。设计了一系列测试场景,以系统评估多个LLMs(LLaMA 3.1、Mistral、GPT-3.5和DeepSeek-R1)在个体和聚合层面的模拟性能,同时考虑了提示设计、上下文学习(ICL)、思维链(CoT)推理、LLM类型、与传统选择模型的整合以及潜在偏差等情境因素。基于云的LLMs并未始终优于较小的本地模型。在本研究中,推理模型DeepSeek-R1取得了最高的平均准确率(77%),并在准确率、因素识别和选择分布对齐方面优于非推理LLMs。在所有模型中,观察到对燃气锅炉和不改造选项的系统性偏差,并偏好更节能的替代方案。研究结果表明,先前的SP选择是最有效的输入因素,而包含额外因素和多样格式的较长提示会导致LLMs失去焦点,从而降低准确率。

英文摘要:

Stated preference (SP) surveys are a key method to research how individuals make trade-offs in hypothetical, also futuristic, scenarios. In energy context this includes key decarbonisation enablement contexts, such as low-carbon technologies, distributed renewable energy generation, and demand-side response [1,2]. However, they tend to be costly, time-consuming, and can be affected by respondent fatigue and ethical constraints. Large language models (LLMs) have demonstrated remarkable capabilities in generating human-like textual responses, prompting growing interest in their application to survey research. This study investigates the use of LLMs to simulate consumer choices in energy-related SP surveys and explores their integration into data analysis workflows. A series of test scenarios were designed to systematically assess the simulation performance of several LLMs (LLaMA 3.1, Mistral, GPT-3.5 and DeepSeek-R1) at both individual and aggregated levels, considering contexts factors such as prompt design, in-context learning (ICL), chain-of-thought (CoT) reasoning, LLM types, integration with traditional choice models, and potential biases. Cloud-based LLMs do not consistently outperform smaller local models. In this study, the reasoning model DeepSeek-R1 achieves the highest average accuracy (77%) and outperforms non-reasoning LLMs in accuracy, factor identification, and choice distribution alignment. Across models, systematic biases are observed against the gas boiler and no-retrofit options, with a preference for more energy-efficient alternatives. The findings suggest that previous SP choices are the most effective input factor, while longer prompts with additional factors and varied formats can cause LLMs to lose focus, reducing accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑