TCEval:利用热舒适性评估人工智能的认知与感知能力
TCEval: Using Thermal Comfort to Assess Cognitive and Perceptual Abilities of AI
AI总结:
TCEval通过热舒适性场景评估AI的跨模态推理、因果关联和适应性决策能力,揭示当前LLM在热舒适性领域存在基础推理能力但缺乏精确因果理解。
摘要:
在LLM任务特定基准测试中存在关键缺口。热舒适性,一种涉及环境因素和个人感知的复杂交互,包括感觉整合和适应性决策,是评估人工智能系统现实世界认知能力的理想范式。为此,我们提出了TCEval,这是首个评估人工智能三大核心认知能力的评估框架,即跨模态推理、因果关联和适应性决策,通过利用热舒适性场景和大语言模型(LLM)代理来实现。方法涉及初始化LLM代理以虚拟人格属性,引导其生成衣物隔热选择和热舒适反馈,并将输出与ASHRAE全球数据库和中国热舒适数据库进行验证。在四个LLM上的实验表明,尽管代理反馈与人类的精确一致性有限,但方向一致性在1 PMV容忍度下显著提高。统计测试显示,LLM生成的PMV分布与人类数据明显偏离,且在离散热舒适分类中代理表现接近随机。这些结果证实了TCEval作为人工智能生态效度认知图灵测试的可行性,表明当前LLM具有基础的跨模态推理能力,但缺乏对热舒适性变量非线性关系的精确因果理解。TCEval补充了传统基准,将AI评估重点从抽象任务熟练度转向具身、情境感知的感知和决策,为推进人工智能在以人为本的应用如智能建筑中提供有价值的见解。
英文摘要:
A critical gap exists in LLM task-specific benchmarks. Thermal comfort, a sophisticated interplay of environmental factors and personal perceptions involving sensory integration and adaptive decision-making, serves as an ideal paradigm for evaluating real-world cognitive capabilities of AI systems. To address this, we propose TCEval, the first evaluation framework that assesses three core cognitive capacities of AI, cross-modal reasoning, causal association, and adaptive decision-making, by leveraging thermal comfort scenarios and large language model (LLM) agents. The methodology involves initializing LLM agents with virtual personality attributes, guiding them to generate clothing insulation selections and thermal comfort feedback, and validating outputs against the ASHRAE Global Database and Chinese Thermal Comfort Database. Experiments on four LLMs show that while agent feedback has limited exact alignment with humans, directional consistency improves significantly with a 1 PMV tolerance. Statistical tests reveal that LLM-generated PMV distributions diverge markedly from human data, and agents perform near-randomly in discrete thermal comfort classification. These results confirm the feasibility of TCEval as an ecologically valid Cognitive Turing Test for AI, demonstrating that current LLMs possess foundational cross-modal reasoning ability but lack precise causal understanding of the nonlinear relationships between variables in thermal comfort. TCEval complements traditional benchmarks, shifting AI evaluation focus from abstract task proficiency to embodied, context-aware perception and decision-making, offering valuable insights for advancing AI in human-centric applications like smart buildings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。