REAL：面向住房交易与服务的大型语言模型能力基准测试
REAL: Benchmarking Abilities of Large Language Models for Housing Transactions and Services
AI总结：
本文提出首个住房交易与服务领域LLM评估基准REAL，包含5316个条目、4大主题和14个类别，实验表明现有LLMs在该领域仍有较大改进空间。
摘要：
大型语言模型（LLMs）的发展极大地推动了聊天机器人在多个领域的进步。当前迫切需要评估LLMs能否像人类一样在住房交易与服务中扮演智能体角色。我们提出了房地产智能体大型语言模型评估基准（REAL），这是首个旨在评估LLMs在住房交易与服务领域能力的评估套件。REAL包含5,316个高质量评估条目，涵盖4个主题：记忆、理解、推理和幻觉。所有这些条目被组织为14个类别，用于评估LLMs是否具备住房交易与服务场景中的知识和能力。此外，REAL被用于评估最先进LLMs的性能。实验结果表明，LLMs要应用于房地产领域仍有显著的改进空间。
英文摘要：
The development of large language models (LLMs) has greatly promoted the progress of chatbot in multiple fields. There is an urgent need to evaluate whether LLMs can play the role of agent in housing transactions and services as well as humans. We present Real Estate Agent Large Language Model Evaluation (REAL), the first evaluation suite designed to assess the abilities of LLMs in the field of housing transactions and services. REAL comprises 5,316 high-quality evaluation entries across 4 topics: memory, comprehension, reasoning and hallucination. All these entries are organized as 14 categories to assess whether LLMs have the knowledge and ability in housing transactions and services scenario. Additionally, the REAL is used to evaluate the performance of most advanced LLMs. The experiment results indicate that LLMs still have significant room for improvement to be applied in the real estate field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
