不考虑语境信息：评估大语言模型对索引性成分的理解
Un-considering Contextual Information: Assessing LLMs' Understanding of Indexical Elements
AI总结：
本文首次系统评估大语言模型对英语索引性成分的共指消解能力，发布包含1600道多选题的English Indexical Dataset，发现模型对I表现良好但对you、here、tomorrow表现不佳，且句法线索对不同索引性成分的作用存在差异。
摘要：
大语言模型（LLMs）在共指消解相关任务中表现出了令人印象深刻的能力。然而，以往研究大多只评估了LLM在处理名词和第三人称代词时的共指消解表现。本研究评估LLM在处理I、you、here和tomorrow等索引性成分时的共指消解表现，这些成分因其语言特性而带来独特挑战。我们提出了首项考察LLM如何解读英语索引性成分的研究，并发布了包含1600道多项选择题的English Indexical Dataset。我们评估了GPT-4o、Claude 3.5 Sonnet、Gemini 1.5 Pro和DeepSeek V3等领先LLM。结果表明，LLM在处理某些索引性成分（I）时表现出色，而在处理其他索引性成分（you、here、tomorrow）时表现挣扎；同时，句法线索（如引号）有助于LLM处理某些索引性成分，却会降低其在其他索引性成分上的表现。代码和数据可在以下地址获取：https://github.com/metehanoguzz/LLMs-Indexicals-English。
英文摘要：
Large Language Models (LLMs) have demonstrated impressive performances in tasks related to coreference resolution. However, previous studies mostly assessed LLM performance on coreference resolution with nouns and third person pronouns. This study evaluates LLM performance on coreference resolution with indexical like I, you, here and tomorrow, which come with unique challenges due to their linguistic properties. We present the first study examining how LLMs interpret indexicals in English, releasing the English Indexical Dataset with 1600 multiple-choice questions. We evaluate pioneering LLMs, including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and DeepSeek V3. Our results reveal that LLMs exhibit an impressive performance with some indexicals (I), while struggling with others (you, here, tomorrow), and that syntactic cues (e.g. quotation) contribute to LLM performance with some indexicals, while they reduce performance with others. Code and data are available at: https://github.com/metehanoguzz/LLMs-Indexicals-English.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
