探索大语言模型的极限：通过MskQA和MskCal对掩码文本处理能力进行系统评估
Exploring the Limits of Large Language Models: A Systematic Evaluation of Masked Text Processing Ability through MskQA and MskCal
AI总结：
本研究通过新任务MskQA和MskCal系统评估LLMs的掩码文本处理能力，发现其性能高度依赖语义线索，且GPT-4o优于4o-mini，揭示了模型依赖表面模式及语义线索在推理中的关键作用。
摘要：
本文通过严格评估大语言模型（LLMs）处理掩码文本的能力，揭示了其局限性。我们引入了两个新任务：MskQA，用于衡量在类似RealtimeQA的掩码问答数据集上的推理能力；以及MskCal，用于评估在掩码算术问题上的数值推理能力。对GPT-4o和4o-mini的测试表明，尽管LLMs对掩码文本表现出一定的韧性，但其性能高度依赖于掩码率和语义线索。具体而言，“完全掩码”（即语义线索完全缺失）相比“部分揭示”（即保留部分语义信息）会导致性能显著下降，这表明LLMs依赖于表面层面的模式。有趣的是，GPT-4o始终优于4o-mini，尤其是在MskCal中，展示了其在处理带掩码文本的数值推理方面更强的能力。这突显了语义线索在LLMs推理过程中的关键作用。我们的研究阐明了背景知识与推理能力在掩码文本处理中的相互作用，为更深入地理解LLM的能力和局限性铺平了道路，并强调了需要更稳健的评估方法来准确衡量其真实理解能力。
英文摘要：
This paper sheds light on the limitations of Large Language Models (LLMs) by rigorously evaluating their ability to process masked text. We introduce two novel tasks: MskQA, measuring reasoning on masked question-answering datasets like RealtimeQA, and MskCal, assessing numerical reasoning on masked arithmetic problems.Testing GPT-4o and 4o-mini reveals that while LLMs exhibit some resilience to masked text, their performance is highly contingent on masking rates and semantic cues. Specifically, "solid masking," where semantic clues are entirely absent, leads to a significant performance drop compared to "partial lifting," where some semantic information is retained, indicating LLMs' reliance on surface-level patterns. Interestingly, GPT-4o consistently outperforms 4o-mini, particularly in MskCal, demonstrating a greater ability to handle numerical reasoning with masked text. This underscores the crucial role of semantic cues in the reasoning process of LLMs. Our study illuminates the interplay between background knowledge and reasoning ability in masked text processing, paving the way for a deeper understanding of LLM capabilities and limitations, and highlighting the need for more robust evaluation methods to accurately assess their true comprehension abilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
