QUENCH：衡量LLMs在印度与非印度语境常识推理中的差距
QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs
AI总结：
QUENCH基准通过YouTube问答视频人工构建文本问答，评估LLMs在印度与非印度地理语境下的常识推理差距，发现模型规模、提示风格和地理语境影响推理能力，并分析常见错误。
摘要：
大型语言模型（LLMs）的兴起催生了对超越传统设置的先进基准测试系统的需求。为此，我们引入了QUENCH，这是一个新颖的基于文本的英语问答基准，由YouTube问答视频人工策划并转录而成。QUENCH包含掩蔽实体和推理依据，供LLMs通过生成进行预测。在地理语境与常识推理的交汇处，QUENCH通过零样本、开放领域的问答设置，帮助评估LLMs的世界知识和演绎能力。我们对7个LLMs和4个指标进行了广泛评估，调查了模型规模、提示风格、地理语境和黄金标注推理依据生成的影响。该基准测试以对LLMs易犯错误的误差分析作为结论。
英文摘要：
The rise of large language models (LLMs) has created a need for advanced benchmarking systems beyond traditional setups. To this end, we introduce QUENCH, a novel text-based English Quizzing Benchmark manually curated and transcribed from YouTube quiz videos. QUENCH possesses masked entities and rationales for the LLMs to predict via generation. At the intersection of geographical context and common sense reasoning, QUENCH helps assess world knowledge and deduction capabilities of LLMs via a zero-shot, open-domain quizzing setup. We perform an extensive evaluation on 7 LLMs and 4 metrics, investigating the influence of model size, prompting style, geographical context, and gold-labeled rationale generation. The benchmarking concludes with an error analysis to which the LLMs are prone.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
