BaziQA-Benchmark：评估大语言模型中的符号性和时间组合推理
BaziQA-Benchmark: Evaluating Symbolic and Temporally Compositional Reasoning in Large Language Models
AI总结：
BaziQA-Benchmark通过结构化推理协议评估大语言模型在符号性和时间组合推理中的表现，揭示其在时间难度和推理顺序上的敏感性及系统性缺陷。
摘要：
我们提出了BaziQA-Benchmark，这是一个用于评估大语言模型中符号性和时间组合推理的标准基准。该基准来源于200个专业精选的多选问题，这些问题来自全球占卜竞赛（2021-2025），每个实例都需要在固定的符号图表上进行结构化推理，并与时间条件互动。与轶事或提示驱动的评估不同，BaziQA-Benchmark能够实现客观评分和跨年、领域和模型家族的受控比较。我们评估了当前语言模型在多轮设置下的表现，并分析了时间难度、推理领域和推理协议之间的性能变化。为进一步探究推理行为，我们引入了一种轻量的结构化推理协议，该协议限制推理顺序而不增加领域知识。结果表明，模型在一定程度上优于随机猜测，但仍然远未达到饱和，表现出对时间组合和推理顺序的明显敏感性，以及在精确时间定位和多条件符号判断上的系统性失败。
英文摘要：
We present BaziQA-Benchmark, a standardized benchmark for evaluating symbolic and temporally compositional reasoning in large language models. The benchmark is derived from 200 professionally curated, multiple-choice problems from the Global Fortune-teller Competition (2021--2025), where each instance requires structured inference over a fixed symbolic chart and interacting temporal conditions. Unlike anecdotal or prompt-driven evaluations, BaziQA-Benchmark enables objective scoring and controlled comparison across years, domains, and model families. We evaluate contemporary language models under a multi-turn setting and analyze performance variation across temporal difficulty, reasoning domains, and inference protocols.To further probe reasoning behavior, we introduce a lightweight Structured Reasoning Protocol that constrains inference order without adding domain knowledge. Results show that models consistently outperform chance but remain far from saturation, exhibiting pronounced sensitivity to temporal composition and reasoning order, as well as systematic failures on precise temporal localization and multi-condition symbolic judgments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
