FinRule-Bench：一个用于金融表和原则联合推理的基准
FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles
AI总结：
FinRule-Bench是一个评估金融表与原则联合推理能力的基准，通过三个任务测试模型在规则验证、识别和多违规诊断中的表现，揭示LLMs在高风险金融分析中的局限性。
摘要：
大型语言模型（LLMs）越来越多地应用于金融分析，但它们在显式会计原则下审计结构化财务报表的能力仍缺乏深入研究。现有基准主要评估问答、数值推理或异常检测在合成损坏数据上的表现，使得不清楚模型是否能可靠地验证或定位规则符合性在正确财务报表上。我们引入了FinRule-Bench，这是一个评估基于规则的金融推理诊断完整性的基准，涵盖四种标准报表类型：资产负债表、现金流量表、损益表和权益表。该基准定义了三个审计任务，需要逐渐更强的推理能力：（i）规则验证，测试单个原则的符合性；（ii）规则识别，需要从提供的规则集中选择违反的原则；（iii）联合规则诊断，需要检测和定位多个同时违反在记录层面。我们评估LLMs在零样本和少样本提示下的表现，并引入一种因果-反事实推理协议，强制决策、解释和反事实判断之间的一致性。在各种任务和报表类型上，我们发现虽然模型在孤立规则验证上表现良好，但在规则鉴别和多违规诊断上性能急剧下降。FinRule-Bench为研究规则导向的推理、诊断覆盖和LLMs在高风险金融分析中的失败模式提供了系统且可重复的测试平台。
英文摘要：
Large language models (LLMs) are increasingly applied to financial analysis, yet their ability to audit structured financial statements under explicit accounting principles remains poorly explored. Existing benchmarks primarily evaluate question answering, numerical reasoning, or anomaly detection on synthetically corrupted data, making it unclear whether models can reliably verify or localize rule compliance on correct financial statements. We introduce FinRule-Bench, a benchmark for evaluating diagnostic completeness in rule-based financial reasoning over real-world financial tables. FinRule-Bench pairs ground-truth financial statements with explicit, human-curated accounting principles and spans four canonical statement types: Balance Sheets, Cash Flow Statements, Income Statements, and Statements of Equity. The benchmark defines three auditing tasks that require progressively stronger reasoning capabilities: (i) rule verification, which tests compliance with a single principle; (ii) rule identification, which requires selecting the violated principle from a provided rule set; and (iii) joint rule diagnosis, which requires detecting and localizing multiple simultaneous violations at the record level. We evaluate LLMs under zero-shot and few-shot prompting, and introduce a causal-counterfactual reasoning protocol that enforces consistency between decisions, explanations, and counterfactual judgments. Across tasks and statement types, we find that while models perform well on isolated rule verification, performance degrades sharply for rule discrimination and multi-violation diagnosis. FinRule-Bench provides a principled and reproducible testbed for studying rule-governed reasoning, diagnostic coverage, and failure modes of LLMs in high-stakes financial analysis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
