RealFin: 大语言模型在用户未明说时如何进行金融推理?
RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?
AI总结:
RealFin通过移除金融问题中的关键前提来评估大语言模型的金融推理能力,发现通用模型易过度猜测,专业模型难以识别缺失信息,揭示了当前评估的不足。
摘要:
可靠的金融推理不仅需要知道如何回答,还需要知道何时无法证明答案。在现实金融实践中,问题常依赖于隐含假设而非显式陈述,导致问题看似可解但缺乏足够的信息来得出确定答案。我们介绍了REALFIN,一个双语基准,通过系统地从考试式问题中移除关键前提,同时保持语言上的合理性来评估金融推理能力。基于此,我们评估模型在三种形式下表现:回答、识别缺失信息和拒绝不合理的选项,并发现当关键条件缺失时,模型表现会持续下降。通用模型倾向于过度承诺和猜测,而大多数金融专业模型无法清晰识别缺失的前提。这些结果突显了当前评估中的关键差距,并表明可靠的金融模型必须知道何时不应回答问题。
英文摘要:
Reliable financial reasoning requires knowing not only how to answer, but also when an answer cannot be justified. In real financial practice, problems often rely on implicit assumptions that are taken for granted rather than stated explicitly, causing problems to appear solvable while lacking enough information for a definite answer. We introduce REALFIN, a bilingual benchmark that evaluates financial reasoning by systematically removing essential premises from exam-style questions while keeping them linguistically plausible. Based on this, we evaluate models under three formulations that test answering, recognizing missing information, and rejecting unjustified options, and find consistent performance drops when key conditions are absent. General-purpose models tend to over-commit and guess, while most finance-specialized models fail to clearly identify missing premises. These results highlight a critical gap in current evaluations and show that reliable financial models must know when a question should not be answered.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。