arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01479cs.CL
收藏

Ebisu:日本金融领域大语言模型基准测试

Ebisu: Benchmarking Large Language Models in Japanese Finance

Xueqing Peng, Ruoyu Xiang, Fan Zhang, Mingzi Song, Mingyang Jiang, Yan Wang, Lingfei Qian, Taiki Hara, Yuqing Guo, Jimin Huang, Junichi Tsujii, Sophia Ananiadou

AI总结:

Ebisu通过两个专家标注任务评估日本金融语言理解,揭示了现有LLMs在处理隐含承诺和嵌套金融术语时的局限性。

摘要:

日本金融领域结合了黏着语、头-final语言结构、混合书写系统以及高语境沟通规范,这些规范依赖于间接表达和隐含承诺,对LLMs构成了重大挑战。我们介绍了Ebisu,一个针对原生日本金融语言理解的基准测试,包含两个基于语言和文化背景、由专家标注的任务:JF-ICR,评估投资者面对问答中的隐含承诺和拒绝识别;JF-TE,评估从专业披露中提取和排序嵌套金融术语的层次结构。我们评估了一组多样化的开源和专有LLMs,涵盖通用型、日本适应型和金融型模型。结果显示,即使是最先进的系统在两项任务上也面临困难。虽然增加模型规模带来了有限的改进,但语言和领域特定的适应并未可靠地提升性能,留下大量未解决的差距。Ebisu提供了一个聚焦的基准测试,以推动基于语言和文化背景的金融NLP发展。所有数据集和评估脚本均已公开发布。

英文摘要:

Japanese finance combines agglutinative, head-final linguistic structure, mixed writing systems, and high-context communication norms that rely on indirect expression and implicit commitment, posing a substantial challenge for LLMs. We introduce Ebisu, a benchmark for native Japanese financial language understanding, comprising two linguistically and culturally grounded, expert-annotated tasks: JF-ICR, which evaluates implicit commitment and refusal recognition in investor-facing Q&A, and JF-TE, which assesses hierarchical extraction and ranking of nested financial terminology from professional disclosures. We evaluate a diverse set of open-source and proprietary LLMs spanning general-purpose, Japanese-adapted, and financial models. Results show that even state-of-the-art systems struggle on both tasks. While increased model scale yields limited improvements, language- and domain-specific adaptation does not reliably improve performance, leaving substantial gaps unresolved. Ebisu provides a focused benchmark for advancing linguistically and culturally grounded financial NLP. All datasets and evaluation scripts are publicly released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑