Ebisu:日本金融领域大语言模型基准测试
Ebisu: Benchmarking Large Language Models in Japanese Finance
AI总结:
Ebisu通过两个专家标注任务评估日本金融语言理解,揭示了现有LLMs在处理隐含承诺和嵌套金融术语时的局限性。
摘要:
日本金融领域结合了黏着语、头-final语言结构、混合书写系统以及高语境沟通规范,这些规范依赖于间接表达和隐含承诺,对LLMs构成了重大挑战。我们介绍了Ebisu,一个针对原生日本金融语言理解的基准测试,包含两个基于语言和文化背景、由专家标注的任务:JF-ICR,评估投资者面对问答中的隐含承诺和拒绝识别;JF-TE,评估从专业披露中提取和排序嵌套金融术语的层次结构。我们评估了一组多样化的开源和专有LLMs,涵盖通用型、日本适应型和金融型模型。结果显示,即使是最先进的系统在两项任务上也面临困难。虽然增加模型规模带来了有限的改进,但语言和领域特定的适应并未可靠地提升性能,留下大量未解决的差距。Ebisu提供了一个聚焦的基准测试,以推动基于语言和文化背景的金融NLP发展。所有数据集和评估脚本均已公开发布。
英文摘要:
Japanese finance combines agglutinative, head-final linguistic structure, mixed writing systems, and high-context communication norms that rely on indirect expression and implicit commitment, posing a substantial challenge for LLMs. We introduce Ebisu, a benchmark for native Japanese financial language understanding, comprising two linguistically and culturally grounded, expert-annotated tasks: JF-ICR, which evaluates implicit commitment and refusal recognition in investor-facing Q&A, and JF-TE, which assesses hierarchical extraction and ranking of nested financial terminology from professional disclosures. We evaluate a diverse set of open-source and proprietary LLMs spanning general-purpose, Japanese-adapted, and financial models. Results show that even state-of-the-art systems struggle on both tasks. While increased model scale yields limited improvements, language- and domain-specific adaptation does not reliably improve performance, leaving substantial gaps unresolved. Ebisu provides a focused benchmark for advancing linguistically and culturally grounded financial NLP. All datasets and evaluation scripts are publicly released.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。