arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.15875cs.CL
收藏

LCTG Bench:LLM 控制文本生成基准

LCTG Bench: LLM Controlled Text Generation Benchmark

AI Shift Inc.(AI Shift公司);CyberAgent Inc.(CyberAgent公司);Tokyo Institute of Technology(东京工业大学)

Kentaro Kurihara, Masato Mita, Peinan Zhang, Shota Sasaki, Ryosuke Ishigami, Naoaki Okazaki

AI总结:

本文提出首个日语LLM可控性评估基准LCTG Bench,提供统一框架评估多语言及日语专用模型,揭示了多语言模型与日语专用模型在可控性上的显著差距。

摘要:

大型语言模型(LLMs)的兴起带来了更多样、更高质量的机器生成文本。然而,其高表达能力使得根据特定业务指令控制输出变得困难。为此,专注于 LLM 可控性的基准已被开发,但仍存在几个问题:(1)它们主要覆盖英语和中文等主要语言,忽视了日语等低资源语言;(2)当前基准采用特定任务的评估指标,缺乏基于不同用例的可控性来选择模型的统一框架。为应对这些挑战,本研究引入了 LCTG Bench,这是首个用于评估 LLM 可控性的日语基准。LCTG Bench 提供了一个评估控制性能的统一框架,使用户能够基于可控性为其用例选择最合适的模型。通过评估 GPT-4 等九款多样的日语专用和多语言 LLM,我们突出了日语 LLM 可控性的现状与挑战,并揭示了多语言模型与日语专用模型之间的显著差距。

英文摘要:

The rise of large language models (LLMs) has led to more diverse and higher-quality machine-generated text. However, their high expressive power makes it difficult to control outputs based on specific business instructions. In response, benchmarks focusing on the controllability of LLMs have been developed, but several issues remain: (1) They primarily cover major languages like English and Chinese, neglecting low-resource languages like Japanese; (2) Current benchmarks employ task-specific evaluation metrics, lacking a unified framework for selecting models based on controllability across different use cases. To address these challenges, this research introduces LCTG Bench, the first Japanese benchmark for evaluating the controllability of LLMs. LCTG Bench provides a unified framework for assessing control performance, enabling users to select the most suitable model for their use cases based on controllability. By evaluating nine diverse Japanese-specific and multilingual LLMs like GPT-4, we highlight the current state and challenges of controllability in Japanese LLMs and reveal the significant gap between multilingual models and Japanese-specific models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑