The rise of large language models (LLMs) has led to more diverse and higher-quality machine-generated text. However, their high expressive power makes it difficult to control outputs based on specific business instructions. In response, benchmarks focusing on the controllability of LLMs have been developed, but several issues remain: (1) They primarily cover major languages like English and Chinese, neglecting low-resource languages like Japanese; (2) Current benchmarks employ task-specific evaluation metrics, lacking a unified framework for selecting models based on controllability across different use cases. To address these challenges, this research introduces LCTG Bench, the first Japanese benchmark for evaluating the controllability of LLMs. LCTG Bench provides a unified framework for assessing control performance, enabling users to select the most suitable model for their use cases based on controllability. By evaluating nine diverse Japanese-specific and multilingual LLMs like GPT-4, we highlight the current state and challenges of controllability in Japanese LLMs and reveal the significant gap between multilingual models and Japanese-specific models.
Evaluating Large Language Models on Controlled Generation Tasks
University of Southern California(南加州大学);University of California, Los Angeles(加州大学洛杉矶分校);Amazon(亚马逊);Google DeepMind(谷歌DeepMind);ETH Zurich(苏黎世联邦理工学院)
How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark
University of Arizona(亚利桑那大学);MOE Key Lab of Computational Linguistics, Peking University(北京大学计算语言学国家重点实验室);University of California, Santa Barbara(加州大学圣巴巴拉分校)
CoCoA-MT: A Dataset and Benchmark for Contrastive Controlled MT with Application to Formality
AWS AI Labs(亚马逊云科技人工智能实验室)
机器翻译(MT)任务通常被公式化为针对输入段返回单个翻译的任务。然而,在许多情况下,多个不同的翻译是有效的,适当的翻译可能取决于预期的目标受众,扬声器的特性,甚至扬声器之间的关系。在处理敬语时会出现一些特殊的问题,特别是从英语翻译成带有正式标记的语言。例如,句子“你确定吗?“可以翻译成德语“Sind Sie sich sicher?(正式登记)或“Bist du dir sicher?“(非正式)。使用错误或不一致的语气可能会被某些文化和人口统计学的用户视为不合适或不和谐。这项工作解决的问题,学习控制目标语言属性,在这种情况下,正式,从少量的标记对比数据。我们引入了一个注释数据集(CoCoA-MT)和相关的评估指标,用于训练和评估六种不同目标语言的形式控制MT模型。我们表明,我们可以通过对标记的对比数据进行微调来训练形式控制模型,在保持整体质量的同时实现高准确度(82%的域内和73%的域外)。
A Controlled Synthetic Benchmark for Educational Aspect-Based Sentiment Analysis
Intelligent Systems, Afeka Academic College of Engineering(阿法卡学术工程智能系统学院);School of Computer Science, Faculty of Sciences, Holon Institute of Technology(霍洛技术学院计算机科学学院)
ContextBias: Controlled Evaluation of Bias Persistence Under Context Shift in Text-to-Image Models
Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Munich Data Science Institute (MDSI)(慕尼黑数据科学研究所);University of Trento(特伦托大学);Orreco(奥雷科公司)