Automatic evaluation of natural language generation has long been an elusive goal in NLP.A recent paradigm fine-tunes pre-trained language models to emulate human judgements for a particular task and evaluation criterion. Inspired by the generalization ability of instruction-tuned models, we propose a learned metric based on instruction tuning. To test our approach, we collected HEAP, a dataset of human judgements across various NLG tasks and evaluation criteria. Our findings demonstrate that instruction tuning language models on HEAP yields good performance on many evaluation tasks, though some criteria are less trivial to learn than others. Further, jointly training on multiple tasks can yield additional performance improvements, which can be beneficial for future tasks with little to no human annotated data.
Agents that can follow language instructions are expected to be useful in a variety of situations such as navigation. However, training neural network-based agents requires numerous paired trajectories and languages. This paper proposes using multimodal generative models for semi-supervised learning in the instruction following tasks. The models learn a shared representation of the paired data, and enable semi-supervised learning by reconstructing unpaired data through the representation. Key challenges in applying the models to sequence-to-sequence tasks including instruction following are le…
University of Toronto（多伦多大学）；Vector Institute（矢量研究所）；University of Waterloo（滑铁卢大学）
By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans. Inspired by classical program synthesis and the human approach to prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic instruction generation and selection. In our method, we treat the instruction as the "program," optimized by searching over a pool of instruction c…
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
ShanghaiTech University（上海科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Meituan（美团）；Alibaba Group（阿里巴巴集团）；The Hong Kong University of Science and Technology（香港科技大学）
Finetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completion
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；National Institute of Healthcare Data Science, Nanjing University（南京大学国家卫生健康数据科学研究院）
Singapore Management University（新加坡管理大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Interests in the automatic generation of cooking recipes have been growing steadily over the past few years thanks to a large amount of online cooking recipes. We present RecipeGPT, a novel online recipe generation and evaluation system. The system provides two modes of text generations: (1) instruction generation from given recipe title and ingredients; and (2) ingredient generation from recipe title and cooking instructions. Its back-end text generation module comprises a generative pre-trained language model GPT-2 fine-tuned on a large cooking recipe dataset. Moreover, the recipe evaluation…
Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?
Tohoku University（东洋大学）；Research and Development Center for Large Language Models, National Institute of Informatics（大型语言模型研究与开发中心，日本信息处理学会）；National Institute of Informatics（日本信息处理学会）
Evaluation of Automatic Speech Recognition Using Generative Large Language Models
Idiap Research Institute（Idiap研究 institute）；EPFL（瑞士联邦理工学院）；Brno University of Technology（布拉格技术大学）；Avignon University（阿维尼翁大学）；Le Mans University（勒曼大学）；Nantes University（南特大学）
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use
Hebrew University（希伯来大学）；Google Research（谷歌研究院）；UCLA（加州大学洛杉矶分校）；Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）；UCSB（加州大学圣塔芭芭拉分校）；Stanford（斯坦福大学）；LAION
Beijing University of Posts and Telecommunications（北京邮电大学）；Meituan Inc.（美团公司）
Recently, prompt-based methods have achieved significant performance in few-shot learning scenarios by bridging the gap between language model pre-training and fine-tuning for downstream tasks. However, existing prompt templates are mostly designed for sentence-level tasks and are inappropriate for sequence labeling objectives. To address the above issue, we propose a multi-task instruction-based generative framework, named InstructionNER, for low-resource named entity recognition. Specifically, we reformulate the NER task as a generation problem, which enriches source sentences with task-spec…
National Institute of Information and Communication Technology（信息通信技术研究所）
This paper focuses on a multimodal language understanding method for carry-and-place tasks with domestic service robots. We address the case of ambiguous instructions, that is, when the target area is not specified. For instance "put away the milk and cereal" is a natural instruction where there is ambiguity regarding the target area, considering environments in daily life. Conventionally, this instruction can be disambiguated from a dialogue system, but at the cost of time and cumbersome interaction. Instead, we propose a multimodal approach, in which the instructions are disambiguated using …
Online one-on-one class is created for highly interactive and immersive learning experience. It demands a large number of qualified online instructors. In this work, we develop six dialogic instructions and help teachers achieve the benefits of one-on-one learning paradigm. Moreover, we utilize neural language models, i.e., long short-term memory (LSTM), to detect above six instructions automatically. Experiments demonstrate that the LSTM approach achieves AUC scores from 0.840 to 0.979 among all six types of instructions on our real-world educational dataset.
Automatic Slide Updating with User-Defined Dynamic Templates and Natural Language Instructions
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Institute of Artificial Intelligence and Future Networks, Beijing Normal University（北京师范大学人工智能与未来网络研究院）；Faculty of Arts and Sciences, Beijing Normal University（北京师范大学文理学院）；Beijing Normal-Hong Kong Baptist University（北京师范大学-香港 Baptist大学）
