Instruction-tuned Large Language Models (LLMs) have recently showcased remarkable advancements in their ability to generate fitting responses to natural language instructions. However, many current works rely on manual evaluation to judge the quality of generated responses. Since such manual evaluation is time-consuming, it does not easily scale to the evaluation of multiple models and model variants. In this short paper, we propose a straightforward but remarkably effective evaluation metric called SemScore, in which we directly compare model outputs to gold target responses using semantic textual similarity (STS). We conduct a comparative evaluation of the model outputs of 12 prominent instruction-tuned LLMs using 8 widely-used evaluation metrics for text generation. We find that our proposed SemScore metric outperforms all other, in many cases more complex, evaluation metrics in terms of correlation to human evaluation. These findings indicate the utility of our proposed metric for the evaluation of instruction-tuned LLMs.
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Utah（犹他大学）
Enabling robots to understand instructions provided via spoken natural language would facilitate interaction between robots and people in a variety of settings in homes and workplaces. However, natural language instructions are often missing information that would be obvious to a human based on environmental context and common sense, and hence does not need to be explicitly stated. In this paper, we introduce Language-Model-based Commonsense Reasoning (LMCR), a new method which enables a robot to listen to a natural language instruction from a human, observe the environment around it, and auto…
Eliciting Causal Abilities in Large Language Models for Reasoning Tasks
BNU-HKBU United International College（北师港浸大）；Hong Kong Baptist University（香港浸会大学）；Beijing Normal University at Zhuhai（北京师范大学珠海校区）；The Chinese University of Hong Kong（香港中文大学）
Automatic Slide Updating with User-Defined Dynamic Templates and Natural Language Instructions
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Institute of Artificial Intelligence and Future Networks, Beijing Normal University（北京师范大学人工智能与未来网络研究院）；Faculty of Arts and Sciences, Beijing Normal University（北京师范大学文理学院）；Beijing Normal-Hong Kong Baptist University（北京师范大学-香港 Baptist大学）
T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Models
Guilin University of Electronic Technology（桂林电子科技大学）；University of Arizona（亚利桑那大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of California, Berkeley（加州大学伯克利分校）；Arizona State University（亚利桑那州立大学）
AFD-INSTRUCTION: A Comprehensive Antibody Instruction Dataset with Functional Annotations for LLM-Based Understanding and Design
National Institute for Data Science in Health and Medicine（健康医学数据科学国家研究院）；Institute of Artificial Intelligence（人工智能研究院）；School of Life Sciences（生命科学学院）；School of Informatics（信息学院）；State Key Laboratory of Vaccines for Infectious Diseases（传染病疫苗国家重点实验室）；Xiang An Biomedicine Laboratory（翔安生物医学实验室）
This research report introduces the generation of textual entailment within the project CSIEC (Computer Simulation in Educational Communication), an interactive web-based human-computer dialogue system with natural language for English instruction. The generation of textual entailment (GTE) is critical to the further improvement of CSIEC project. Up to now we have found few literatures related with GTE. Simulating the process that a human being learns English as a foreign language we explore our naive approach to tackle the GTE problem and its algorithm within the framework of CSIEC, i.e. rule…
An important, yet largely unstudied, problem in student data analysis is to detect misconceptions from students' responses to open-response questions. Misconception detection enables instructors to deliver more targeted feedback on the misconceptions exhibited by many students in their class, thus improving the quality of instruction. In this paper, we propose a new natural language processing-based framework to detect the common misconceptions among students' textual responses to short-answer questions. We propose a probabilistic model for students' textual responses involving misconceptions …
University of Southern California（南加州大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；The Pennsylvania State University（宾夕法尼亚州立大学）；University of California, Davis（加利福尼亚大学戴维斯分校）
Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）；Arizona State University（亚利桑那州立大学）
Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct t…
OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Yuanpei College, Peking University（北京大学元培学院）；Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
InstructProtein: Aligning Human and Protein Language via Knowledge Instruction
Zhejiang University（浙江大学）；ZJU-Hangzhou Global Scientific and Technological Innovation Center（浙江大学杭州国际科创中心）；AZFT Joint Lab for Knowledge Engine（AZFT知识引擎联合实验室）
GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning
Peking University（北京大学）；Mila（米拉研究所）；The University of Hong Kong（香港大学）；University of Oxford（牛津大学）；Hong Kong University of Science and Technology（香港科技大学）
Adobe Systems（奥多比系统公司）；Mila, Université de Montréal（米拉-蒙特利尔大学）
In this work, we focus on the problem of grounding language by training an agent to follow a set of natural language instructions and navigate to a target object in an environment. The agent receives visual information through raw pixels and a natural language instruction telling what task needs to be achieved and is trained in an end-to-end way. We develop an attention mechanism for multi-modal fusion of visual and textual modalities that allows the agent to learn to complete the task and achieve language grounding. Our experimental results show that our attention mechanism outperforms the ex…
An Evaluation of Interleaved Instruction Tuning on Semantic Reasoning Performance in an Audio MLLM
The Graduate Center, CUNY（CUNY研究生中心）；Brooklyn College, CUNY（CUNY布鲁克林学院）；Borough of Manhattan Community College, CUNY（CUNY曼哈顿社区学院）；The City College of New York, CUNY（CUNY纽约城市学院）
