Recent work studies the cognitive capabilities of language models through psychological tests designed for humans. While these studies are helpful for understanding the general capabilities of these models, there is no guarantee that a model possessing sufficient capabilities to pass those tests would actually use those capabilities in performing real-life tasks. In this work, we formulate task-oriented cognitive capabilities, which are human-like cognitive capabilities that language models leverage to perform tasks. These capabilities are (i) the ability to quickly generate good candidate utterances (the search capability) (ii) the ability to predict how a listener interprets those utterances and choose the most appropriate one (the pragmatic capability). We design an evaluation scheme for comparing these capabilities of a language model with those of a human. Applying this scheme to examine various models in a navigation instruction generation problem, we find that their pragmatic capability is severely lacking. This insight leads us to augment them with better models of the listener and obtain a significant boost of 11% in success rate in guiding real humans. Our work advocates for having a principled procedure for aligning language models with humans that involves (i) formulating task-oriented capabilities, (ii) devising a method to quantify their deficiency, and (iii) iteratively improving them.
MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Xiaomi AI Lab(小米AI实验室);School of Computer Science, Wuhan University(武汉大学计算机学院)
MoDS: Model-oriented Data Selection for Instruction Tuning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Wuhan AI Research(武汉人工智能研究院)
ESAinsTOD: A Unified End-to-End Schema-Aware Instruction-Tuning Framework for Task-Oriented Dialog Modeling
Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology(社会科学与信息检索研究中心,哈尔滨工业大学);School of Computer Science and Engineering, Central South University(计算机科学与工程学院,中南大学)
ReVision: A Dataset and Baseline VLM for Privacy-Preserving Task-Oriented Visual Instruction Rewriting
School of Information(信息学院);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Department of Statistics and Data Science(统计与数据科学系);Yale University(耶鲁大学);School of Computing and Augmented Intelligence(计算与增强智能学院)
Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining
Engineering Research Center of Big Data Intelligence, Ministry of Education(教育部大数据智能工程研究中心);Fujian Key Laboratory of Network Computing and Intelligent Information Processing(福建省网络计算与智能信息处理重点实验室);College of Computer and Data Science, Fuzhou University(福州大学计算机与数据科学学院);Department of Computer Science, Emory University(埃默里大学计算机科学系)
EVIT: Event-Oriented Instruction Tuning for Event Reasoning
School of Computer Science, Peking University(北京大学计算机学院);Key Laboratory of High Confidence Software Technologies (PKU), MOE, China(高可信软件技术重点实验室(北京大学));Advanced Institute of Big Data(大数据先进研究院)
Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Ant Group Joint Research Center for Knowledge Graphs, Zhejiang University(浙江大学-蚂蚁集团知识图谱联合研究中心);Hangzhou Innovation Center, Zhejiang University(浙江大学杭州创新中心)
FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schema
University of Warwick(华威大学);Tecent YouTu Lab(腾讯优酷实验室);East China Normal University(华东师范大学);King’s College London(伦敦国王学院);The Alan Turing Institute(艾伦·图灵研究所)
Cognition Chain for Explainable Psychological Stress Detection on Social Media
University of Oxford(牛津大学);Tsinghua University(清华大学);Beijing Normal University(北京师范大学);Wuhan University(武汉大学);Oxford Suzhou Centre for Advanced Research(牛津苏州高等研究院)
Cognitive Fatigue in Autoregressive Transformers: Formalization and Measurement
Guru Gobind Singh Indraprastha University, India(古鲁·戈宾德·辛格·印度普拉斯塔大学);Artificial Intelligence Institute, University of South Carolina, USA(人工智能研究所,南卡罗来纳大学);Indian Institute of Technology, Kanpur, India(印度理工学院,坎浦尔);Indian AI Research Organization, India(印度人工智能研究组织)
University of Southern California(南加州大学);University of Chicago(芝加哥大学);Temple University(天普大学);University of Birmingham(伯明翰大学);Toyota Technological Institute at Chicago(芝加哥丰田技术研究院);Penn State University(宾夕法尼亚州立大学)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Baichuan Inc.(百川公司);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context
School of Data Science, Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院、深圳大数据研究院);Bytedance(字节跳动)