Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining
Engineering Research Center of Big Data Intelligence, Ministry of Education（教育部大数据智能工程研究中心）；Fujian Key Laboratory of Network Computing and Intelligent Information Processing（福建省网络计算与智能信息处理重点实验室）；College of Computer and Data Science, Fuzhou University（福州大学计算机与数据科学学院）；Department of Computer Science, Emory University（埃默里大学计算机科学系）
Yanchao Tan, Hang Lv, Pengxiang Zhan, Shiping Wang, Carl Yang
Graphs with abundant attributes are essential in modeling interconnected entities and enhancing predictions across various real-world applications. Traditional Graph Neural Networks (GNNs) often require re-training for different graph tasks and datasets. Although the emergence of Large Language Models (LLMs) has introduced new paradigms in natural language processing, their potential for generic graph mining, training a single model to simultaneously handle diverse tasks and datasets, remains under-explored. To this end, our novel framework MuseGraph, seamlessly integrates the strengths of GNNs and LLMs into one foundation model for graph mining across tasks and datasets. This framework first features a compact graph description to encapsulate key graph information within language token limitations. Then, we propose a diverse instruction generation mechanism with Chain-of-Thought (CoT)-based instruction packages to distill the reasoning capabilities from advanced LLMs like GPT-4. Finally, we design a graph-aware instruction tuning strategy to facilitate mutual enhancement across multiple tasks and datasets while preventing catastrophic forgetting of LLMs' generative abilities. Our experimental results demonstrate significant improvements in five graph tasks and ten datasets, showcasing the potential of our MuseGraph in enhancing the accuracy of graph-oriented downstream tasks while improving the generation abilities of LLMs.
ESAinsTOD: A Unified End-to-End Schema-Aware Instruction-Tuning Framework for Task-Oriented Dialog Modeling
Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology（社会科学与信息检索研究中心，哈尔滨工业大学）；School of Computer Science and Engineering, Central South University（计算机科学与工程学院，中南大学）
A Hybrid Supervised-LLM Pipeline for Actionable Suggestion Mining in Unstructured Customer Reviews
Department of Computer Science & Information Systems, Birla Institute of Technology and Science, Pilani, India（印度比拉理工学院计算机科学与信息系统系）；Birdeye Inc.（Birdeye公司）
MoDS: Model-oriented Data Selection for Instruction Tuning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Wuhan AI Research（武汉人工智能研究院）
ReVision: A Dataset and Baseline VLM for Privacy-Preserving Task-Oriented Visual Instruction Rewriting
School of Information（信息学院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Department of Statistics and Data Science（统计与数据科学系）；Yale University（耶鲁大学）；School of Computing and Augmented Intelligence（计算与增强智能学院）
Guided troubleshooting is an inherent task in the domain of technical support services. When a customer experiences an issue with the functioning of a technical service or a product, an expert user helps guide the customer through a set of steps comprising a troubleshooting procedure. The objective is to identify the source of the problem through a set of diagnostic steps and observations, and arrive at a resolution. Procedures containing these set of diagnostic steps and observations in response to different problems are common artifacts in the body of technical support documentation. The abi…
EVIT: Event-Oriented Instruction Tuning for Event Reasoning
School of Computer Science, Peking University（北京大学计算机学院）；Key Laboratory of High Confidence Software Technologies (PKU), MOE, China（高可信软件技术重点实验室（北京大学））；Advanced Institute of Big Data（大数据先进研究院）
TAL Education Group（好未来教育集团）；Guangdong Institute of Smart Education（广东智慧教育研究院）；Jinan University（暨南大学）
Online dialogic instructions are a set of pedagogical instructions used in real-world online educational contexts to motivate students, help understand learning materials, and build effective study habits. In spite of the popularity and advantages of online learning, the education technology and educational data mining communities still suffer from the lack of large-scale, high-quality, and well-annotated teaching instruction datasets to study computational approaches to automatically detect online dialogic instructions and further improve the online teaching effectiveness. Therefore, in this …
Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ZJU-Ant Group Joint Research Center for Knowledge Graphs, Zhejiang University（浙江大学-蚂蚁集团知识图谱联合研究中心）；Hangzhou Innovation Center, Zhejiang University（浙江大学杭州创新中心）
Cornell University（康奈尔大学）；City University of New York（纽约市立大学）
We analyze language change over time in a collaborative, goal-oriented instructional task, where utility-maximizing participants form conventions and increase their expertise. Prior work studied such scenarios mostly in the context of reference games, and consistently found that language complexity is reduced along multiple dimensions, such as utterance length, as conventions are formed. In contrast, we find that, given the ability to increase instruction utility, instructors increase language complexity along these previously studied dimensions to better collaborate with increasingly skilled …
University of Southern California（南加州大学）；University of Chicago（芝加哥大学）；Temple University（天普大学）；University of Birmingham（伯明翰大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）；Penn State University（宾夕法尼亚州立大学）
LLMs-based Few-Shot Disease Predictions using EHR: A Novel Approach Combining Predictive Agent Reasoning and Critical Agent Instruction
Emory University（埃默里大学）；University of Washington（华盛顿大学）；UCSD（加利福尼亚大学圣迭戈分校）；Rollins School of Public Health, Emory University（埃默里大学罗林斯公共卫生学院）；School of Medicine, Emory University（埃默里大学医学院）
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information, Renmin University of China（中国人民大学信息学院）；Baichuan Inc.（百川公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；University of Sydney（悉尼大学）
Can Frontier LLMs Replace Annotators in Biomedical Text Mining? Analyzing Challenges and Exploring Solutions
The University of Tokyo（东京大学）；Graduate School of Frontier Sciences（前沿科学研究生院）；Department of Computational Biology and Medical Sciences（计算生物学与医学科学系）；Database Center for Life Science（生命科学数据库中心）；Joint Support-Center for Data Science Research（数据科学研究联合支持中心）
Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context
School of Data Science, Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院、深圳大数据研究院）；Bytedance（字节跳动）
Energy Estimates Across Layers of Computing: From Devices to Large-Scale Applications in Machine Learning for Natural Language Processing, Scientific Computing, and Cryptocurrency Mining
SLAC National Laboratory（SLAC国家实验室）；Stanford University（斯坦福大学）
A Prompt Learning Framework for Source Code Summarization
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）
To perform tasks specified by natural language instructions, autonomous agents need to extract semantically meaningful representations of language and map it to visual elements and actions in the environment. This problem is called task-oriented language grounding. We propose an end-to-end trainable neural architecture for task-oriented language grounding in 3D environments which assumes no prior linguistic or perceptual knowledge and requires only raw pixels from the environment and the natural language instruction as input. The proposed model combines the image and text representations using…
MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Xiaomi AI Lab（小米AI实验室）；School of Computer Science, Wuhan University（武汉大学计算机学院）
As labeling cost for different modules in task-oriented dialog (ToD) systems is high, a major challenge in practice is to learn different tasks with the least amount of labeled data. Recently, prompting methods over pre-trained language models (PLMs) have shown promising results for few-shot learning in ToD. To better utilize the power of PLMs, this paper proposes Comprehensive Instruction (CINS) that exploits PLMs with extra task-specific instructions. We design a schema (definition, constraint, prompt) of instructions and their customized realizations for three important downstream tasks in …
