This paper aims to efficiently enable Large Language Models (LLMs) to use multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have shown great potential for tool usage through sophisticated prompt engineering. Nevertheless, these models typically rely on prohibitive computational costs and publicly inaccessible data. To address these challenges, we propose the GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and OPT, to use tools. It generates an instruction-following dataset by prompting an advanced teacher with various multi-modal contexts. By using the Low-Rank Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs to solve a range of visual problems, including visual comprehension and image generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to use tools, which is performed in both zero-shot and fine-tuning ways. Extensive experiments demonstrate the effectiveness of our method on various language models, which not only significantly improves the accuracy of invoking seen tools, but also enables the zero-shot capacity for unseen tools. The code and demo are available at https://github.com/StevenGrove/GPT4Tools.
One Size doesn't Fit All: A Personalized Conversational Tutoring Agent for Mathematics Instruction
School of Computer Science, Wuhan University(武汉大学计算机学院);DAMO Academy, Alibaba Group(阿里巴巴达摩院);College of Computer and Cyber Security, Hebei Normal University(河北师范大学计算机与网络空间安全学院)
针对数学教学中学习者差异被忽视的问题,本文提出 PACE 个性化对话辅导智能体,结合学习风格画像、苏格拉底式反馈与个性化训练,提升教学适应性和学习效果。
TeachObs: A Human-Validated Benchmark for Multimodal Teaching Observation and Model Evaluation
Indiana University Bloomington(印第安纳大学布卢明顿分校);Pai Chai University(培才大学);Seoul National University(首尔国立大学);Ewha Womans University(成均馆大学);University of Wolverhampton(沃尔夫汉普顿大学);Korea University Sejong Campus(韩国大学世宗校区)
IKS-Instruct: A 24,000-Example Multilingual Dataset for Teaching Language Models Indian Knowledge Systems
Intrinsic Lab, Centre for Sensors, Instrumentation and Cyber-Physical System Engineering (Centre for SeNSE), Indian Institute of Technology Delhi(印度理工学院德里分校传感器、仪器仪表和网络物理系统工程中心(传感中心)本征实验室);RSL Quantum, FITT, IIT Delhi(印度理工学院德里分校 FITT 学院 RSL 量子实验室)
TasTe: Teaching Large Language Models to Translate through Self-Reflection
Institute of Computing and Intelligence, Harbin Institute of Technology(哈尔滨工业大学计算与智能学院);Pattern Recognition Center, WeChat AI, Tencent Inc(腾讯微信AI模式识别中心)
Cross-Modal Instructions for Robot Motion Generation
College of Connected Computing, Vanderbilt University(连接计算学院,范德比尔特大学);Robotics Institute, Carnegie Mellon University(机器人研究所,卡内基梅隆大学);School of Computer Science, The University of Sydney(计算机科学学院,悉尼大学)