Chat-TS: Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data
Electrical and Computer Engineering, Queen’s University(皇后大学电气与计算机工程学院);Mechanical and Materials Engineering, Queen’s University(皇后大学机械与材料工程学院);Ingenuity Labs Research Institute, Queen’s University(皇后大学创新实验室研究 institute)
Large language models are being rapidly deployed across many fields such as healthcare, finance, transportation, and energy, where time-series data are fundamental components. The current works are still limited in their ability to perform reasoning that involves both time-series and the corresponding textual content. We address this gap by introducing Chat-TS, a large language model (LLM) based framework designed to support reasoning over time series and textual data. Unlike traditional models, Chat-TS integrates time-series tokens into LLMs' vocabulary, enhancing its reasoning ability over both modalities without compromising core natural language capabilities. To support learning and evaluation, we contribute new datasets: the TS Instruct Training Dataset (pairing diverse time-series data with relevant text instructions and responses for instruction tuning), the TS Instruct Question and Answer (QA) Gold Dataset (multiple-choice questions to evaluate multimodal reasoning), and a TS Instruct Quantitative Probing Set (a small subset of TS Instruct QA reasoning tasks alongside math and decision-making questions for LLM evaluation). We design a training strategy to preserve the inherent reasoning capabilities of LLMs while augmenting them for time-series reasoning. Experiments show that Chat-TS achieves state-of-the-art performance in multimodal reasoning tasks by maintaining strong natural language proficiency while improving time-series reasoning.
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Bytedance Inc.(字节跳动公司)
ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Model
State Key Laboratory of Oral Diseases & National Clinical Research Center for Oral Diseases & Department of Oral and Maxillofacial Surgery, West China Hospital of Stomatology, Sichuan University(四川大学华西口腔医院口腔疾病研究国家重点实验室、国家口腔疾病临床医学研究中心、口腔颌面外科);Department of Civil, Environmental & Construction Engineering, University of Central Florida(中佛罗里达大学土木、环境与建筑工程系);College of Transportation Engineering, University of Central Florida(中佛罗里达大学交通工程学院);School of Transportation and Logistics, Southwest Jiaotong University(西南交通大学交通运输与物流学院);C2SMART Center, Tandon School of Engineering, New York University(纽约大学坦登工程学院C2SMART中心);State Key Laboratory of Oral Diseases & National Clinical Research Center for Oral Diseases & Eastern Clinic, West China Hospital of Stomatology, Sichuan University(四川大学华西口腔医院口腔疾病研究国家重点实验室、国家口腔疾病临床医学研究中心、东院区)
Enhancing Depression-Diagnosis-Oriented Chat with Psychological State Tracking
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);Shanghai Institute of Al for Education, East China Normal University(东华大学上海人工智能教育研究所);School of Psychology and Cognitive Science, East China Normal University(东华大学心理学与认知科学学院)
ChatSR: Multimodal Large Language Models for Scientific Formula Discovery
AnnLab, Institute of Semiconductors, Chinese Academy of Sciences, Beijing, China(安 lab,半导体研究所,中国科学院,北京,中国);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing, China(电子、电气与通信工程学院,中国科学院大学,北京,中国);Zhongguancun Academy, Beijing, China(中关村学院,北京,中国);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 101408, China(先进交叉科学学院,中国科学院大学,北京101408,中国);College of Materials Science and Opto-Electronic Technology, University of Chinese Academy of Sciences, Beijing, 100049, China(材料科学与光电技术学院,中国科学院大学,北京100049,中国);School of Integrated Circuits, University of Chinese Academy of Sciences, Beijing 100049, China(集成电路学院,中国科学院大学,北京100049,中国)
EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training
Institute of Computer Science, University of Tartu(塔尔图大学计算机科学研究院);Department of Software Science, Tallinn University of Technology(塔林技术大学软件科学系);Institute of the Estonian Language, Tallinn, Estonia(爱沙尼亚语言研究院);School of Humanities, Tallinn University(塔林大学人文学院)
Enhancing User-Oriented Proactivity in Open-Domain Dialogues with Critic Guidance
South China University of Technology(华南理工大学);Peng Cheng Laboratory(鹏城实验室);Pazhou Laboratory(琶洲实验室);Tencent AI Lab(腾讯AI实验室);Tencent Robotics X Lab(腾讯机器人X实验室);Hong Kong Polytechnic University(香港理工大学);Hunan University(湖南大学)
Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Wuhan AI Research(武汉人工智能研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
HIPPO: Enhancing the Table Understanding Capability of LLMs through Hybrid-Modal Preference Optimization
School of Computer Science and Engineering, Northeastern University, Shenyang, China(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Tsinghua University, Beijing, China(清华大学计算机科学与技术系);Huawei Technologies Co., Ltd(华为技术有限公司)
ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrieval
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Université de Montréal(蒙特利尔大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Waseda University(早稻田大学)
Testing the Depth of ChatGPT's Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3.5's Abilities in Regard to Recognizing and Generating ASCII-Art Are Not Totally Lacking
Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration
Department of Computer Science, University of Kaiserslautern-Landau (RPTU)(科斯拉尔特伦大学计算机科学系);SDS, German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));INRAE, UMR TETIS, University of Montpellier(蒙彼利埃大学UMR TETIS);CIRAD, UMR TETIS, University of Montpellier(蒙彼利埃大学UMR TETIS);INRIA, EVERGREEN, University of Montpellier(蒙彼利埃大学)