Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models
University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Nanjing University（南京大学）
Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, Tieniu Tan
With the emergence of pretrained vision-language models (VLMs), considerable efforts have been devoted to fine-tuning them for downstream tasks. Despite the progress made in designing efficient fine-tuning methods, such methods require access to the model's parameters, which can be challenging as model owners often opt to provide their models as a black box to safeguard model ownership. This paper proposes a \textbf{C}ollabo\textbf{ra}tive \textbf{F}ine-\textbf{T}uning (\textbf{CraFT}) approach for fine-tuning black-box VLMs to downstream tasks, where one only has access to the input prompts and the output predictions of the model. CraFT comprises two modules, a prompt generation module for learning text prompts and a prediction refinement module for enhancing output predictions in residual style. Additionally, we introduce an auxiliary prediction-consistent loss to promote consistent optimization across these modules. These modules are optimized by a novel collaborative training algorithm. Extensive experiments on few-shot classification over 15 datasets demonstrate the superiority of CraFT. The results show that CraFT achieves a decent gain of about 12\% with 16-shot datasets and only 8,000 queries. Moreover, CraFT trains faster and uses only about 1/80 of the memory footprint for deployment, while sacrificing only 1.62\% compared to the white-box method. Our code is publicly available at https://github.com/mrflogs/CraFT .
University of Southern California（南加州大学）；Information Sciences Institute, University of Southern California（南加州大学信息科学研究所）；University of California, Los Angeles（加州大学洛杉矶分校）
Commonsense question answering (QA) requires background knowledge which is not explicitly stated in a given context. Prior works use commonsense knowledge graphs (KGs) to obtain this knowledge for reasoning. However, relying entirely on these KGs may not suffice, considering their limited coverage and the contextual dependence of their knowledge. In this paper, we augment a general commonsense QA framework with a knowledgeable path generator. By extrapolating over existing paths in a KG with a state-of-the-art language model, our generator learns to connect a pair of entities in text with a dy…
University of California, Riverside（加州大学河滨分校）；US Army CCDC Army Research Lab（美国陆军作战能力发展司令部陆军研究实验室）
There has been a recent surge in research on adversarial perturbations that defeat Deep Neural Networks (DNNs) in machine vision; most of these perturbation-based attacks target object classifiers. Inspired by the observation that humans are able to recognize objects that appear out of place in a scene or along with other unlikely objects, we augment the DNN with a system that learns context consistency rules during training and checks for the violations of the same during testing. Our approach builds a set of auto-encoders, one for each object class, appropriately trained so as to output a di…
Bridging Collaborative Filtering and Large Language Models with Dynamic Alignment, Multimodal Fusion and Evidence-grounded Explanations
Department of Software \& Microelectronics, Peking University, Beijing, China；Economic Law School, China University of Political Science；Financial Media, Peking University, ChangSha, China
Sequence prediction models can be learned from example sequences with a variety of training algorithms. Maximum likelihood learning is simple and efficient, yet can suffer from compounding error at test time. Reinforcement learning such as policy gradient addresses the issue but can have prohibitively poor exploration efficiency. A rich set of other algorithms such as RAML, SPG, and data noising, have also been developed from different perspectives. This paper establishes a formal connection between these algorithms. We present a generalized entropy regularized policy optimization formulation,…
Hong Kong University of Science and Technology（香港科技大学）；Massachusetts Institute of Technology（麻省理工学院）；Adobe Research（Adobe研究院）；Children’s Hospital of Zhejiang University School of Medicine（浙江大学医学院附属儿童医院）
Machine learning (ML) is increasingly applied to Electronic Health Records (EHRs) to solve clinical prediction tasks. Although many ML models perform promisingly, issues with model transparency and interpretability limit their adoption in clinical practice. Directly using existing explainable ML techniques in clinical settings can be challenging. Through literature surveys and collaborations with six clinicians with an average of 17 years of clinical experience, we identified three key challenges, including clinicians' unfamiliarity with ML features, lack of contextual information, and the nee…
University of Technology Sydney（悉尼科技大学）；Monash University（莫纳什大学）
Modeling multivariate time series has long been a subject that has attracted researchers from a diverse range of fields including economics, finance, and traffic. A basic assumption behind multivariate time series forecasting is that its variables depend on one another but, upon looking closely, it is fair to say that existing methods fail to fully exploit latent spatial dependencies between pairs of variables. In recent years, meanwhile, graph neural networks (GNNs) have shown high capability in handling relational dependencies. GNNs require well-defined graph structures for information propa…
Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulation
University of Granada（格拉纳达大学）；DaSCI Andalusian Institute in Data Science and Computational Intelligence（安达卢西亚数据科学与计算智能研究所（DaSCI））；TECNALIA（泰克纳利亚研究中心）；Basque Research and Technology Alliance (BRTA)（巴斯克研究与技术联盟（BRTA））；University of the Basque Country (UPV/EHU)（巴斯克大学（UPV/EHU））；University of Vienna（维也纳大学）；Cornell University（康奈尔大学）；ADIA Lab（阿布扎比投资局实验室）；Khalifa University of Science and Technology（哈利法科技大学）
Codiscovering graphical structure and functional relationships within data: A Gaussian Process framework for connecting the dots
Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA 91125, USA（计算与数学科学系，加州理工学院，帕萨迪纳，CA 91125，美国）；Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA 91109, USA（喷气推进实验室，加州理工学院，帕萨迪纳，CA 91109，美国）
Connecting the Dots: A Machine Learning Ready Dataset for Ionospheric Forecasting Models
Department of Information and Computer Science（信息与计算机科学系）；University of Hawai‘i at Mānoa（夏威夷大学毛纳罗亚分校）；Department of Engineering Science（工程科学系）；University of Oxford（牛津大学）；Università degli Studi di Roma Sapienza（罗马大学）；Free Flight Research Lab（自由飞行研究实验室）；University of New Hampshire（新罕布什尔大学）；European Space Agency (ESA)（欧洲航天局）；NASA Jet Propulsion Laboratory（美国宇航局喷气推进实验室）；NASA Headquarters（美国宇航局总部）；Space Weather Technology, Research, and Education Center（空间天气技术、研究与教育中心）；University of Colorado Boulder（科罗拉多大学博尔德分校）；Department of Computer Science（计算机科学系）；University of York & King’s College London（约克大学及伦敦国王学院）
