University of Virginia(弗吉尼亚大学);EleutherAI;University of Exeter(埃克塞特大学);Harvard Medical School(哈佛医学院);University of Oxford(牛津大学);Thomson-Reuters Foundational Research(汤森路透基础研究部);Mass General Brigham(麻省总医院布里格姆医疗系统);UNC Chapel Hill(北卡罗来纳大学教堂山分校)
Test-time interventions for language models can enhance factual accuracy, mitigate harmful outputs, and improve model efficiency without costly retraining. But despite a flood of new methods, different types of interventions are largely developing independently. In practice, multiple interventions must be applied sequentially to the same model, yet we lack standardized ways to study how interventions interact. We fill this gap by introducing composable interventions, a framework to study the effects of using multiple interventions on the same language models, featuring new metrics and a unified codebase. Using our framework, we conduct extensive experiments and compose popular methods from three emerging intervention categories -- Knowledge Editing, Model Compression, and Machine Unlearning. Our results from 310 different compositions uncover meaningful interactions: compression hinders editing and unlearning, composing interventions hinges on their order of application, and popular general-purpose metrics are inadequate for assessing composability. Taken together, our findings showcase clear gaps in composability, suggesting a need for new multi-objective interventions. All of our code is public: https://github.com/hartvigsen-group/composable-interventions.
MLJ: A Julia package for composable machine learning
University of Auckland(奥克兰大学);Alan Turing Institute(艾伦·图灵研究所);New Zealand eScience Infrastructure(新西兰电子科学基础设施);University College London(伦敦大学学院);Imperial College London(帝国理工学院);University of St Andrews(圣安德鲁斯大学);University of Warwick(华威大学)
MLJ(Machine Learing in Julia)是一个开源软件包,提供了一个通用接口,用于与Julia和其他语言编写的机器学习模型进行交互。它提供了用于选择、调优、评估、组合和比较这些模型的工具和元算法,重点是灵活的模型组合。在这个设计概述中,我们详细介绍了框架的主要新颖之处,以及Julia相对于占主导地位的多语言替代方案的明显优势。
Distributed Sparse Interventions in Language Models
Max Planck School of Cognition(马克斯·普朗克认知学院);Center for Lifespan Psychology, Max Planck Institute for Human Development(马克斯·普朗克人类发展研究所寿命心理学中心);Machine Learning Group, Technische Universität Berlin(柏林工业大学机器学习小组);Berlin Institute for the Foundations of Learning and Data (BIFOLD)(柏林学习与数据基础研究所)
CausalGaze: Unveiling Hallucinations via Counterfactual Graph Intervention in Large Language Models
College of Electronic Engineering, National University of Defense Technology(国防科技大学电子工程学院);Anhui Province Key Laboratory of Cyberspace Security Situation Awareness and Evaluation(安徽省网络空间安全态势感知与评估重点实验室);Institute of Computer Application, China Academy of Engineering Physics(中国工程物理研究院计算机应用研究所)
Can Large Language Models Help Experimental Design for Causal Discovery?
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));MBZUAI(穆罕默德·本·扎耶德人工智能大学);Hong Kong Baptist University(香港浸会大学);The University of Sydney(悉尼大学);Carnegie Mellon University(卡内基梅隆大学)
InfluenceField: A Differentiable Field with Interventionally Identifiable Causal Structure for Multimodal World Modeling
Mathematical Institute, University of Oxford(牛津大学数学研究所);Department of Computer Science, University of Oxford(牛津大学计算机科学系);School of Mathematical Sciences, University of Nottingham Ningbo China(宁波诺丁汉大学数学科学学院)