Machine learning (ML) practitioners are increasingly tasked with developing models that are aligned with non-technical experts' values and goals. However, there has been insufficient consideration on how practitioners should translate domain expertise into ML updates. In this paper, we consider how to capture interactions between practitioners and experts systematically. We devise a taxonomy to match expert feedback types with practitioner updates. A practitioner may receive feedback from an expert at the observation- or domain-level, and convert this feedback into updates to the dataset, loss function, or parameter space. We review existing work from ML and human-computer interaction to describe this feedback-update taxonomy, and highlight the insufficient consideration given to incorporating feedback from non-technical experts. We end with a set of open questions that naturally arise from our proposed taxonomy and subsequent survey.
Capturing and incorporating expert knowledge into machine learning models for quality prediction in manufacturing
Fraunhofer Institute for Machine Tools and Forming Technology (IWU)(弗劳恩霍夫机床与成型技术研究所(IWU));Fraunhofer Institute for Industrial Mathematics (ITWM)(弗劳恩霍夫工业数学研究所(ITWM));Institute of Mechatronic Engineering Technische Universität Dresden(德累斯顿工业大学机电工程研究所)
KINN: Incorporating Expert Knowledge in Neural Networks
German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);TU Kaiserslautern(凯泽斯劳滕工业大学);National University of Sciences and Technology (NUST)(国立科学技术大学);National Center of Artificial Intelligence(国家人工智能中心)
Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts
School of Computer Science, University of Bristol, UK.(计算机科学学院,布里斯托大学);School of Physics, University of Bristol, UK.(物理学院,布里斯托大学);School of Engineering Mathematics and Technology, University of Bristol, UK.(工程数学与技术学院,布里斯托大学)
On Generalization and Distributional Update for Mimicking Observations with Adequate Exploration
Shanghai Institute of Satellite Engineering(上海卫星工程研究所);Department of Mathematics, College of Sciences, Shanghai University(上海大学数学系);School of Computer Science, University of Bristol(布里斯托大学计算机科学学院);Newtouch Center for Mathematics of Shanghai University(上海大学数学中心)
AQuA -- Combining Experts' and Non-Experts' Views To Assess Deliberation Quality in Online Discussions Using LLMs
Heinrich Heine University Düsseldorf(杜塞尔多夫海因里希·海涅大学);Technical University Ilmenau(伊尔默瑙工业大学);Johannes Gutenberg University Mainz(美因茨约翰内斯·古腾堡大学);Technical University Dortmund(多特蒙德工业大学)
Improving drug sensitivity predictions in precision medicine through active expert knowledge elicitation
Helsinki Institute for Information Technology HIIT(赫尔辛基信息技术研究所(HIIT));Aalto University(阿尔托大学);Institute for Molecular Medicine Finland (FIMM)(芬兰分子医学研究所(FIMM));University of Helsinki(赫尔辛基大学)