Mutual information is widely used in artificial intelligence, in a descriptive way, to measure the stochastic dependence of discrete random variables. In order to address questions such as the reliability of the empirical value, one must consider sample-to-population inferential approaches. This paper deals with the distribution of mutual information, as obtained in a Bayesian framework by a second-order Dirichlet prior distribution. The exact analytical expression for the mean and an analytical approximation of the variance are reported. Asymptotic approximations of the distribution are proposed. The results are applied to the problem of selecting features for incremental learning and classification of the naive Bayes classifier. A fast, newly defined method is shown to outperform the traditional approach based on empirical mutual information on a number of real data sets. Finally, a theoretical development is reported that allows one to efficiently extend the above methods to incomplete samples in an easy and effective way.
Feature selection intelligent algorithm with mutual information and steepest ascent strategy
LRIT, Faculty of Sciences, Mohamed V - Agdal University(穆罕默德五世-阿格达勒大学理学院LRIT);LRGE, ENSET, Mohamed V - Souissi University(穆罕默德五世-苏伊西大学ENSET的LRGE)
遥感是一种为数据采掘应用提供知识的高级技术。原则上,高光谱图像是一种遥感工具,可提供精确的区域分类。HSI包含超过一百张地面实况(GT)地图的图像。有些图像携带了相关信息,但另一些图像描述了冗余信息,或者它们受到大气噪声的影响。目的是降低HSI的维数。许多研究使用互信息(MI)或MI的归一化形式来选择适当的频带。在本文中,我们设计了一个算法的基础上,也MI,我们结合MI和最速上升算法,以改进的对称不确定性系数为基础的策略来选择相关波段的分类HSI。该算法是一种特征选择工具和包装策略。我们对HSI AVIRIS 92 AV 3C进行了研究。这是一个控制冗余的人工智能系统;我们必须明确结果算法和人类决策的差异,这可以被视为人类决策可能与智能算法不同的案例研究。高光谱图像、分类、特征选择、互信息、Redundance、最陡上升。人工智能
Sparse and geometry-aware generalisation of the mutual information for joint discriminative clustering and feature selection
Université Côte d’Azur(蔚蓝海岸大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);Maasai Team(马赛团队);CHU de Québec Research Centre(魁北克大学中心医院研究中心);Laval University(拉瓦尔大学)
Not All Knowledge Is Created Equal: Mutual Distillation of Confident Knowledge
Hasso Plattner Institute(哈索·普拉特纳研究所);University of Oxford(牛津大学);Zenith Ai(Zenith人工智能公司);Renmin University of China(中国人民大学);Queen’s University Belfast(贝尔法斯特女王大学)
互知识蒸馏(MKD)通过从另一个模型中提取知识来改进模型。然而,\textit{not all knowledge is certain and correct},特别是在不利条件下。例如,由于不期望的记忆\cite{zhang2017understanding,arpit2017closer},标签噪声通常导致较不可靠的模型。错误的知识会误导学习,而不是帮助学习。这个问题可以通过两个方面来处理:(i)提高知识来自的模型的可靠性(即,知识源的可靠性);(ii)选择可靠的知识进行蒸馏。在文献中,使模型更可靠被广泛研究,而选择性MKD很少受到关注。因此,我们重点研究选择性MKD。具体地说,设计了一个通用的MKD框架:\underline{C}可信知识选择和\underline{M}可信\underline{D}提取(CMD). CMD的关键组成部分是一个通用的知识选择公式,使选择阈值静态(CMD-S)或渐进(CMD-P)。此外,CMD涵盖了两种特殊情况:零知识和所有知识,从而形成了一个统一的MKD框架。大量的实验证明了CMD的有效性,并彻底证明CMD的设计。例如,CMD-P在对标签噪声的鲁棒性方面获得了新的最先进的结果。
MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness
Radboud University Nijmegen(拉德博德大学尼姆维根分校);Vrije Universiteit Amsterdam(自由大学阿姆斯特丹);University of Zagreb Faculty of Electrical Engineering and Computing(Zagreb大学电子工程与计算学院)
Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection
Department of Electrical and Electronics Engineering, Bilkent University, Ankara, Turkey(电气与电子工程系,比尔肯大学,安卡拉,土耳其);School of Computer Science, University of Birmingham, Birmingham, UK(计算机科学学院,伯明翰大学,伯明翰,英国)