Faculdade de Computação, Universidade Federal do Pará（帕拉联邦大学计算机学院）；Centro de Informática, Universidade Federal de Pernambuco（伯南布哥联邦大学信息中心）；Instituto Tecnológico Vale（淡水河谷技术研究所）
Lucas F. F. Cardoso, Vitor C. A. Santos, Regiane S. K. Francês, Ricardo B. C. Prudêncio, Ronnie C. O. Alves
备注：
Paper published at the BRACIS 2020 conference, 15 pages, 4 figures
Despite the availability of benchmark machine learning (ML) repositories (e.g., UCI, OpenML), there is no standard evaluation strategy yet capable of pointing out which is the best set of datasets to serve as gold standard to test different ML algorithms. In recent studies, Item Response Theory (IRT) has emerged as a new approach to elucidate what should be a good ML benchmark. This work applied IRT to explore the well-known OpenML-CC18 benchmark to identify how suitable it is on the evaluation of classifiers. Several classifiers ranging from classical to ensembles ones were evaluated using IRT models, which could simultaneously estimate dataset difficulty and classifiers' ability. The Glicko-2 rating system was applied on the top of IRT to summarize the innate ability and aptitude of classifiers. It was observed that not all datasets from OpenML-CC18 are really useful to evaluate classifiers. Most datasets evaluated in this work (84%) contain easy instances in general (e.g., around 10% of difficult instances only). Also, 80% of the instances in half of this benchmark are very discriminating ones, which can be of great use for pairwise algorithm comparison, but not useful to push classifiers abilities. This paper presents this new evaluation methodology based on IRT as well as the tool decodIRT, developed to guide IRT estimation over ML benchmarks.
Decoding Attention from Gaze: A Benchmark Dataset and End-to-End Models
Indian Institute of Technology, Kharagpur（印度理工学院克勒格布尔分校）；Carnegie Mellon University（卡内基梅隆大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance
Indiana University Bloomington（印第安纳大学伯明顿分校）；Center for Complex Networks and Systems（复杂网络与系统中心）
NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery
University of Minnesota（明尼苏达大学）；Alljoined；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Stability AI/Medical AI Research Center (MedARC)（Stability AI/医疗AI研究中心（MedARC））
Do EEG Foundation Models Transfer to Speech? A Benchmark on Overt and Imagined Speech Decoding
University of Granada, Spain（格拉纳达大学）；Research Centre for Information and Communication Technologies (CITIC-UGR), Spain（信息与通信技术研究中心）；University of Chouaib Doukkali, Morocco（舒艾卜·杜卡利大学）；Brain, Mind, and Behavior Research Center (CIMCYC), University of Granada, Spain（大脑、心智与行为研究中心）
Northwestern University（西北大学）；Shirley Ryan Ability Lab（雪莉·瑞安能力实验室）；University of Pennsylvania（宾夕法尼亚大学）；Columbia University（哥伦比亚大学）；Zuckerman Mind Brain Behavior Institute（扎克曼脑与行为研究所）
The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices
Department of Statistics, Ludwig Maximilian University, Munich, Germany（统计系，路德维希-马克西米利安大学，慕尼黑，德国）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））
Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation
Institute of Science Tokyo（东京科学研究所）；National Institute of Information and Communications Technology（信息与通信技术国家研究所）；Nara Institute of Science and Technology（奈良科学技術研究所）
National Laboratory of Pattern Recognition, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Structural Information Learning Machinery: Learning from Observing, Associating, Optimizing, Decoding, and Abstracting
State Key Laboratory of Software Development Environment（软件开发环境国家重点实验室）；School of Computer Science, Beihang University（北京航空航天大学计算机学院）
In the present paper, we propose the model of { \it structural information learning machines} (SiLeM for short), leading to a mathematical definition of learning by merging the theories of computation and information. Our model shows that the essence of learning is { \it to gain information}, that to gain information is { \it to eliminate uncertainty} embedded in a data space, and that to eliminate uncertainty of a data space can be reduced to an optimization problem, that is, an { \it information optimization problem}, which can be realized by a general { \it encoding tree method}. The princi…
Stable Neural Decoding Across Sessions via Task-Conditioned Latent Alignment for Brain-Machine Interfaces
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Pittsburgh（匹兹堡大学）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Iterative Encoding-Decoding VAEs Anomaly Detection in NOAA's DART Time Series: A Machine Learning Approach for Enhancing Data Integrity for NASA's GRACE-FO Verification and Validation
University of California at Los Angeles（加利福尼亚大学洛杉矶分校）；NASA Jet Propulsion Laboratory（美国国家航空航天局喷气推进实验室）
Decoding complexity: how machine learning is redefining scientific discovery
Alice Author Affiliation（Alice作者机构）；Tom Theory Affiliation（Tom理论机构）；Emma Experiments Affiliation（Emma实验机构）；Affiliation, Department, City, Country（机构、部门、城市、国家）；Simon Simulations Affiliation（Simon模拟机构）
Application of an attention-based CNN-BiLSTM framework for in vivo two-photon calcium imaging of neuronal ensembles: decoding complex bilateral forelimb movements from unilateral M1
West Virginia University（西弗吉尼亚大学）；University of Miami（迈阿密大学）；Leonard M. Miller School of Medicine（伦纳德·M·米勒医学院）；Rockefeller Neuroscience Institute（洛克菲勒神经科学研究所）
Enhancing Motor Imagery Decoding in Brain Computer Interfaces using Riemann Tangent Space Mapping and Cross Frequency Coupling
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications（北京邮电大学信息与通信工程学院）；University of Sheffield（谢菲尔德大学）；School of Automation, Beijing University of Posts and Telecommunications（北京邮电大学自动化学院）
