The paradigm of scaling Large Language Models (LLMs) in both parameter size and test time has pushed the boundaries of AI capabilities, but at the cost of making the traditional generative evaluation paradigm prohibitively expensive, therefore making the latency of LLM's in-training downstream performance evaluation unbearable. However, simple metrics like training loss (perplexity) are not always correlated with downstream performance, as sometimes their trends diverge from the actual task outcomes. This dilemma calls for a method that is computationally efficient and sufficiently accurate in measuring model capabilities. To address this challenge, we introduce a new in-training evaluation paradigm that uses a lightweight probe for monitoring downstream performance. The probes take the internal representations of LLM checkpoints (during training) as input and directly predict the checkpoint's performance on downstream tasks measured by success probability (i.e., pass@1). We design several probe architectures, validating their effectiveness using the OLMo3-7B's, OLMo3-32B's, and Apertus-8B's checkpoints across a diverse set of downstream tasks. The probes can accurately predict a checkpoint's performance (with avg. AUROC$>$0.75), have decent generalizability across checkpoints (earlier predicts later), and reduce the computation latency from $\sim$1 hr (using conventional generative evaluation method) to $\sim$3 min. In sum, this work presents a practical and scalable in-training downstream evaluation paradigm, enabling a more agile, informed, and efficient LLM development process.
Between the Layers Lies the Truth: Uncertainty Estimation in LLMs Using Intra-Layer Local Information Scores
Faculty of Data and Decision Sciences, Technion --- Israel Institute of Technology(数据与决策科学学院,技术离子研究所);Faculty of Computer Science, Technion --- Israel Institute of Technology(计算机科学学院,技术离子研究所);Faculty of Biomedical Engineering, Technion --- Israel Institute of Technology(生物医学工程学院,技术离子研究所)
Answer Probing-Guided Search for Diverse Solution Exploration of LLMs
University of Science and Technology of China(中国科学技术大学);Zhongguancun Academy(中关村学院);Alibaba Group(阿里巴巴集团);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学)
InA-Probe: Instruction-Aware Active Probing for Time Series Forecasting with LLMs
Nanyang Technological University(南洋理工大学);Khalifa University(哈利法大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Singapore University of Technology and Design(新加坡科技设计大学)
From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs
School of Information Science, Beijing Language and Culture University(北京语言大学信息科学学院);School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院)
School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland(苏黎世联邦理工学院计算机与通信科学系);Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Leiden, The Netherlands(莱顿大学高级计算机科学研究所);College of Management of Technology, EPFL, Lausanne, Switzerland(苏黎世联邦理工学院技术管理学院)
PAL: Probing Audio Encoders via LLMs -- Audio Information Transfer into LLMs
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK(视觉、语音和信号处理中心(CVSSP),英国萨里大学);Surrey Institute for People-Centred AI, University of Surrey, Guildford, GU2 7XH, UK(萨里大学以人为中心的人工智能研究所,英国Guildford);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE(穆罕默德·本·扎耶德人工智能大学(MBZUAI),阿布扎赫德)
Probing the Difficulty Perception Mechanism of Large Language Models
Institute of Digital Twin, EIT(数字孪生研究所,EIT);Wuhan University of Science and Technology(武汉科技大学);Zhejiang University(浙江大学);Hong Kong Polytechnic University(香港理工大学);Shenzhen Institutes of Advanced Technology, CAS(深圳先进技术研究院,中国科学院)
Same Values, Different Languages? From Multilingual Probing to Steering LLMs Toward Chinese Social Values
School of Information Science and Technology, Beijing Foreign Studies University(北京外国语大学信息科学与技术学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Beihang University(北京航空航天大学)
Do Models Hear Like Us? Probing the Representational Alignment of Audio LLMs and Naturalistic EEG
School of Computer Science, Chongqing University(重庆大学计算机科学学院);Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua University(清华大学计算机科学与技术系、人工智能研究院);School of Economics and Business Administration, Chongqing University(重庆大学经济与商业管理学院);School of Artificial Intelligence, Southwest University(西南大学人工智能学院);The First Affiliated Hospital of Chongqing Medical University(重庆医科大学第一附属医院)