Test-time scaling enhances large language model performance by allocating additional compute resources during inference. Best-of-N (BoN) sampling serves as a common sampling-based scaling technique, broadening the search space in parallel to find better solutions from the model distribution. However, its cost-performance trade-off is still underexplored. Two main challenges limit the efficiency of BoN sampling: (1) Generating N full samples consumes substantial GPU memory, reducing inference capacity under limited resources. (2) Reward models add extra memory and latency overhead, and training strong reward models introduces potential training data costs. Although some studies have explored efficiency improvements, none have addressed both challenges at once. To address this gap, we propose Self-Truncation Best-of-N (ST-BoN), a decoding method that avoids fully generating all N samples and eliminates the need for reward models. It leverages early sampling consistency in the model's internal states to identify the most promising path and truncate suboptimal ones. In terms of cost, ST-BoN reduces dynamic GPU memory usage by over 80% and inference latency by 50%. In terms of cost-performance trade-off, ST-BoN achieves the same performance as Full-BoN while saving computational cost by 70%-80%, and under the same cost, it can improve accuracy by 3-4 points.
$ϕ$-Decoding: Adaptive Foresight Sampling for Balanced Inference-Time Exploration and Exploitation
Shanghai AI Lab(上海人工智能实验室);Xi’an Jiaotong University(西安交通大学);The University of Hong Kong(香港大学);Peking University(北京大学);National University of Singapore(新加坡国立大学)
Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta(阿尔伯塔大学计算机科学系与阿尔伯塔机器智能研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能讲席)
Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs
Australian National University(澳大利亚国立大学);Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学);Amazon(亚马逊);University of Technology Sydney(悉尼科技大学)
How Sampling Affects the Detectability of Machine-written texts: A Comprehensive Study
Sorbonne Université, CNRS, ISIR, Paris France(索邦大学、国家科学研究中心、ISIR、巴黎法国);CNRS, International Laboratory on Learning Systems, Montréal, Canada(国家科学研究中心、学习系统国际实验室、蒙特利尔加拿大);Mila - Québec AI Institute, Montréal, Canada(魁北克AI研究所、蒙特利尔加拿大);CentraleSupélec, Université Paris-Saclay, Gif-sur-Yvette, France(CentraleSupélec、巴黎萨克雷大学、吉夫-sur-伊夫特法国)
Impact of Decoding Methods on Human Alignment of Conversational LLMs
Birla Institute of Technology and Science, Pilani(比拉理工与科学学院(皮拉尼));National University of Singapore(新加坡国立大学);NUS Center for Trusted Internet and Community(新加坡国立大学可信互联网与社区中心);Department of Communications and New Media, National University of Singapore(新加坡国立大学传播与新媒体系)
本文研究了Beam Search、Top K Sampling和Nucleus Sampling等解码方法对LLM与人类对话对齐的影响,提出了内容、风格和心理测量导向的新衡量标准,发现较少beam数和较低P值对齐效果更好,且任务导向与开放式数据集表现不同。
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络与信息安全学院)
Robust Multi-Objective Controlled Decoding of Large Language Models
University College London(伦敦大学学院);Ulsan National Institute of Science and Technology(釜山国立科学技术研究所);University of Basel(巴塞尔大学);University College London AI Centre(伦敦大学学院人工智能中心)
RheoSampling: Resolving the One-Hot Dilemma in Stochastic Dynamic-Tree Speculative Decoding
National Center for Mathematics and Interdisciplinary Sciences (NCMIS), AMSS, CAS(中国科学院数学与系统科学研究院国家数学与交叉科学中心);The University of Tokyo(东京大学);SKLMS and AMSS, Chinese Academy of Sciences(中国科学院数学与系统科学研究院,科学与工程计算国家重点实验室);School of Mathematical Sciences, University of Chinese Academy of Sciences(中国科学院大学数学科学学院);Lenovo AI Technology Center(联想AI技术中心)