With the advancement of large language models (LLMs), solving complex reasoning tasks has gained increasing attention. Inference-time computation methods (e.g., Best-of-N, beam search, et al.) are particularly valuable as they can enhance reasoning performance without modifying model parameters or requiring additional training. However, these techniques come with implementation challenges, and most existing methods remain at the proof-of-concept stage with limited practical adoption due to their computational complexity and varying effectiveness across different tasks. In this paper, we investigate and benchmark diverse inference-time computation strategies across reasoning tasks of varying complexity. Since most current methods rely on a proposer-verifier pipeline that first generates candidate solutions (e.g., reasoning solutions) and then selects the best one based on reward signals (e.g., RLHF rewards, process rewards), our research focuses on optimizing both candidate solution generation (e.g., instructing prompts, hyperparameters such as temperature and top-p) and reward mechanisms (e.g., self-evaluation, reward types). Through extensive experiments (more than 20,000 A100-80G GPU hours with over 1,000 experiments) across a variety of models (e.g., Llama, Qwen, and Mistral families) of various sizes, our ablation studies reveal that previously overlooked strategies can significantly enhance performance (e.g., tuning temperature can improve reasoning task performance by up to 5%). Furthermore, we establish a standardized benchmark for inference-time computation by systematically evaluating six representative methods across eight reasoning tasks. These findings provide a stronger foundation for future research. The code is available at https://github.com/usail-hkust/benchmark_inference_time_computation_LLM
Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Guangzhou HKUST Fok Ying Tung Research Institute(广州香港科技大学霍英东研究院);The Hong Kong University of Science and Technology(香港科技大学);AI Thrust(AI 方向)
Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification
Mass General Brigham, Harvard Medical School(麻省总医院布莱根,哈佛医学院);Brigham and Women’s Hospital/Dana-Farber Cancer Institute(布莱根妇女医院/丹娜-法伯癌症研究所);Technische Universität Darmstadt(达姆施塔特工业大学);Maastricht University(马斯特里赫特大学)
Replace-then-Perturb: Targeted Adversarial Attacks With Visual Reasoning for Vision-Language Models
Institute of New Media and Communications, Seoul National University(首尔国立大学新媒体与通信研究所);Pohang University of Science and Technology (POSTECH)(浦项科技大学);Department of Electrical and Computer Engineering, Seoul National University(首尔国立大学电气与计算机工程系)
SignalLLM: A General-Purpose LLM Agent Framework for Automated Signal Processing
Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore(南洋理工大学电子与电气工程学院);MARS Lab, Nanyang Technological University, Singapore(南洋理工大学MARS实验室)
School of Computing, National University of Singapore, Singapore(新加坡国立大学计算机学院);Department of Statistics, University of Oxford, United Kingdom(英国牛津大学统计系);School of Intelligence Science(智能科学学院);Technology, Nanjing University, China(技术学院,南京大学,中国)