Large language models (LLMs) demonstrate impressive performance but lack the flexibility to adapt to human preferences quickly without retraining. In this work, we introduce Test-time Preference Optimization (TPO), a framework that aligns LLM outputs with human preferences during inference, removing the need to update model parameters. Rather than relying on purely numerical rewards, TPO translates reward signals into textual critiques and uses them as textual rewards to iteratively refine its response. Evaluations on benchmarks covering instruction following, preference alignment, safety, and mathematics reveal that TPO progressively improves alignment with human preferences. Notably, after only a few TPO steps, the initially unaligned Llama-3.1-70B-SFT model can surpass the aligned counterpart, Llama-3.1-70B-Instruct. Furthermore, TPO scales efficiently with both the search width and depth during inference. Through case studies, we illustrate how TPO exploits the innate capacity of LLM to interpret and act upon reward signals. Our findings establish TPO as a practical, lightweight alternative for test-time preference optimization, achieving alignment on the fly. Our code is publicly available at https://github.com/yafuly/TPO.
Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generation
University of Science and Technology of China(科学技术大学);National University of Singapore(国立新加坡大学);The Chinese University of Hong Kong(香港中文大学);National Engineering Laboratory for BITA, University of Science and Technology of China(BITA国家工程实验室,科学技术大学)
SoFA: Shielded On-the-fly Alignment via Priority Rule Following
University of Chinese Academy of Sciences(中国科学院大学);Alibaba Group(阿里巴巴集团);State Key Laboratory of Computer Science(计算机科学国家重点实验室);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所)
Align on the Fly: Adapting Chatbot Behavior to Established Norms
Shanghai Jiao Tong University(上海交通大学);The Hong Kong Polytechnic University(香港理工大学);Generative AI Research Lab (GAIR)(生成式人工智能研究实验室(GAIR));Carnegie Mellon University(卡内基梅隆大学);University of Waterloo(滑铁卢大学);The Hong Kong University of Science and Technology(香港科技大学);Google DeepMind(谷歌DeepMind);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
Department of Computer Sciences, University of Wisconsin-Madison(威斯康星大学麦迪逊分校计算机科学系);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机科学与数据科学学院)
SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);School of Computer Science and Electronic Engineering, University of Surrey(Surrey大学计算机科学与电子工程学院)
Text as a Universal Interface for Transferable Personalization
Software College, Northeastern University(东北大学软件学院);Ant Group(蚂蚁集团);Ant International(蚂蚁国际);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学商学院)
Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Xi’an Jiaotong University(西安交通大学);The Chinese University of Hong Kong(香港中文大学);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Huazhong University of Science and Technology(华中科技大学)
Biomedical Visual Instruction Tuning with Clinician Preference Alignment
Stanford University(斯坦福大学);Emory University(埃默里大学);University of California, Berkeley(加州大学伯克利分校);University of Washington(华盛顿大学);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
Personalized Decision Modeling: Utility Optimization or Textualized-Symbolic Reasoning
Department of Civil and Systems Engineering(土木与系统工程系);Johns Hopkins University(约翰霍普金斯大学);Department of Systems and Information Engineering(系统与信息工程系);University of Virginia(弗吉尼亚大学);Johns Hopkins Data Science and AI Institute(约翰霍普金斯大学数据科学与人工智能研究所)
Zero2Text: Zero-Training Cross-Domain Inversion Attacks on Textual Embeddings
School of Computing, Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院计算学院);Graduate School of Information Security, Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院信息安全研究生院);Department of Artificial Intelligence, University of Seoul, Seoul, Republic of Korea(首尔大学人工智能系);Department of Computer Science, University of Seoul, Seoul, Republic of Korea(首尔大学计算机科学系)