Large language model (LLM) agents have recently demonstrated impressive capabilities in various domains like open-ended conversation and multi-step decision-making. However, it remains challenging for these agents to solve strategic language games, such as Werewolf, which demand both strategic decision-making and free-form language interactions. Existing LLM agents often suffer from intrinsic bias in their action distributions and limited exploration of the unbounded text action space, resulting in suboptimal performance. To address these challenges, we propose Latent Space Policy Optimization (LSPO), an iterative framework that combines game-theoretic methods with LLM fine-tuning to build strategic language agents. LSPO leverages the observation that while the language space is combinatorially large, the underlying strategy space is relatively compact. We first map free-form utterances into a finite latent strategy space, yielding an abstracted extensive-form game. Then we apply game-theoretic methods like Counterfactual Regret Minimization (CFR) to optimize the policy in the latent space. Finally, we fine-tune the LLM via Direct Preference Optimization (DPO) to align with the learned policy. By iteratively alternating between these steps, our LSPO agents progressively enhance both strategic reasoning and language communication. Experiment on the Werewolf game shows that our agents iteratively expand the strategy space with improving performance and outperform existing Werewolf agents, underscoring their effectiveness in free-form language games with strategic interactions.
Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolf
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Cooperative AI Lab, Department of Informatics, King’s College London(伦敦国王学院合作人工智能实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);University of Liverpool(利物浦大学);Nanjing Artificial Intelligence Research of IA(中国科学院南京人工智能研究院);AI Centre, Department of Computer Science, UCL(伦敦大学学院人工智能中心)
DVM: Towards Controllable LLM Agents in Social Deduction Games
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Singapore Management University(新加坡管理大学);University of Science and Technology of China(中国科学技术大学)
STRIDE: Strategic Iterative Decision-Making for Retrieval-Augmented Multi-Hop Question Answering
University of Science;Technology of China \& State Key Laboratory of Cognitive Intelligence Hefei Anhui China;Technology of China \& State Key Laboratory of Cognitive Intelligence
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Salesforce AI Research(Salesforce AI研究院);The Hong Kong University of Science and Technology(香港科技大学)
EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle
Zhejiang University(浙江大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);East China Normal University(华东师范大学);Fudan University(复旦大学);Central South University(中南大学);Shanghai Innovation Institute(上海创新研究院);Shanghai Jiao Tong University(上海交通大学);University of Science and Technology of China(中国科学技术大学)
Supply Chain Optimization via Generative Simulation and Iterative Decision Policies
School of Computing and Augmented Intelligence(计算与增强智能学院);Arizona State University(亚利桑那州立大学);NEC Labs America(NEC美国实验室);Data Science and System Security(数据科学与系统安全)
CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulation
University of Copenhagen(哥本哈根大学);IIIT Ranchi(印度信息技术学院兰契分校);ISI Kolkata(印度统计学院加尔各答分校);NIT Andhra Pradesh(印度国家理工学院安得拉邦分校);IGDTUW(英迪拉·甘地德里技术女子大学);IIT Kharagpur(印度理工学院卡哈拉格普尔分校);Google DeepMind(谷歌DeepMind);Google(谷歌);AI Institute, University of South Carolina(南卡罗来纳大学人工智能研究所)