We consider the problem of aligning a large language model (LLM) to model the preferences of a human population. Modeling the beliefs, preferences, and behaviors of a specific population can be useful for a variety of different applications, such as conducting simulated focus groups for new products, conducting virtual surveys, and testing behavioral interventions, especially for interventions that are expensive, impractical, or unethical. Existing work has had mixed success using LLMs to accurately model human behavior in different contexts. We benchmark and evaluate two well-known fine-tuning approaches and evaluate the resulting populations on their ability to match the preferences of real human respondents on a survey of preferences for battery electric vehicles (BEVs). We evaluate our models against their ability to match population-wide statistics as well as their ability to match individual responses, and we investigate the role of temperature in controlling the trade-offs between these two. Additionally, we propose and evaluate a novel loss term to improve model performance on responses that require a numeric response.
APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Training
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China(计算与智能学院,哈尔滨工业大学,深圳,中国);Huawei Cloud Computing Technologies Co., Ltd.(华为云计算技术有限公司);School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen, China(智能科学与工程学院,哈尔滨工业大学,深圳,中国)
University of Cambridge(剑桥大学);Utrecht University(乌得勒支大学);University of York(约克大学);University College London, University of London(伦敦大学学院(伦敦大学));TKH AI Technology(TKH人工智能技术公司);University of Pennsylvania(宾夕法尼亚大学)