Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Tasks
Center for Data Science, New York University, New York City, U.S.A.(数据科学中心,纽约大学,纽约市,美国)
Angelica Chen, Samuel D. Stanton, Frances Ding, Robert G. Alberstein, Andrew M. Watkins, Richard Bonneau, Vladimir Gligorijević, Kyunghyun Cho, Nathan C. Frey
备注:
Supercedes arXiv:2407.00236v1. arXiv admin note: text overlap with arXiv:2407.00236
Although large language models (LLMs) have shown promise in biomolecule optimization problems, they incur heavy computational costs and struggle to satisfy precise constraints. On the other hand, specialized solvers like LaMBO-2 offer efficiency and fine-grained control but require more domain expertise. Comparing these approaches is challenging due to expensive laboratory validation and inadequate synthetic benchmarks. We address this by introducing Ehrlich functions, a synthetic test suite that captures the geometric structure of biophysical sequence optimization problems. With prompting alone, off-the-shelf LLMs struggle to optimize Ehrlich functions. In response, we propose LLOME (Language Model Optimization with Margin Expectation), a bilevel optimization routine for online black-box optimization. When combined with a novel preference learning loss, we find LLOME can not only learn to solve some Ehrlich functions, but can even perform as well as or better than LaMBO-2 on moderately difficult Ehrlich variants. However, LLMs also exhibit some likelihood-reward miscalibration and struggle without explicit rewards. Our results indicate LLMs can occasionally provide significant benefits, but specialized solvers are still competitive and incur less overhead.
Benchmarking Generalist and Specialist Time-Series Foundation Models for PPG-Based Health Assessment
Nell Hodgson Woodruff School of Nursing(Nell Hodgson Woodruff护理学院);School of Computer Science(计算机科学学院);Department of Pediatrics(儿科系);Department of Computer Science(计算机科学系);Department of Epidemiology(流行病学系);Department of Anesthesiology(麻醉学系);Department of Surgery(外科系);Department of Medicine(医学系);School of Medicine(医学院)
SHM-Agents: A Generalist-Specialist Integrated Agent System for Structural Health Monitoring
Key Lab of Smart Prevention and Mitigation of Civil Engineering Disasters of the Ministry of Industry and Information Technology, Harbin Institute of Technology, Harbin(工业和信息化部智能防灾减灾重点实验室,哈尔滨工业大学,哈尔滨);Key Lab of Structures Dynamic Behavior and Control of the Ministry of Education, Harbin Institute of Technology, Harbin(教育部结构动态行为与控制重点实验室,哈尔滨工业大学,哈尔滨);School of Civil Engineering, Harbin Institute of Technology, Harbin(哈尔滨工业大学土木工程学院)