利用大型语言模型将任务相关上下文编译进贝叶斯优化
Harnessing Large Language Models to Compile Task-Relevant Context into Bayesian Optimisation
浏览论文内容
中文总结 AI 辅助
本研究提出HarBO框架，利用大型语言模型将任务相关上下文编译为可执行代码整合进贝叶斯优化，理论分析遗憾界，实验证明在合成与真实基准上性能可比专门方法，但通用框架在陌生领域效果有限。
中文摘要 AI 辅助
将丰富的任务相关上下文（如领域知识和外部观察）整合进来是一项关键能力，但对贝叶斯优化（BO）而言仍具挑战性。近期，实践者开始通过编码框架使用大型语言模型（LLMs）来生成并执行BO程序。在这种新兴实践中，后验信念不仅由贝叶斯推断塑造，还受LLM生成的模型和数据工件影响，为任务上下文以可执行代码形式进入BO提供了灵活途径。为研究LLM能否以及如何被利用来编译多样化的上下文信号以用于BO，我们将LLM编译的BO形式化为广义上下文决策制定。我们提出HarBO，一个专为BO设计的框架，通过经过验证的多阶段工作流将广义上下文编译进标准BO的核心工件中。我们的理论分析了不完美编译下的遗憾以及添加新上下文的影响。在合成函数和真实世界基准测试中，我们发现LLM框架能有效地将上下文编译进标准BO，达到与基于LLM嵌入的专门方法和直接LLM在环的BO方法相当的性能。通用编码框架在熟悉的领域（如超参数优化）中有效，但在不熟悉的、上下文丰富的领域中表现不足。综合这些结果，LLM框架被确立为一条有前景但并非自动可靠的途径，用于使丰富的任务上下文在BO中可用。
英文摘要
Incorporating rich task-relevant context, such as domain knowledge and external observations, is a key capability yet remains challenging for Bayesian optimisation (BO). Recently, practitioners have started to use large language models (LLMs) to generate and execute BO programs through coding harnesses. In such emerging practices, the posterior belief is shaped not only by Bayesian inference but also by LLM-generated model and data artefacts, offering a flexible route for task context to enter BO as executable code. To study whether and how LLMs can be harnessed to compile diverse contextual signals for BO, we formulate LLM-compiled BO as generalised-context decision making. We propose HarBO, a BO-specialised harness that compiles generalised context into the core artefacts of standard BO through a validated multi-stage workflow. Our theory analyses the regret under imperfect compilation and the effect of adding new context. Across synthetic functions and real-world benchmarks, we find that LLM harnesses can effectively compile context into standard BO, achieving competitive performance with specialised LLM-embedding-based and direct LLM-in-the-loop BO methods. General coding harnesses can be effective in familiar domains such as hyperparameter optimisation, but fall short in unfamiliar, context-rich domains. Together, these results establish LLM harnesses as a promising, but not automatically reliable, route for making rich task context usable in BO.
发表机构
- The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
- University College London (UCL)（伦敦大学学院）
- Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
机构由 AI 辅助整理，请以论文原文为准。
