在有限预算下对齐LLM：基于启发式奖励模型的推理时对齐
Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models
AI总结：
提出HIA方法，利用启发式奖励模型和两阶段过滤在有限推理预算下实现无需微调的黑盒推理时对齐，在真实数据集上优于现有基线，并支持低预算场景。
摘要：
将LLM与用户偏好对齐对于实际应用至关重要，但通常需要昂贵的微调或推理，迫使在对齐质量和计算成本之间进行权衡。现有的推理时方法通常忽略这种平衡，仅关注优化策略的性能。我们提出HIA（启发式引导的推理时对齐），一种无需微调、兼容黑盒的方法，使用轻量级提示优化器、启发式奖励模型和两阶段过滤来减少推理调用，同时保持对齐质量。在真实世界提示数据集HelpSteer和ComPRed上，在相同推理预算下，HIA在多目标、目标条件任务中优于最佳N采样、束搜索和贪心搜索基线。我们还发现，HIA在低推理预算下有效，仅需一或两次响应查询，为可扩展、个性化的LLM部署提供了实用解决方案。
英文摘要：
Aligning LLMs with user preferences is crucial for real-world use but often requires costly fine-tuning or expensive inference, forcing trade-offs between alignment quality and computational cost. Existing inference-time methods typically ignore this balance, focusing solely on the optimized policy's performance. We propose HIA (Heuristic-Guided Inference-time Alignment), a tuning-free, black-box-compatible approach that uses a lightweight prompt optimizer, heuristic reward models, and two-stage filtering to reduce inference calls while preserving alignment quality. On real-world prompt datasets, HelpSteer and ComPRed, HIA outperforms best-of-N sampling, beam search, and greedy search baselines in multi-objective, goal-conditioned tasks under the same inference budget. We also find that HIA is effective under low-inference budgets with as little as one or two response queries, offering a practical solution for scalable, personalized LLM deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
