RubricsTree: 面向个人健康代理在健康记忆与医疗技能上的可扩展且不断演进的开放式评估
RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skills
AI总结：
提出RubricsTree框架，通过专家对齐的层次化分类法（含100多个原子布尔规则）和上下文自适应路由，实现可扩展、可审计且不断演进的开放式评估，在HealthBench上使模型性能提升高达约66%。
摘要：
基于LLM的个人健康代理利用用户健康（传感器）指标，为缓解全球医疗资源获取不均提供了有希望的途径。然而，大规模临床部署仍受限于开放式评估瓶颈：医生标注可靠但成本高且不可扩展，而LLM作为评判者的评估虽可扩展但主观、不一致，且有时临床对齐不佳。我们引入了RubricsTree，一个可扩展的评估框架，具有专家对齐的层次化分类法，包含超过100个原子级、临床可验证的布尔规则，这些规则通过迭代的人机协同策展协议（由经验丰富的医生领导的专家小组）从4000个真实用户查询的洞察中演化而来。一个上下文感知的自适应路由器每查询仅激活相关的自动加权规则子集，提供可扩展评估所需的吞吐量，同时保持专家对齐的质量。通过系统的元评估，我们展示了RubricsTree：(i) 在具有挑战性的开放式查询上，专家对齐程度显著超过强大的大规模评估基线；(ii) 可靠地惩罚上下文退化的响应；(iii) 当用作结构化指令、文本反馈或性能优化的训练奖励时，在HealthBench上为Gemini、GPT和Qwen模型系列带来高达约66%的相对提升。因此，RubricsTree为产品级个人健康AI的持续优化提供了可扩展、可审计且不断演进的评估基础设施。
英文摘要：
The LLM-empowered personal health agents with user health (sensor) metrics have offered a promising pathway to alleviate global disparities in healthcare access. However, large-scale clinical deployment remains constrained by an open-ended evaluation bottleneck: physician annotation is reliable but costly and unscalable, while LLM-as-a-judge evaluators are scalable but subjective, inconsistent, and sometimes clinically misaligned. We introduce RubricsTree, a scalable evaluation framework with an expert-aligned hierarchical taxonomy of over 100 atomic, clinically-verifiable Boolean rubrics, evolving from the insights of 4,000 real user queries through an iterative human-in-the-loop curation protocol with an expertise panel led by an experienced physician. A context-aware adaptive router activates only the relevant auto-weighted rubric subset per query, providing the throughput needed for scalable evaluation with expert-aligned quality. Through a systematic meta-evaluation, we show that RubricsTree (i) substantially exceeds a strong large-scale evaluation baseline in expert alignment on challenging open-ended queries; (ii) reliably penalizes contextually degraded responses; and (iii) when used as structured instructions, text feedback, or training rewards for performance optimization, yields up to ~66% relative gains on HealthBench for Gemini, GPT, and Qwen model families. RubricsTree thus provides a scalable, auditable, and evolving evaluation infrastructure required for the continuous optimization of product-level personal healthcare AI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
