MeetBench-XL: 一种经过校准的多维评估和学习双策略代理用于实时会议
MeetBench-XL: Calibrated Multi-Dimensional Evaluation and Learned Dual-Policy Agents for Real-Time Meetings
AI总结:
MeetBench-XL通过多维评估和学习双策略代理提升实时会议AI助手的性能与效率
摘要:
企业会议环境需要能够处理多样化操作任务的AI助手,从实时讨论中的快速事实核查到跨会议分析以进行战略规划,在严格的时间延迟、成本和隐私约束下。现有会议基准主要集中在简化的问题回答上,未能反映真实世界的企业工作流程,其中查询自然地从多利益相关者协作中产生,跨越长时间的时间背景,并需要工具增强的推理。我们通过一个扎根数据集和一个学习代理框架来解决这一差距。首先,我们介绍了MeetAll,一个双语和多模态语料库,来源于231个企业会议,共计140小时。问题通过一种由企业指导的协议注入,该协议经过领域专家审查和人类可区分性研究验证。与纯合成基准不同,该协议基于四个企业关键维度:认知负荷、时间背景跨度、领域专业知识和可操作任务执行,通过与金融、医疗和科技行业利益相关者的访谈进行校准。第二,我们提出了MeetBench XL,一种与人类判断对齐的多维评估协议,测量事实准确性、意图对齐、响应效率、结构清晰度和完整性。第三,我们提出了MeetMaster XL,一个学习的双策略代理,共同优化查询路由在快速和慢速推理路径之间以及工具调用,包括检索、跨会议聚合和网络搜索。一个轻量级分类器能够通过最小的开销实现准确的路由,实现优于单一模型基线的高质量延迟权衡。与商业系统相比的实验显示了一致的改进,支持了消融测试、鲁棒性测试和一个真实世界部署案例研究。资源:https://github.com/huyuelin/MeetBench。
英文摘要:
Enterprise meeting environments require AI assistants that handle diverse operational tasks, from rapid fact checking during live discussions to cross meeting analysis for strategic planning, under strict latency, cost, and privacy constraints. Existing meeting benchmarks mainly focus on simplified question answering and fail to reflect real world enterprise workflows, where queries arise organically from multi stakeholder collaboration, span long temporal contexts, and require tool augmented reasoning. We address this gap through a grounded dataset and a learned agent framework. First, we introduce MeetAll, a bilingual and multimodal corpus derived from 231 enterprise meetings totaling 140 hours. Questions are injected using an enterprise informed protocol validated by domain expert review and human discriminability studies. Unlike purely synthetic benchmarks, this protocol is grounded in four enterprise critical dimensions: cognitive load, temporal context span, domain expertise, and actionable task execution, calibrated through interviews with stakeholders across finance, healthcare, and technology sectors. Second, we propose MeetBench XL, a multi dimensional evaluation protocol aligned with human judgment that measures factual fidelity, intent alignment, response efficiency, structural clarity, and completeness. Third, we present MeetMaster XL, a learned dual policy agent that jointly optimizes query routing between fast and slow reasoning paths and tool invocation, including retrieval, cross meeting aggregation, and web search. A lightweight classifier enables accurate routing with minimal overhead, achieving a superior quality latency tradeoff over single model baselines. Experiments against commercial systems show consistent gains, supported by ablations, robustness tests, and a real world deployment case study.Resources: https://github.com/huyuelin/MeetBench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。