MEJO：基于跨任务与任务内联合优化的MLLM参与的外科手术三元组识别
MEJO: MLLM-Engaged Surgical Triplet Recognition via Inter- and Intra-Task Joint Optimization
AI总结：
针对外科手术三元组识别中的跨任务与任务内优化冲突，提出MLLM参与联合优化框架MEJO，通过共享-特定解耦方案和协调梯度学习策略，结合MLLM增强语义提示，有效提升识别性能。
摘要：
外科手术三元组识别涉及识别器械、动词、目标及其组合，是一个受长尾数据分布困扰的复杂手术场景理解挑战。受益于跨任务协同促进的主流多任务学习范式在识别三元组方面展现出良好的性能，但仍有两大关键挑战：1）由纠缠任务通用和任务特定表示引起的跨任务优化冲突；2）由类别不平衡训练数据引起的任务内优化冲突。为克服这些困难，我们提出了MLLM参与联合优化（MEJO）框架，为外科手术三元组识别同时赋能跨任务和任务内优化。针对跨任务优化，我们引入了共享-特定解耦（S$^2$D）学习方案，将表示分解为任务共享和任务特定组件。为增强任务共享表示，我们构建了一个由多模态大语言模型（MLLM）驱动的概率提示池，利用专家级语义线索动态增强视觉特征。此外，通过覆盖时间-空间维度的不同任务提示对全面的任务特定线索进行建模，有效缓解了跨任务歧义。为解决任务内优化冲突，我们开发了协调梯度学习（CGL）策略，该策略剖析并重新平衡源自头部和尾部类别的正负梯度，以实现更协调的学习行为。在CholecT45和CholecT50数据集上的大量实验证明了我们提出框架的优越性，验证了其在处理优化冲突方面的有效性。
英文摘要：
Surgical triplet recognition, which involves identifying instrument, verb, target, and their combinations, is a complex surgical scene understanding challenge plagued by long-tailed data distribution. The mainstream multi-task learning paradigm benefiting from cross-task collaborative promotion has shown promising performance in identifying triples, but two key challenges remain: 1) inter-task optimization conflicts caused by entangling task-generic and task-specific representations; 2) intra-task optimization conflicts due to class-imbalanced training data. To overcome these difficulties, we propose the MLLM-Engaged Joint Optimization (MEJO) framework that empowers both inter- and intra-task optimization for surgical triplet recognition. For inter-task optimization, we introduce the Shared-Specific-Disentangled (S$^2$D) learning scheme that decomposes representations into task-shared and task-specific components. To enhance task-shared representations, we construct a Multimodal Large Language Model (MLLM) powered probabilistic prompt pool to dynamically augment visual features with expert-level semantic cues. Additionally, comprehensive task-specific cues are modeled via distinct task prompts covering the temporal-spatial dimensions, effectively mitigating inter-task ambiguities. To tackle intra-task optimization conflicts, we develop a Coordinated Gradient Learning (CGL) strategy, which dissects and rebalances the positive-negative gradients originating from head and tail classes for more coordinated learning behaviors. Extensive experiments on the CholecT45 and CholecT50 datasets demonstrate the superiority of our proposed framework, validating its effectiveness in handling optimization conflicts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
