用于临床信息丰富且可解释的医学图像理解的工具瓶颈框架
A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding
AI总结：
针对医学图像理解中工具组合难的问题，提出工具瓶颈框架（TBF），利用工具瓶颈模型（TBM）组合VLM选择的工具，通过神经网络计算融合工具输出，在组织病理学和皮肤病学任务中表现出色，提升医学图像理解且使预测更具可解释性。
摘要：
近期由视觉语言模型（VLM）驱动的工具使用框架通过专用工具增强模型预测来改进图像理解。这些框架利用VLM和预先指定的工具箱将预测任务分解为多个工具调用（通常是深度学习模型）以进行预测。常用的工具组合方法是通过VLM生成代码或自然语言中的函数调用使用文本，但在医学图像理解中表现不佳。为此，我们提出了一种用于医学图像理解的工具瓶颈框架（TBF），它使用学习到的工具瓶颈模型（TBM）来组合VLM选择的工具。给定图像和任务时，TBF利用现成的医学VLM从工具箱中选择提取临床相关特征的工具，由TBM进行组合，TBM在输出最终预测前通过神经网络计算并融合工具输出。我们为TBM提出了一种简单有效的策略以使用任意VLM工具选择进行预测。我们在组织病理学和皮肤病学任务上评估TBF，发现其优势使该框架在数据有限的情况下能与基于深度学习的分类器、VLM和现有最先进的工具使用框架表现相当或更优。项目细节和代码可通过给定链接获取。
英文摘要：
Recent tool-use frameworks powered by vision-language models (VLMs) improve image understanding by grounding model predictions with specialized tools. Broadly, these frameworks leverage VLMs and a pre-specified toolbox to decompose the prediction task into multiple tool calls (often deep learning models) which are composed to make a prediction. The dominant approach to composing tools is using text, via function calls embedded in VLM-generated code or natural language. However, these methods often perform poorly on medical image understanding, where salient information is encoded as spatially-localized features that are difficult to compose or fuse via text alone. To address this, we propose a tool-use framework for medical image understanding called the Tool Bottleneck Framework (TBF), which composes VLM-selected tools using a learned Tool Bottleneck Model (TBM). For a given image and task, TBF leverages an off-the-shelf medical VLM to select tools from a toolbox that each extract clinically-relevant features. Instead of text-based composition, these tools are composed by the TBM, which computes and fuses the tool outputs using a neural network before outputting the final prediction. We propose a simple and effective strategy for TBMs to make predictions with any arbitrary VLM tool selection. Overall, our framework not only improves tool-use in medical imaging contexts, but also yields more interpretable, clinically-grounded predictors. We evaluate TBF on tasks in histopathology and dermatology and find that these advantages enable our framework to perform on par with or better than deep learning-based classifiers, VLMs, and state-of-the-art tool-use frameworks, with particular gains in data-limited regimes. The project details and the code are available at https://christinaliu2020.github.io/tbm/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
