MM-Forecast：一种基于大语言模型的多模态时间事件预测方法
MM-Forecast: A Multimodal Approach to Temporal Event Forecasting with Large Language Models
AI总结：
本文提出MM-Forecast框架，通过识别图像在时间事件预测中的突出与互补功能并转化为文字描述，将其融入大语言模型预测流程，并构建多模态数据集MidEast-TE-mm验证了该方法能显著提升预测性能。
摘要：
我们研究了一个新兴且引人入胜的问题：利用大语言模型进行多模态时间事件预测。与使用文本或图模态相比，利用图像进行时间事件预测的研究尚未被充分探索，尤其是在大语言模型（LLMs）时代。为了弥补这一空白，我们特别关注两个关键问题：1）为什么图像有助于时间事件预测，以及2）如何将图像集成到基于LLM的预测框架中。为了回答这些研究问题，我们提出识别图像在时间事件预测场景中扮演的两种基本功能，即突出（highlighting）和互补（complementary）。然后，我们开发了一个名为MM-Forecast的新框架。该框架采用图像功能识别模块，利用多模态大语言模型（MLLMs）将这些功能识别为文字描述，并随后将这些功能描述纳入基于LLM的预测模型中。为了评估我们的方法，我们通过扩展现有事件数据集MidEast-TE-mini并添加图像，构建了一个新的多模态数据集MidEast-TE-mm。实证研究表明，我们的MM-Forecast能够正确识别图像功能，并且进一步地，纳入这些文字功能描述显著提升了预测性能。数据集、代码和提示可在https://github.com/LuminosityX/MM-Forecast获取。
英文摘要：
We study an emerging and intriguing problem of multimodal temporal event forecasting with large language models. Compared to using text or graph modalities, the investigation of utilizing images for temporal event forecasting has not been fully explored, especially in the era of large language models (LLMs). To bridge this gap, we are particularly interested in two key questions of: 1) why images will help in temporal event forecasting, and 2) how to integrate images into the LLM-based forecasting framework. To answer these research questions, we propose to identify two essential functions that images play in the scenario of temporal event forecasting, i.e., highlighting and complementary. Then, we develop a novel framework, named MM-Forecast. It employs an Image Function Identification module to recognize these functions as verbal descriptions using multimodal large language models (MLLMs), and subsequently incorporates these function descriptions into LLM-based forecasting models. To evaluate our approach, we construct a new multimodal dataset, MidEast-TE-mm, by extending an existing event dataset MidEast-TE-mini with images. Empirical studies demonstrate that our MM-Forecast can correctly identify the image functions, and further more, incorporating these verbal function descriptions significantly improves the forecasting performance. The dataset, code, and prompts are available at https://github.com/LuminosityX/MM-Forecast.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
