Temporal Sentence Grounding (TSG), which aims to localize moments from videos based on the given natural language queries, has attracted widespread attention. Existing works are mainly designed for short videos, failing to handle TSG in long videos, which poses two challenges: i) complicated contexts in long videos require temporal reasoning over longer moment sequences, and ii) multiple modalities including textual speech with rich information require special designs for content understanding in long videos. To tackle these challenges, in this work we propose a Grounding-Prompter method, which is capable of conducting TSG in long videos through prompting LLM with multimodal information. In detail, we first transform the TSG task and its multimodal inputs including speech and visual, into compressed task textualization. Furthermore, to enhance temporal reasoning under complicated contexts, a Boundary-Perceptive Prompting strategy is proposed, which contains three folds: i) we design a novel Multiscale Denoising Chain-of-Thought (CoT) to combine global and local semantics with noise filtering step by step, ii) we set up validity principles capable of constraining LLM to generate reasonable predictions following specific formats, and iii) we introduce one-shot In-Context-Learning (ICL) to boost reasoning through imitation, enhancing LLM in TSG task understanding. Experiments demonstrate the state-of-the-art performance of our Grounding-Prompter method, revealing the benefits of prompting LLM with multimodal information for TSG in long videos.
AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompters
New York University Abu Dhabi(纽约大学阿布扎克校区);Nanyang Technological University(南洋理工大学);Zhejiang University(浙江大学);University of Electronic Science and Technology of China(电子科技大学);The Hong Kong Polytechnic University(香港理工大学)
Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering
School of Information Science and Technology, Hangzhou Normal University(杭州师范大学信息科学与技术学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院);School of Computer Science and Technology, Hangzhou Dianzi University(杭州电子科技大学计算机科学与技术学院)
IP-Prompter: Training-Free Theme-Specific Image Generation via Dynamic Visual Prompting
MAIS, Institute of Automation, CAS(MAIS,自动化研究所,中国科学院);China School of Artificial Intelligence, UCAS(中国人工智能学院,UCAS);ByteDance Inc.(字节跳动公司);University of Konstanz(康斯坦茨大学);National Cheng-Kung University(国立成功大学)
Grounding Large Language Models in Reaction Knowledge Graphs for Synthesis Retrieval
Department of Intelligent Systems, Delft University of Technology(智能系统系,代尔夫特理工大学);Department of Chemical Engineering, Delft University of Technology(化学工程系,代尔夫特理工大学)
3DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4o
Shanghai AI Lab(上海人工智能实验室);Dalian University of Technology(大连理工大学);School of Geodesy and Geomatics, Wuhan University(武汉大学测绘学院);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学)
提出 3DAxisPrompt,通过 3D 坐标轴与 SAM 掩码为 GPT-4o 提供几何先验,并在四个数据集上验证其 3D 定位与推理能力。
VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought
Department of Computer Science and Artificial Intelligence, Dongguk University(东国大学计算机科学与人工智能系);Department of Electronics and Electrical Engineering, Dongguk University(东国大学电子与电气工程系)
Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding
Australian Institute for Machine Learning, University of Adelaide(澳大利亚机器学习研究所,阿德莱德大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Macquarie University(麦考瑞大学);Hanoi University of Science and Technology(河内科学技术大学);University of Wollongong(沃林根大学);Flinders University(弗林德斯大学)
Enhancing Vision-Language Models for Autonomous Driving through Task-Specific Prompting and Spatial Reasoning
University of Chinese Academy of Sciences(中国科学院大学);Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences(中国科学院空间利用技术与工程中心)
Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
Department of Computer Science, City University of Hong Kong (Dongguan)(香港城市大学(东莞)计算机科学系);College of Computing and Data Science (CCDS), Nanyang Technological University(南洋理工大学计算与数据科学学院);School of Computer Science, Northwestern Polytechnical University(西北工业大学计算机科学学院)
ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoning
Tsinghua University(清华大学);International Digital Economy Academy(国际数字经济学院);Beihang University(北航);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
University of Freiburg(弗赖堡大学);University of Technology Nuremberg(纽伦堡工业大学)
Recent works have shown that Large Language Models (LLMs) can be applied to ground natural language to a wide variety of robot skills. However, in practice, learning multi-task, language-conditioned robotic skills typically requires large-scale data collection and frequent human intervention to reset the environment or help correcting the current policies. In this work, we propose a novel approach to efficiently learn general-purpose language-conditioned robot skills from unstructured, offline and reset-free data in the real world by exploiting a self-supervised visuo-lingual affordance model,…
P4OMP: Retrieval-Augmented Prompting for OpenMP Parallelism in Serial Code
Mathematics \& Information Technology Concordia University of Edmonton Edmonton, Alberta, Canada;Computer Science University of Manitoba Winnipeg, Manitoba, Canada
LARAG: Link-Aware Retrieval Strategy for RAG Systems in Hyperlinked Technical Documentation
Rulex s.r.l.(Rulex公司);Department of Mathematics (DIMA), University of Genoa(热那亚大学数学系);Department of Computer Science, Bioengineering, Robotics, and Systems Engineering (DIBRIS), University of Genoa(热那亚大学计算机科学、生物工程、机器人学和系统工程系)
Facial-Expression-Aware Prompting for Empathetic LLM Tutoring
Neurosciences Graduate Program, University of California San Diego(加州大学圣地亚哥分校神经科学研究生项目);Department of Cognitive Science, University of California San Diego(加州大学圣地亚哥分校认知科学系);Department of Computer Science and Engineering, University of California San Diego(加州大学圣地亚哥分校计算机科学与工程系);Department of Electrical and Computer Engineering, University of California San Diego(加州大学圣地亚哥分校电气与计算机工程系);Department of Mathematics, University of California San Diego(加州大学圣地亚哥分校数学系);Face the FACS;Halıcıoğlu Data Science Institute, University of California San Diego(Halıcıoğlu数据科学研究所)
Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding
School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);School of Software, Beihang University(北京航空航天大学软件学院);Meituan(美团);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程学院)