arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.10434cs.CV
收藏

Spotter+GPT:利用LLM将手语识别转化为句子

Spotter+GPT: Turning Sign Spottings into Sentences with LLMs

CVSSP;University of Surrey(萨里大学)

Ozge Mercanoglu Sincan, Richard Bowden

AI总结:

本文提出轻量级模块化手语翻译框架Spotter+GPT,通过手语识别器识别视频中的手语并交由LLM转化为口语句子,避免端到端训练,显著降低计算成本。

摘要:

手语翻译(SLT)是一项极具挑战性的任务,旨在从手语视频中生成口语句子。本文介绍了一种轻量级、模块化的SLT框架Spotter+GPT,该框架利用大型语言模型(LLMs)的能力,避免了繁重的端到端训练。Spotter+GPT将SLT任务分为两个不同阶段。首先,手语识别器识别输入视频中的单个手语。然后将识别出的手语传递给LLM,由其将其转化为有意义的口语句子。Spotter+GPT消除了对SLT特定训练的需求。这显著降低了计算成本和时间要求。Spotter的源代码和预训练权重可在https://gitlab.surrey.ac.uk/cogvispublic/sign-spotter获取。

英文摘要:

Sign Language Translation (SLT) is a challenging task that aims to generate spoken language sentences from sign language videos. In this paper, we introduce a lightweight, modular SLT framework, Spotter+GPT, that leverages the power of Large Language Models (LLMs) and avoids heavy end-to-end training. Spotter+GPT breaks down the SLT task into two distinct stages. First, a sign spotter identifies individual signs within the input video. The spotted signs are then passed to an LLM, which transforms them into meaningful spoken language sentences. Spotter+GPT eliminates the requirement for SLT-specific training. This significantly reduces computational costs and time requirements. The source code and pretrained weights of the Spotter are available at https://gitlab.surrey.ac.uk/cogvispublic/sign-spotter.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑