SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos
University of Liège（列日大学）；KAUST（阿卜杜拉国王科技大学）；Aalborg University（奥尔堡大学）；Milestone Systems
Adrien Deliège, Anthony Cioppa, Silvio Giancola, Meisam J. Seikavandi, Jacob V. Dueholm, Kamal Nasrollahi, Bernard Ghanem, Thomas B. Moeslund, Marc Van Droogenbroeck
备注：
Paper accepted for the CVsports workshop at CVPR2021. This document contains 8 pages + references + supplementary material
Understanding broadcast videos is a challenging task in computer vision, as it requires generic reasoning capabilities to appreciate the content offered by the video editing. In this work, we propose SoccerNet-v2, a novel large-scale corpus of manual annotations for the SoccerNet video dataset, along with open challenges to encourage more research in soccer understanding and broadcast production. Specifically, we release around 300k annotations within SoccerNet's 500 untrimmed broadcast soccer videos. We extend current tasks in the realm of soccer to include action spotting, camera shot segmentation with boundary detection, and we define a novel replay grounding task. For each task, we provide and discuss benchmark results, reproducible with our open-source adapted implementations of the most relevant works in the field. SoccerNet-v2 is presented to the broader research community to help push computer vision closer to automatic solutions for more general video understanding and production purposes.
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
In this paper, we introduce SoccerNet, a benchmark for action spotting in soccer videos. The dataset is composed of 500 complete soccer games from six main European leagues, covering three seasons from 2014 to 2017 and a total duration of 764 hours. A total of 6,637 temporal annotations are automatically parsed from online match reports at a one minute resolution for three main classes of events (Goal, Yellow/Red Card, and Substitution). As such, the dataset is easily scalable. These annotations are manually refined to a one second resolution by anchoring them at a single timestamp following w…
This brief technical report describes our submission to the Action Spotting SoccerNet Challenge 2022. The challenge was part of the CVPR 2022 ActivityNet Workshop. Our submission was based on a recently proposed method which focuses on increasing temporal precision via a densely sampled set of detection anchors. Due to its emphasis on temporal precision, this approach had shown significant improvements in the tight average-mAP metric. Tight average-mAP was used as the evaluation criterion for the challenge, and is defined using small temporal evaluation tolerances, thus being more sensitive to…
University of Liège（列日大学）；EVS Broadcast Equipment（EVS广播设备公司）；KAUST（阿卜杜拉国王科技大学）
Soccer broadcast video understanding has been drawing a lot of attention in recent years within data scientists and industrial companies. This is mainly due to the lucrative potential unlocked by effective deep learning techniques developed in the field of computer vision. In this work, we focus on the topic of camera calibration and on its current limitations for the scientific community. More precisely, we tackle the absence of a large-scale calibration dataset and of a public calibration network trained on such a dataset. Specifically, we distill a powerful commercial calibration tool in a …
This work aims at generating captions for soccer videos using deep learning. In this context, this paper introduces a dataset, model, and triple-level evaluation. The dataset consists of 22k caption-clip pairs and three visual features (images, optical flow, inpainting) for ~500 hours of \emph{SoccerNet} videos. The model is divided into three parts: a transformer learns language, ConvNets learn vision, and a fusion of linguistic and visual features generates captions. The paper suggests evaluating generated captions at three levels: syntax (the commonly used evaluation metrics such as BLEU-sc…
KAUST（阿卜杜拉国王科技大学）；University of Liège（列日大学）；EVS Broadcast Equipment（EVS广播设备公司）；Sportradar（Sportradar公司）；UCLouvain（鲁汶天主教大学）；EPFL（洛桑联邦理工学院）；Baidu Research（百度研究院）
The SoccerNet 2022 challenges were the second annual video understanding challenges organized by the SoccerNet team. In 2022, the challenges were composed of 6 vision-based tasks: (1) action spotting, focusing on retrieving action timestamps in long untrimmed videos, (2) replay grounding, focusing on retrieving the live moment of an action shown in a replay, (3) pitch localization, focusing on detecting line and goal part elements, (4) camera calibration, dedicated to retrieving the intrinsic and extrinsic camera parameters, (5) player re-identification, focusing on retrieving the same players…
University of Liege (ULiège)（列日大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Sportradar（Sportradar公司）；UCLouvain（鲁汶大学）；EPFL（洛桑联邦理工学院）；EVS Broadcast Equipment（EVS广播设备公司）；Baidu Research（百度研究院）；NASK National Research Institute（NASK国家研究院）；Université Libre de Bruxelles (ULB)（布鲁塞尔自由大学）；Sharif University of Technology（谢里夫理工大学）；Footovision（Footovision公司）；Mobius Labs（Mobius Labs公司）；Institut de Robòtica i Informàtica Industrial, CSIC-UPC（CSIC-UPC机器人与工业自动化研究所）；TAHAKOM（TAHAKOM公司）；AIBrain（AIBrain公司）
University of Liège（列日大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Spiideo（斯皮迪奥公司）；Aalborg University（奥尔堡大学）；SpAItial（斯帕蒂亚尔公司）；Center for Robotics, Mines Paris, PSL（巴黎矿业学院机器人中心（巴黎文理研究大学））；Footovision（Footovision公司）；Shanghai Jiao Tong University（上海交通大学）；Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；EVS Broadcast Equipment（EVS广播设备公司）；Pioneer Center for Artificial Intelligence（先锋人工智能中心）；Lund University（隆德大学）；TAHAKOM（TAHAKOM公司）；Mohamed Bin Zayed University for Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）；Sber AI（Sber人工智能公司）；Salute For Business（Salute For Business公司）；Intelligent Perception and Image Understanding Lab, Xidian University（西安电子科技大学智能感知与图像理解实验室）；South China University of Technology（华南理工大学）；Hefei University of Technology（合肥工业大学）；Kyungpook National University（庆北国立大学）；Leipzig University of Applied Sciences（莱比锡应用科学大学）；Friedrich-Alexander University Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；University of Seoul（首尔大学）；Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（电子科技大学深圳高等研究院）；Nanjing University（南京大学）；University of Science, Ho Chi Minh City（胡志明市科技大学）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Sportradar（Sportradar公司）；UCLouvain（鲁汶大学）；EPFL（洛桑联邦理工学院）；EVS Broadcast Equipment（EVS广播设备公司）；Baidu Research（百度研究院）；Sharif University of Technology（谢里夫理工大学）；Footovision（Footovision公司）；Zewail City of Science, Technology and Innovation（扎维尔科学技术与创新城）；Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Nagoya University（名古屋大学）；Research Center for Applied Mathematics and Machine Intelligence, Zhejiang Lab（之江实验室应用数学与机器智能研究中心）；AIBrain（AIBrain公司）；OPPO Research Institute（OPPO研究院）；University of the Bundeswehr Munich（慕尼黑联邦国防军大学）；Meituan（美团公司）；Tencent Youtu Lab（腾讯优图实验室）；Amazon Prime Video Sport（亚马逊Prime Video体育部门）；Sogang University（西江大学）；The University of Tokyo（东京大学）；Spiideo（Spiideo公司）；University of Tsukuba（筑波大学）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Normandie Univ, INSA Rouen, LITIS（诺曼底大学、鲁茵国立应用科学学院、LITIS机构）
In this paper, we propose a study on multi-modal (audio and video) action spotting and classification in soccer videos. Action spotting and classification are the tasks that consist in finding the temporal anchors of events in a video and determine which event they are. This is an important application of general activity understanding. Here, we propose an experimental study on combining audio and video information at different stages of deep neural network architectures. We used the SoccerNet benchmark dataset, which contains annotated events for 500 soccer game videos from the Big Five Europ…
We present a model for temporally precise action spotting in videos, which uses a dense set of detection anchors, predicting a detection confidence and corresponding fine-grained temporal displacement for each anchor. We experiment with two trunk architectures, both of which are able to incorporate large temporal contexts while preserving the smaller-scale features required for precise localization: a one-dimensional version of a u-net, and a Transformer encoder (TE). We also suggest best practices for training models of this kind, by applying Sharpness-Aware Minimization (SAM) and mixup data …
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Toward the goal of automatic production for sports broadcasts, a paramount task consists in understanding the high-level semantic information of the game in play. For instance, recognizing and localizing the main actions of the game would allow producers to adapt and automatize the broadcast production, focusing on the important details of the game and maximizing the spectator engagement. In this paper, we focus our analysis on action spotting in soccer broadcast, which consists in temporally localizing the main actions in a soccer game. To that end, we propose a novel feature pooling method b…
In problems such as sports video analytics, it is difficult to obtain accurate frame level annotations and exact event duration because of the lengthy videos and sheer volume of video data. This issue is even more pronounced in fast-paced sports such as ice hockey. Obtaining annotations on a coarse scale can be much more practical and time efficient. We propose the task of event detection in coarsely annotated videos. We introduce a multi-tower temporal convolutional network architecture for the proposed task. The network, with the help of multiple receptive fields, processes information at va…
L3S Research Center（L3S研究中心）；Leibniz University Hannover（汉诺威莱布尼茨大学）；TIB – Leibniz Information Centre for Science and Technology（德国科学技术莱布尼茨信息中心）；German Sport University Cologne（德国科隆体育大学）
Computer-aided support and analysis are becoming increasingly important in the modern world of sports. The scouting of potential prospective players, performance as well as match analysis, and the monitoring of training programs rely more and more on data-driven technologies to ensure success. Therefore, many approaches require large amounts of data, which are, however, not easy to obtain in general. In this paper, we propose a pipeline for the fully-automated extraction of positional data from broadcast video recordings of soccer matches. In contrast to previous work, the system integrates al…
Field Converter: Geometry-Initialized Temporal Residual Refinement for World-Grounded Player Pose Estimation from Soccer Broadcasts
Institut de Biomécanique Humaine Georges Charpak (IBHGC), Arts et Métiers ParisTech（乔治·沙帕克人体生物力学研究所（IBHGC），巴黎高科国立高等工艺学校）；French Football Federation (FFF)（法国足球联合会（FFF））
In this work, we propose a novel way of efficiently localizing a soccer field from a single broadcast image of the game. Related work in this area relies on manually annotating a few key frames and extending the localization to similar images, or installing fixed specialized cameras in the stadium from which the layout of the field can be obtained. In contrast, we formulate this problem as a branch and bound inference in a Markov random field where an energy function is defined in terms of field cues such as grass, lines and circles. Moreover, our approach is fully automatic and depends only o…
SoccerChat: Integrating Multimodal Data for Enhanced Soccer Game Understanding
SimulaMet and OsloMet, Norway（SimulaMet和OsloMet，挪威）；Forzasys, Norway（Forzasys，挪威）；SimulaMet, Norway（SimulaMet，挪威）；SimulaMet, OsloMet, and Forzasys, Norway（SimulaMet、OsloMet和Forzasys，挪威）；University of Central Florida, USA（佛罗里达中央大学）
L3S Research Center（L3S研究中心）；Leibniz University Hannover（汉诺威莱布尼茨大学）；TIB – Leibniz Information Centre for Science and Technology（莱布尼茨科学技术信息中心）
Sports field registration in broadcast videos is typically interpreted as the task of homography estimation, which provides a mapping between a planar field and the corresponding visible area of the image. In contrast to previous approaches, we consider the task as a camera calibration problem. First, we introduce a differentiable objective function that is able to learn the camera pose and focal length from segment correspondences (e.g., lines, point clouds), based on pixel-level annotations for segments of a known calibration object. The calibration module iteratively minimizes the segment r…
With rapidly evolving internet technologies and emerging tools, sports related videos generated online are increasing at an unprecedentedly fast pace. To automate sports video editing/highlight generation process, a key task is to precisely recognize and locate the events in the long untrimmed videos. In this tech report, we present a two-stage paradigm to detect what and when events happen in soccer broadcast videos. Specifically, we fine-tune multiple action recognition models on soccer data to extract high-level semantic features, and design a transformer based temporal detection module to …
University of Liège（列日大学）；KAUST（阿卜杜拉国王科技大学）；Aalborg University（奥尔堡大学）
In video understanding, action spotting consists in temporally localizing human-induced events annotated with single timestamps. In this paper, we propose a novel loss function that specifically considers the temporal context naturally present around each action, rather than focusing on the single annotated frame to spot. We benchmark our loss on a large dataset of soccer videos, SoccerNet, and achieve an improvement of 12.8% over the baseline. We show the generalization capability of our loss for generic activity proposals and detection on ActivityNet, by spotting the beginning and the end of…
South China University of Technology（华南理工大学）；Johns Hopkins University（约翰霍普金斯大学）；Peking University（北京大学）；University of Electronic Science and Technology of China（电子科技大学）
University of British Columbia（不列颠哥伦比亚大学）；National University of Defense Technology（国防科学技术大学）
This work addresses camera selection, the task of predicting which camera should be "on air" from multiple candidate cameras for soccer broadcast. The task is challenging because of the scarcity of learning data with all candidate views. Meanwhile, broadcast videos are freely available on the Internet (e.g. Youtube). However, these videos only record the selected camera views, omitting the other candidate views. To overcome this problem, we first introduce a random survival forest (RSF) method to impute the incomplete data effectively. Then, we propose a spatial-appearance heatmap to describe …
