Multi-view data capture permits free-viewpoint video (FVV) content creation. To this end, several users must capture video streams, calibrated in both time and pose, framing the same object/scene, from different viewpoints. New-generation network architectures (e.g. 5G) promise lower latency and larger bandwidth connections supported by powerful edge computing, properties that seem ideal for reliable FVV capture. We have explored this possibility, aiming to remove the need for bespoke synchronisation hardware when capturing a scene from multiple viewpoints, making it possible through off-the-shelf mobiles. We propose a novel and scalable data capture architecture that exploits edge resources to synchronise and harvest frame captures. We have designed an edge computing unit that supervises the relaying of timing triggers to and from multiple mobiles, in addition to synchronising frame harvesting. We empirically show the benefits of our edge computing unit by analysing latencies and show the quality of 3D reconstruction outputs against an alternative and popular centralised solution based on Unity3D.
Universitá degli Studi di Trento（特伦托大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
We propose a system to capture nearly-synchronous frame streams from multiple and moving handheld mobiles that is suitable for dynamic object 3D reconstruction. Each mobile executes Simultaneous Localisation and Mapping on-board to estimate its pose, and uses a wireless communication channel to send or receive synchronisation triggers. Our system can harvest frames and mobile poses in real time using a decentralised triggering strategy and a data-relay architecture that can be deployed either at the Edge or in the Cloud. We show the effectiveness of our system by employing it for 3D skeleton a…
Mobile-4DGS: Unified Static-Dynamic Real-time Mobile Gaussian Splatting
University of Technology Sydney（悉尼科技大学）；Yale University（耶鲁大学）；City University of Macau（澳门城市大学）；Australian National University（澳大利亚国立大学）；Adelaide University（阿德莱德大学）
An Object-Centered Data Acquisition Method for 3D Gaussian Splatting using Mobile Phones
School of Electronics and Information, Northwestern Polytechnical University（电子信息学院，西北工业大学）；College of Engineering, Xi'an International University（工程学院，西安国际大学）
3D photography is a new medium that allows viewers to more fully experience a captured moment. In this work, we refer to a 3D photo as one that displays parallax induced by moving the viewpoint (as opposed to a stereo pair with a fixed viewpoint). 3D photos are static in time, like traditional photos, but are displayed with interactive parallax on mobile or desktop screens, as well as on Virtual Reality devices, where viewing it also includes stereo. We present an end-to-end system for creating and viewing 3D photos, and the algorithmic and design choices therein. Our 3D photos are captured in…
Toward Semantic Communication for Real-time Mobile 3D Reconstruction
School of Electrical and Electronic Engineering, North China Electric Power University（华北电力大学电气与电子工程学院）；James Watt School of Engineering, University of Glasgow（格拉斯哥大学詹姆斯·瓦特工程学院）；School of Computer Science and Technology, China University of Petroleum (East China)（中国石油大学（华东）计算机科学与技术学院）
针对实时移动 3D 重建中几何估计对通信失真敏感的问题，提出语义通信框架，含语义收发器输出带置信度图的重建图像，引入置信度引导几何估计方法，仿真显示该框架能保持图像质量并提升姿态估计精度和 3D 结构一致性。
High user interaction capability of mobile devices can help improve the accuracy of mobile visual search systems. At query time, it is possible to capture multiple views of an object from different viewing angles and at different scales with the mobile device camera to obtain richer information about the object compared to a single view and hence return more accurate results. Motivated by this, we developed a mobile multi-view object image search system, using a client-server architecture. Multi-view images of objects acquired by the mobile clients are processed and local features are sent to …
High-quality 3D ground-truth shapes are critical for 3D object reconstruction evaluation. However, it is difficult to create a replica of an object in reality, and even 3D reconstructions generated by 3D scanners have artefacts that cause biases in evaluation. To address this issue, we introduce a novel multi-view RGBD dataset captured using a mobile device, which includes highly precise 3D ground-truth annotations for 153 object models featuring a diverse set of 3D structures. We obtain precise 3D ground-truth shape without relying on high-end 3D scanners by utilising LEGO models with known g…
Embodied Multimodal Grounding for Open-Vocabulary Mobile Manipulation via Semantic 3D Gaussian Splatting
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Midea Group（美的集团）；The Hong Kong University of Science and Technology（香港科技大学）
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Calgary（卡尔加里大学）
This manuscript provides a new framework for calibration of optical instruments, in particular mobile cameras, using large-scale circular black and white target fields. New methods were introduced for (i) matching targets between images; (ii) adjusting the systematic eccentricity error of target centers; and (iii) iteratively improving the calibration solution through a free-network self-calibrating bundle adjustment. It was observed that the proposed target matching effectively matched circular targets in 270 mobile phone images from a complete calibration laboratory with robustness to Type I…
Beyond Appearance: A Multi-cue Framework and Large-scale Benchmark for Pedestrian Association and Tracking on Mobile Aerial-Ground Platforms
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机科学学院）；Ningbo Institute, Northwestern Polytechnical University（西北工业大学宁波研究院）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（国家空天地海一体化大数据应用技术工程实验室）；Shenzhen University of Advanced Technology（深圳先进技术研究院）
With the rapid development of wireless networking and mobile devices, anytime and anywhere data access becomes readily available nowadays. Given the crowdsourced content capturing and sharing, the preferred content length becomes shorter and shorter, even for such multimedia data as video. A representative is Twitter's Vine service, which, mainly targeting mobile users, enables them to create ultra-short video clips and instantly post and share with their followers. In this paper, we present an initial study on this new generation of instant video clip sharing service enabled by mobile platfor…
Multi-view product image queries can improve retrieval performance over single view queries significantly. In this paper, we investigated the performance of deep convolutional neural networks (ConvNets) on multi-view product image search. First, we trained a VGG-like network to learn deep ConvNets representations of product images. Then, we computed the deep ConvNets representations of database and query images and performed single view queries, and multi-view queries using several early and late fusion approaches. We performed extensive experiments on the publicly available Multi-View Object …
FVV Live is a novel real-time, low-latency, end-to-end free viewpoint system including capture, transmission, synthesis on an edge server and visualization and control on a mobile terminal. The system has been specially designed for low-cost and real-time operation, only using off-the-shelf components.
With the emergence of naked-eye 3D mobile devices, mobile 3D video services are becoming increasingly important for video service providers, such as Youtube and Netflix, while multi-view 3D videos have the potential to inspire a variety of innovative applications. However, enabling multi-view 3D video services may overwhelm WiFi networks when every view of a video are multicasted. In this paper, therefore, we propose to incorporate depth-image-based rendering (DIBR), which allows each mobile client to synthesize the desired view from nearby left and right views, in order to effectively reduce …
With the continuous growth in the consumer markets of mobile smartphones and increasingly in augmented reality wearable devices, several avenues of research investigate the relationships between the quality perceived by mobile users and the delivery mechanisms at play to support a high quality of experience for mobile users. In this paper, we present the first study that evaluates the relationships of mobile movie quality and the viewer-perceived quality thereof in an augmented reality setting with see-through devices. We find that participants tend to overestimate the video quality and exhibi…
