SelfReDepth: Self-Supervised Real-Time Depth Restoration for Consumer-Grade Sensors
Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；Instituto de Engenharia de Sistemas e Computadores, Investigação e Desenvolvimento（系统与计算机工程研究所）；Institute for System and Robotics（系统与机器人研究所）；Human Technology Institute, University of Technology Sydney（悉尼科技大学人类技术学院）
Alexandre Duarte, Francisco Fernandes, João M. Pereira, Catarina Moreira, Jacinto C. Nascimento, Joaquim Jorge
Depth maps produced by consumer-grade sensors suffer from inaccurate measurements and missing data from either system or scene-specific sources. Data-driven denoising algorithms can mitigate such problems. However, they require vast amounts of ground truth depth data. Recent research has tackled this limitation using self-supervised learning techniques, but it requires multiple RGB-D sensors. Moreover, most existing approaches focus on denoising single isolated depth maps or specific subjects of interest, highlighting a need for methods to effectively denoise depth maps in real-time dynamic environments. This paper extends state-of-the-art approaches for depth-denoising commodity depth devices, proposing SelfReDepth, a self-supervised deep learning technique for depth restoration, via denoising and hole-filling by inpainting full-depth maps captured with RGB-D sensors. The algorithm targets depth data in video streams, utilizing multiple sequential depth frames coupled with color data to achieve high-quality depth videos with temporal coherence. Finally, SelfReDepth is designed to be compatible with various RGB-D sensors and usable in real-time scenarios as a pre-processing step before applying other depth-dependent algorithms. Our results demonstrate our approach's real-time performance on real-world datasets. They show that it outperforms state-of-the-art denoising and restoration performance at over 30fps on Commercial Depth Cameras, with potential benefits for augmented and mixed-reality applications.
Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing, National Research Council（国家研究委员会先进制造智能工业技术与系统研究所）；University of Salento（萨伦托大学）
Plant phenotyping, that is, the quantitative assessment of plant traits including growth, morphology, physiology, and yield, is a critical aspect towards efficient and effective crop management. Currently, plant phenotyping is a manually intensive and time consuming process, which involves human operators making measurements in the field, based on visual estimates or using hand-held devices. In this work, methods for automated grapevine phenotyping are developed, aiming to canopy volume estimation and bunch detection and counting. It is demonstrated that both measurements can be effectively pe…
Towards Consumer-Grade Cybersickness Prediction: Multi-Model Alignment for Real-Time Vision-Only Inference
The Hong Kong University of Science and Technology(Guangzhou)（香港科学与技术大学（广州））；Nanyang Technological University（南洋理工大学）；Nanyang Technological University Singapore（南洋理工大学新加坡）
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design
Tsinghua University（清华大学）；Xingyun Integrated Circuits Co., Ltd.（星云集成电路有限公司）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
TSAI Lab, University of New Brunswick, Fredericton, NB, Canada（TSAI实验室，新不伦瑞克大学，弗雷德里克顿，NB，加拿大）；Analytics Everywhere Lab, University of New Brunswick, Fredericton, NB, Canada（全方位分析实验室，新不伦瑞克大学，弗雷德里克顿，NB，加拿大）
通过实测OLMoE-1B-7B与稠密模型在Apple M2 Pro和Jetson Orin Nano上的性能，发现MoE的活跃参数优势在笔记本电脑上部分实现，在边缘设备上完全丧失，推理成本与总参数而非活跃参数相关。
Riverbed litter monitoring using consumer-grade aerial-aquatic speedy scanner (AASS) and deep learning based super-resolution reconstruction and detection network
Diagnosing Shoulder Disorders Using Multimodal Large Language Models and Consumer-Grade Cameras
Bytedance（字节跳动）；Peking University（北京大学）；Peking University People’s Hospital（北京大学人民医院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；The Chinese University of Hong Kong（香港中文大学）
The University of Memphis（孟菲斯大学）；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）
The increasing quality and affordability of consumer electroencephalogram (EEG) headsets make them attractive for situations where medical grade devices are impractical. Predicting and tracking cognitive states is possible for tasks that were previously not conducive to EEG monitoring. For instance, monitoring operators for states inappropriate to the task (e.g. drowsy drivers), tracking mental health (e.g. anxiety) and productivity (e.g. tiredness) are among possible applications for the technology. Consumer grade EEG headsets are affordable and relatively easy to use, but they lack the resol…
University of Delaware（特拉华大学）；eBay Research Labs（eBay研究实验室）
Recent advances in consumer depth sensors have created many opportunities for human body measurement and modeling. Estimation of 3D body shape is particularly useful for fashion e-commerce applications such as virtual try-on or fit personalization. In this paper, we propose a method for capturing accurate human body shape and anthropometrics from a single consumer grade depth sensor. We first generate a large dataset of synthetic 3D human body models using real-world body size distributions. Next, we estimate key body measurements from a single monocular depth image. We combine body measuremen…
SwiftVR: Real-Time One-Step Generative Video Restoration
State Key Laboratory of Internet of Things for Smart City, Department of Computer and Information Science, University of Macau（澳门大学智慧城市物联网国家重点实验室）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
Universidad Politécnica de Madrid（马德里理工大学）；Universidad Autónoma de Madrid（马德里自治大学）；Huawei Technologies Co., Ltd.（华为技术有限公司）
FVV Live is a novel end-to-end free-viewpoint video system, designed for low cost and real-time operation, based on off-the-shelf components. The system has been designed to yield high-quality free-viewpoint video using consumer-grade cameras and hardware, which enables low deployment costs and easy installation for immersive event-broadcasting or videoconferencing. The paper describes the architecture of the system, including acquisition and encoding of multiview plus depth data in several capture servers and virtual view synthesis on an edge server. All the blocks of the system have been des…
With the recent proliferation of consumer-grade 360° cameras, it is worth revisiting visual perception challenges with spherical cameras given the potential benefit of their global field of view. To this end we introduce a spherical convolutional hourglass network (SCHN) for the dense labeling on the sphere. The SCHN is invariant to camera orientation (lifting the usual requirement for `upright' panoramic images), and its design is scalable for larger practical datasets. Initial experiments show promising results on a spherical semantic segmentation task.
MyoVision: A Mobile Research Tool and NEATBoost-Attention Ensemble Framework for Real Time Chicken Breast Myopathy Detection
Department of Biological and Agricultural Engineering, University of Arkansas（亚拉巴马大学生物与农业工程系）；Department of Food Science, University of Arkansas（亚拉巴马大学食品科学系）
Heriot-Watt University（赫瑞-瓦特大学）；The University of Edinburgh（爱丁堡大学）；STMicroelectronics（意法半导体）
The process of tracking human anatomy in computer vision is referred to pose estimation, and it is used in fields ranging from gaming to surveillance. Three-dimensional pose estimation traditionally requires advanced equipment, such as multiple linked intensity cameras or high-resolution time-of-flight cameras to produce depth images. However, there are applications, e.g.~consumer electronics, where significant constraints are placed on the size, power consumption, weight and cost of the usable technology. Here, we demonstrate that computational imaging methods can achieve accurate pose estima…
University of Southern California（南加州大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）；Adobe Research（奥多比研究院）
We present an end-to-end system for reconstructing complete watertight and textured models of moving subjects such as clothed humans and animals, using only three or four handheld sensors. The heart of our framework is a new pairwise registration algorithm that minimizes, using a particle swarm strategy, an alignment error metric based on mutual visibility and occlusion. We show that this algorithm reliably registers partial scans with as little as 15% overlap without requiring any initial correspondences, and outperforms alternative global registration algorithms. This registration algorithm …
