For individuals with blindness or low vision (BLV), navigating complex environments can pose serious risks. Large Vision-Language Models (LVLMs) show promise for generating scene descriptions, but their effectiveness for BLV users remains underexplored. To address this gap, we conducted a user study with eight BLV participants to systematically evaluate preferences for six types of LVLM descriptions. While they helped to reduce fear and improve actionability, user ratings showed wide variation in sufficiency and conciseness. Furthermore, GPT-4o--despite its strong potential to refine descriptions--was not consistently preferred by participants. We use the insights obtained from the user study to build training data for building our new automatic evaluation metric that can capture BLV preferences effectively. Our findings underscore the urgent need for BLV-centered evaluation metrics and human-in-the-loop feedback to advance LVLM description quality for accessibility.
A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interaction
Tandon School of Engineering, New York University（纽约大学坦登工程学院）；NYU Langone Health, New York University（纽约大学朗格尼健康中心）；New York University Abu Dhabi（纽约大学阿布扎比分校）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Facebook Reality Labs（Facebook现实实验室）；NVIDIA（英伟达）
Previous blind or No Reference (NR) video quality assessment (VQA) models largely rely on features drawn from natural scene statistics (NSS), but under the assumption that the image statistics are stationary in the spatial domain. Several of these models are quite successful on standard pictures. However, in Virtual Reality (VR) applications, foveated video compression is regaining attention, and the concept of space-variant quality assessment is of interest, given the availability of increasingly high spatial and temporal resolution contents and practical ways of measuring gaze direction. Dis…
Querying Multimodal Scientific Papers with AI: Practices and Preferences Across Blind, Low-Vision, and Sighted Scientists
Paul G. Allen School of Computer Science \& Engineering University of Washington Seattle WA USA；The Information School University of Washington Allen Institute for AI Seattle WA USA；Allen Institute for AI Seattle WA USA；University of Washington；Allen Institute for AI
State Key Laboratory of ASIC & System, College of Microelectronics, Fudan University（复旦大学微电子学院专用集成电路与系统国家重点实验室）；Laboratory for Image and Video Engineering (LIVE), Department of Electrical and Computer Engineering, The University of Texas at Austin（德克萨斯大学奥斯汀分校电气与计算机工程系图像与视频工程实验室）
Video quality assessment (VQA) remains an important and challenging problem that affects many applications at the widest scales. Recent advances in mobile devices and cloud computing techniques have made it possible to capture, process, and share high resolution, high frame rate (HFR) videos across the Internet nearly instantaneously. Being able to monitor and control the quality of these streamed videos can enable the delivery of more enjoyable content and perceptually optimized rate control. Accordingly, there is a pressing need to develop VQA models that can be deployed at enormous scales. …
University of Electronic Science and Technology of China（电子科技大学）
Outdoor vision robotic systems and autonomous cars suffer from many image-quality issues, particularly haze, defocus blur, and motion blur, which we will define generically as "blindness issues". These blindness issues may seriously affect the performance of robotic systems and could lead to unsafe decisions being made. However, existing solutions either focus on one type of blindness only or lack the ability to estimate the degree of blindness accurately. Besides, heavy computation is needed so that these solutions cannot run in real-time on practical systems. In this paper, we provide a meth…
The aim of this work is to contribute to the development of a tactile device for visually impaired and blind persons in order to let them to understand actions of the surrounding people and to interact with them. First, based on the state-of-the-art methods of human action recognition from RGB-D sequences, we use the skeleton information provided by Kinect, with the disentangled and unified multi-scale Graph Convolutional (MS-G3D) model to recognize the performed actions. We tested this model on real scenes and found some of constraints and limitations. Next, we apply a fusion between skeleton…
University of California, Berkeley（加州大学伯克利分校）；University of California, San Diego（加州大学圣地亚哥分校）
We study the problem of deblurring light fields of general 3D scenes captured under 3D camera motion and present both theoretical and practical contributions. By analyzing the motion-blurred light field in the primal and Fourier domains, we develop intuition into the effects of camera motion on the light field, show the advantages of capturing a 4D light field instead of a conventional 2D image for motion deblurring, and derive simple methods of motion deblurring in certain cases. We then present an algorithm to blindly deblur light fields of general scenes without any estimation of scene geom…
Chongqing University of Posts and Telecommunications（重庆邮电大学）；Chongqing University（重庆大学）
Most existing deep-learning-based single image dynamic scene blind deblurring (SIDSBD) methods usually design deep networks to directly remove the spatially-variant motion blurs from one inputted motion blurred image, without blur kernels estimation. In this paper, inspired by the Projective Motion Path Blur (PMPB) model and deformable convolution, we propose a novel constrained deformable convolutional network (CDCN) for efficient single image dynamic scene blind deblurring, which simultaneously achieves accurate spatially-variant motion blur kernels estimation and the high-quality image rest…
Seoul National University（首尔大学）；Inha University（仁荷大学）
Removing camera motion blur from a single light field is a challenging task since it is highly ill-posed inverse problem. The problem becomes even worse when blur kernel varies spatially due to scene depth variation and high-order camera motion. In this paper, we propose a novel algorithm to estimate all blur model variables jointly, including latent sub-aperture image, camera motion, and scene depth from the blurred 4D light field. Exploiting multi-view nature of a light field relieves the inverse property of the optimization by utilizing strong depth cues and multi-view blur observation. The…
We recover a video of the motion taking place in a hidden scene by observing changes in indirect illumination in a nearby uncalibrated visible region. We solve this problem by factoring the observed video into a matrix product between the unknown hidden scene video and an unknown light transport matrix. This task is extremely ill-posed, as any non-negative factorization will satisfy the data. Inspired by recent work on the Deep Image Prior, we parameterize the factor matrices using randomly initialized convolutional neural networks trained in a one-off manner, and show that this results in dec…
Australian National University（澳大利亚国立大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）
Blind Perspective-n-Point (PnP) is the problem of estimating the position and orientation of a camera relative to a scene, given 2D image points and 3D scene points, without prior knowledge of the 2D-3D correspondences. Solving for pose and correspondences simultaneously is extremely challenging since the search space is very large. Fortunately it is a coupled problem: the pose can be found easily given the correspondences and vice versa. Existing approaches assume that noisy correspondences are provided, that a good pose prior is available, or that the problem size is small. We instead propos…
Technique Center Ihome Corporation（Ihome公司技术中心）；IBM Research（IBM研究院）
In this paper, we propose an assistive model that supports professional interior designers to produce industrial interior decoration solutions and to meet the personalized preferences of the property owners. The proposed model is able to automatically produce the layout of objects of a particular indoor scene according to property owners' preferences. In particular, the model consists of the extraction of abstract graph, conditional graph generation, and conditional scene instantiation. We provide an interior layout dataset that contains real-world 11000 designs from professional designers. Ou…
LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences
South China University of Technology（华南理工大学）；Tencent Robotics X（腾讯Robotics X实验室）；Northeastern University（东北大学）；UMass Amherst（马萨诸塞大学阿默斯特分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Pazhou Laboratory（琶洲实验室）；Sichuan University（四川大学）
Don Mariano Marcos Memorial State University（唐马里亚诺马科斯纪念州立大学）；Ateneo de Manila University（马尼拉雅典耀大学）
Text detection in natural scenes has been a significant and active research subject in computer vision and document analysis because of its wide range of applications as evidenced by the emergence of the Robust Reading Competition. One of the algorithms which has good text detection performance in the said competition is the Character Region Awareness for Text Detection (CRAFT). Employing the ICDAR 2013 dataset, this study investigates the impact of automatic image classification and blind deconvolution as image pre-processing steps to further enhance the text detection performance of CRAFT. T…
Shanghai Jiao Tong University（上海交通大学）；City University of Hong Kong（香港城市大学）；Shenzhen Research Institute, City University of Hong Kong（香港城市大学深圳研究院）
We aim at advancing blind image quality assessment (BIQA), which predicts the human perception of image quality without any reference information. We develop a general and automated multitask learning scheme for BIQA to exploit auxiliary knowledge from other tasks, in a way that the model parameter sharing and the loss weighting are determined automatically. Specifically, we first describe all candidate label combinations (from multiple tasks) using a textual template, and compute the joint probability from the cosine similarities of the visual-textual embeddings. Predictions of each task can …
Multi-Frame Blind Manifold Deconvolution for Rotating Synthetic Aperture Imaging
School of Mathematics, Statistics and Actuarial Science, University of Kent（肯特大学数学、统计与精算科学学院）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
Xidian University（西安电子科技大学）；University of Technology, Sydney（悉尼科技大学）
Blind image quality assessment (BIQA) aims to predict perceptual image quality scores without access to reference images. State-of-the-art BIQA methods typically require subjects to score a large number of images to train a robust model. However, subjective quality scores are imprecise, biased, and inconsistent, and it is challenging to obtain a large scale database, or to extend existing databases, because of the inconvenience of collecting images, training the subjects, conducting subjective experiments, and realigning human quality evaluations. To combat these limitations, this paper explor…
HSE University（高等经济大学）；St. Petersburg Department of Steklov Institute of Mathematics（斯捷克洛夫数学研究所圣彼得堡分所）；Samsung-PDMI Joint AI Center（三星-PDMI联合人工智能中心）
In this paper user modeling task is examined by processing a gallery of photos and videos on a mobile device. We propose novel engine for user preference prediction based on scene recognition, object detection and facial analysis. At first, all faces in a gallery are clustered and all private photos and videos with faces from large clusters are processed on the embedded system in offline mode. Other photos may be sent to the remote server to be analyzed by very deep models. The visual features of each photo are obtained from scene recognition and object detection models. These features are agg…
Preference-Driven Active 3D Scene Representation for Robotic Inspection in Nuclear Decommissioning
School of Computing Science, University of Glasgow（格拉斯哥大学计算科学学院）；University of Manchester（曼彻斯特大学）；James Watt School of Engineering, University of Glasgow（格拉斯哥大学詹姆斯·瓦特工程学院）
Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding
Shanghai AI Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；The Hong Kong University of Science and Technology（香港科技大学）；S-Lab, Nanyang Technological University（南洋理工大学S-Lab实验室）
State-of-the-art atmospheric turbulence image restoration methods utilize standard image processing tools such as optical flow, lucky region and blind deconvolution to restore the images. While promising results have been reported over the past decade, many of the methods are agnostic to the physical model that generates the distortion. In this paper, we revisit the turbulence restoration problem by analyzing the reference frame generation and the blind deconvolution steps in a typical restoration pipeline. By leveraging tools in large deviation theory, we rigorously prove the minimum number o…
University of California, Santa Barbara（加州大学圣巴巴拉分校）；University of California（加州大学）
Retinal degenerative diseases cause profound visual impairment in more than 10 million people worldwide, and retinal prostheses are being developed to restore vision to these individuals. Analogous to cochlear implants, these devices electrically stimulate surviving retinal cells to evoke visual percepts (phosphenes). However, the quality of current prosthetic vision is still rudimentary. Rather than aiming to restore "natural" vision, there is potential merit in borrowing state-of-the-art computer vision algorithms as image processing techniques to maximize the usefulness of prosthetic vision…
WinDB: HMD-free and Distortion-free Panoptic Video Fixation Learning
Beihang University（北京航空航天大学）；China University of Petroleum（中国石油大学）；Chinese Academy of Medical Sciences（中国医学科学院）；Pengcheng Laboratory（鹏城实验室）；Stony Brook University（石溪大学）；ETH Zurich（苏黎世联邦理工学院）
3D LiDAR scanners are playing an increasingly important role in autonomous driving as they can generate depth information of the environment. However, creating large 3D LiDAR point cloud datasets with point-level labels requires a significant amount of manual annotation. This jeopardizes the efficient development of supervised deep learning algorithms which are often data-hungry. We present a framework to rapidly create point clouds with accurate point-level labels from a computer game. The framework supports data collection from both auto-driving scenes and user-configured scenes. Point cloud…
IAASARS, National Observatory of Athens（雅典国家天文台 IAASARS）
Estimation of the number of endmembers existing in a scene constitutes a critical task in the hyperspectral unmixing process. The accuracy of this estimate plays a crucial role in subsequent unsupervised unmixing steps i.e., the derivation of the spectral signatures of the endmembers (endmembers' extraction) and the estimation of the abundance fractions of the pixels. A common practice amply followed in literature is to treat endmembers' number estimation and unmixing, independently as two separate tasks, providing the outcome of the former as input to the latter. In this paper, we go beyond t…
COMSATS University Islamabad（伊斯兰堡科姆萨特大学）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
Text detection in natural scene images has applications for autonomous driving, navigation help for elderly and blind people. However, the research on Urdu text detection is usually hindered by lack of data resources. We have developed a dataset of scene images with Urdu text. We present the use of machine learning methods to perform detection of Urdu text from the scene images. We extract text regions using channel enhanced Maximally Stable Extremal Region (MSER) method. First, we classify text and noise based on their geometric properties. Next, we use a support vector machine for early disc…
