A range of video modeling tasks, from optical flow to multiple object tracking, share the same fundamental challenge: establishing space-time correspondence. Yet, approaches that dominate each space differ. We take a step towards bridging this gap by extending the recent contrastive random walk formulation to much denser, pixel-level space-time graphs. The main contribution is introducing hierarchy into the search problem by computing the transition matrix between two frames in a coarse-to-fine manner, forming a multiscale contrastive random walk when extended in time. This establishes a unified technique for self-supervised learning of optical flow, keypoint tracking, and video object segmentation. Experiments demonstrate that, for each of these tasks, the unified model achieves performance competitive with strong self-supervised approaches specific to that task. Project webpage: https://jasonbian97.github.io/flowwalk
Advanced Technology Development Centre（先进技术发展中心）；Department of Computer Science and Engineering（计算机科学与工程系）；Department of Electrical Engineering（电气工程系）；Indian Institute of Technology Kharagpur（印度理工学院卡哈拉格普尔分校）；India（印度（注：根据上下文，此处“India”为国家，非机构，原数据可能存在格式问题，按要求不提取国家信息，但按输出要求保留英文机构名））
Coronary artery disease accounts for a large number of deaths across the world and clinicians generally prefer using x-ray computed tomography or magnetic resonance imaging for localizing vascular pathologies. Interventional imaging modalities like intravascular ultrasound (IVUS) are used to adjunct diagnosis of atherosclerotic plaques in vessels, and help assess morphological state of the vessel and plaque, which play a significant role for treatment planning. Since speckle intensity in IVUS images are inherently stochastic in nature and challenge clinicians with accurate visibility of the ve…
Sounds reach one microphone in a stereo pair sooner than the other, resulting in an interaural time delay that conveys their directions. Estimating a sound's time delay requires finding correspondences between the signals recorded by each microphone. We propose to learn these correspondences through self-supervision, drawing on recent techniques from visual tracking. We adapt the contrastive random walk of Jabri et al. to learn a cycle-consistent representation from unlabeled stereo sounds, resulting in a model that performs on par with supervised methods on "in the wild" internet recordings. …
Multiscale Switch for Semi-Supervised and Contrastive Learning in Medical Ultrasound Image Segmentation
Department of Health Technology and Informatics, The Hong Kong Polytechnic University（健康科技与信息学系，香港理工大学）；Centre for Smart Health and School of Nursing, The Hong Kong Polytechnic University（智能健康中心及护理学院，香港理工大学）；Department of Imaging and Interventional Radiology, The Chinese University of Hong Kong（影像与介入放射科，香港中文大学）
UC Berkeley（加州大学伯克利分校）；University of Michigan（密歇根大学）
This paper proposes a simple self-supervised approach for learning a representation for visual correspondence from raw video. We cast correspondence as prediction of links in a space-time graph constructed from video. In this graph, the nodes are patches sampled from each frame, and nodes adjacent in time can share a directed edge. We learn a representation in which pairwise similarity defines transition probability of a random walk, so that long-range correspondence is computed as a walk along the graph. We optimize the representation to place high probability along paths of similarity. Targe…
Learning Gait Representation from Massive Unlabelled Walking Videos: A Benchmark
Southern University of Science and Technology（南方科技大学）；Beijing Normal University（北京师范大学）；Watrix Technology Limited Co. Ltd.（银河水滴科技有限公司）；University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
University of Michigan（密歇根大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
We present a method for simultaneously localizing multiple sound sources within a visual scene. This task requires a model to both group a sound mixture into individual sources, and to associate them with a visual signal. Our method jointly solves both tasks at once, using a formulation inspired by the contrastive random walk of Jabri et al. We create a graph in which images and separated sounds correspond to nodes, and train a random walker to transition between nodes from different modalities with high return probability. The transition probabilities for this walk are determined by an audio-…
