Recent urbanization has coincided with the enrichment of geotagged data, such as street view and point-of-interest (POI). Region embedding enhanced by the richer data modalities has enabled researchers and city administrators to understand the built environment, socioeconomics, and the dynamics of cities better. While some efforts have been made to simultaneously use multi-modal inputs, existing methods can be improved by incorporating different measures of 'proximity' in the same embedding space - leveraging not only the data that characterizes the regions (e.g., street view, local businesses pattern) but also those that depict the relationship between regions (e.g., trips, road network). To this end, we propose a novel approach to integrate multi-modal geotagged inputs as either node or edge features of a multi-graph based on their relations with the neighborhood region (e.g., tiles, census block, ZIP code region, etc.). We then learn the neighborhood representation based on a contrastive-sampling scheme from the multi-graph. Specifically, we use street view images and POI features to characterize neighborhoods (nodes) and use human mobility to characterize the relationship between neighborhoods (directed edges). We show the effectiveness of the proposed methods with quantitative downstream tasks as well as qualitative analysis of the embedding space: The embedding we trained outperforms the ones using only unimodal data as regional inputs.
Stanford University（斯坦福大学）；Renmin University of China（中国人民大学）
Understanding intrinsic patterns and predicting spatiotemporal characteristics of cities require a comprehensive representation of urban neighborhoods. Existing works relied on either inter- or intra-region connectivities to generate neighborhood representations but failed to fully utilize the informative yet heterogeneous data within neighborhoods. In this work, we propose Urban2Vec, an unsupervised multi-modal framework which incorporates both street view imagery and point-of-interest (POI) data to learn neighborhood embeddings. Specifically, we use a convolutional neural network to extract …
A longitudinal geospatial multimodal dataset of post-discharge frailty, physiology, mobility, and neighborhoods
University of Toronto（多伦多大学）；KITE Research Institute, Toronto Rehabilitation Institute, University Health Network（大学健康网络多伦多康复研究所KITE研究院）；American University of the Middle East（中东美国大学）
The abundance of multimodal data (e.g. social media posts) has inspired interest in cross-modal retrieval methods. Popular approaches rely on a variety of metric learning losses, which prescribe what the proximity of image and text should be, in the learned space. However, most prior methods have focused on the case where image and text convey redundant information; in contrast, real-world image-text pairs convey complementary information with little overlap. Further, images in news articles and media portray topics in a visually diverse fashion; thus, we need to take special care to ensure a …
Feed-forward networks are widely used in cross-modal applications to bridge modalities by mapping distributed vectors of one modality to the other, or to a shared space. The predicted vectors are then used to perform e.g., retrieval or labeling. Thus, the success of the whole system relies on the ability of the mapping to make the neighborhood structure (i.e., the pairwise similarities) of the predicted vectors akin to that of the target vectors. However, whether this is achieved has not been investigated yet. Here, we propose a new similarity measure and two ad hoc experiments to shed light o…
The novel coronavirus disease (COVID-19) has crushed daily routines and is still rampaging through the world. Existing solution for nonpharmaceutical interventions usually needs to timely and precisely select a subset of residential urban areas for containment or even quarantine, where the spatial distribution of confirmed cases has been considered as a key criterion for the subset selection. While such containment measure has successfully stopped or slowed down the spread of COVID-19 in some countries, it is criticized for being inefficient or ineffective, as the statistics of confirmed cases…
In-Context Learning to Assess Built Environment Impacts on Perceived Neighborhood Walkability Among Mobility-impaired Older Adults
Future Health Technologies, Singapore-ETH Centre（未来健康技术中心，新加坡-ETH中心）；Institute of Geriatrics and Active Ageing, Tan Tock Seng Hospital（陈笃生医院老年病学与 active 老龄化研究所）；Geriatric Medicine, Khoo Teck Puat Hospital（邱德拔医院老年医学科）；Department of Orthopedic Surgery, Woodlands Health Campus（伍德兰兹健康园区骨科）
Parkinson's disease (PD) is a slowly progressive debilitating neurodegenerative disease which is prominently characterised by motor symptoms. Indoor localisation, including number and speed of room to room transitions, provides a proxy outcome which represents mobility and could be used as a digital biomarker to quantify how mobility changes as this disease progresses. We use data collected from 10 people with Parkinson's, and 10 controls, each of whom lived for five days in a smart home with various sensors. In order to more effectively localise them indoors, we propose a transformer-based ap…
Beyond travel mode: urban context shapes active mobility's mental health effects over time
Martin Centre for Architectural and Urban Studies, University of Cambridge（剑桥大学马丁建筑与城市研究中心）；Cambridge Public Health, University of Cambridge（剑桥大学剑桥公共卫生）
Understanding the determinants underlying variations in urban health status is important for informing urban design and planning, as well as public health policies. Multiple heterogeneous urban features could modulate the prevalence of diseases across different neighborhoods in cities and across different cities. This study examines heterogeneous features related to socio-demographics, population activity, mobility, and the built environment and their non-linear interactions to examine intra- and inter-city disparity in prevalence of four disease types: obesity, diabetes, cancer, and heart dis…
Driving, Fast or Slow? Neuro-Symbolic Guidance for Motion Prediction in Multi-Modal Ground Mobility
Artificial Intelligence and Machine Learning Lab, TU Darmstadt（达姆施塔特工业大学人工智能与机器学习实验室）；Honda Research Institute（本田研究所）；Hessian Center for AI (hessian.AI)（黑森州人工智能中心）；Centre for Cognitive Science（认知科学中心）；German Center for AI (DFKI)（德国人工智能研究中心）；Uncertainty in Artificial Intelligence Lab, TU Eindhoven（埃因霍温理工大学人工智能不确定性实验室）
Synergistic Fusion of Topological Structure and Temporal Semantics of Mobility for Urban Region Embedding
Urban AI Institute, Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院城市人工智能研究院）；Graduate School of Data Science, KAIST（韩国科学技术院数据科学研究生院）；Department of Industrial Engineering and Systems, KAIST（韩国科学技术院工业工程与系统系）；Department of Civil and Environmental Engineering, KAIST（韩国科学技术院土木与环境工程系）
Most real world applications of image retrieval such as Adobe Stock, which is a marketplace for stock photography and illustrations, need a way for users to find images which are both visually (i.e. aesthetically) and conceptually (i.e. containing the same salient objects) as a query image. Learning visual-semantic representations from images is a well studied problem for image retrieval. Filtering based on image concepts or attributes is traditionally achieved with index-based filtering (e.g. on textual tags) or by re-ranking after an initial visual embedding based retrieval. In this paper, w…
Embeddings mapping high-dimensional discrete input to lower-dimensional continuous vector spaces have been widely adopted in machine learning applications as a way to capture domain semantics. Interviewing 13 embedding users across disciplines, we find comparing embeddings is a key task for deployment or downstream analysis but unfolds in a tedious fashion that poorly supports systematic exploration. In response, we present the Embedding Comparator, an interactive system that presents a global comparison of embedding spaces alongside fine-grained inspection of local neighborhoods. It systemati…
