Phase Retrieval using Single-Instance Deep Generative Prior

利用语义角色上下文化视频特征进行多实例文本-视频检索：EPIC-KITCHENS-100 2022 多实例检索挑战赛

Exploiting Semantic Role Contextualized Video Features for Multi-Instance Text-Video Retrieval EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022

提出一种语义角色解析与自注意力多空间嵌入方法，用于 EPIC-KITCHENS-100 文本-视频多实例检索，并在 nDCG 和 mAP 上分别取得第 3、第 4 名。

Robust Compressive Phase Retrieval via Deep Generative Priors

This paper proposes a new framework to regularize the highly ill-posed and non-linear phase retrieval problem through deep generative priors using simple gradient descent algorithm. We experimentally show effectiveness of proposed algorithm for random Gaussian measurements (practically relevant in imaging through scattering media) and Fourier friendly measurements (relevant in optical set ups). We demonstrate that proposed approach achieves impressive results when compared with traditional hand engineered priors including sparsity and denoising frameworks for number of measurements and robustn…

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering

Generative models for open domain question answering have proven to be competitive, without resorting to external knowledge. While promising, this approach requires to use models with billions of parameters, which are expensive to train and query. In this paper, we investigate how much these models can benefit from retrieving text passages, potentially containing evidence. We obtain state-of-the-art results on the Natural Questions and TriviaQA open benchmarks. Interestingly, we observe that the performance of this method significantly improves when increasing the number of retrieved passages.…

Exact asymptotics for phase retrieval and compressed sensing with random generative priors

We consider the problem of compressed sensing and of (real-valued) phase retrieval with random measurement matrix. We derive sharp asymptotics for the information-theoretically optimal performance and for the best known polynomial algorithm for an ensemble of generative priors consisting of fully connected deep neural networks with random weight matrices and arbitrary activations. We compare the performance to sparse separable priors and conclude that generative priors might be advantageous in terms of algorithmic performance. In particular, while sparsity does not allow to perform compressive…

PtyGenography：使用生成模型进行相位恢复问题的正则化

PtyGenography: using generative models for regularization of the phase retrieval problem

本文针对相位恢复等逆问题，提出一种新的统一重建方法，通过使用生成模型作为信号先验进行正则化，有效减轻了不同噪声水平下对生成模型的过拟合并缓解重建偏差。

QueStER：生成式关键词检索的查询规范

QueStER: Query Specification for Generative keyword-based Retrieval

QueStER通过生成显式的关键词检索规范，结合生成式查询重写和传统检索的效率，提升跨领域检索性能。

Normalized Contrastive Learning for Text-Video Retrieval

Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness. In this work, however, we reveal that cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance. Specifically, we show that many test instances are either over- or under-represented during retrieval, significantly hurting the retrieval performance. To address this problem, we propose Normalized Contrastive Learning (NCL) which utilizes the Sinkhorn-Knopp algorithm to compute the instance-wise bi…

可解释的贝叶斯多视角生成式检索

eXplainable Bayesian Multi-Perspective Generative Retrieval

该研究将贝叶斯不确定性校准与多视角检索结合，利用LIME和SHAP解释分数增强重排序，在问答和事实验证任务的三个KILT数据集上显著提升性能。

IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval Models

This paper provides a unified account of two schools of thinking in information retrieval modelling: the generative retrieval focusing on predicting relevant documents given a query, and the discriminative retrieval focusing on predicting relevancy given a query-document pair. We propose a game theoretical minimax game to iteratively optimise both models. On one hand, the discriminative model, aiming to mine signals from labelled and unlabelled data, provides guidance to train the generative model towards fitting the underlying relevance distribution over documents given the query. On the othe…

Evaluating a Generative Adversarial Framework for Information Retrieval

Recent advances in Generative Adversarial Networks (GANs) have resulted in its widespread applications to multiple domains. A recent model, IRGAN, applies this framework to Information Retrieval (IR) and has gained significant attention over the last few years. In this focused work, we critically analyze multiple components of IRGAN, while providing experimental and theoretical evidence of some of its shortcomings. Specifically, we identify issues with the constant baseline term in the policy gradients optimization and show that the generator harms IRGAN's performance. Motivated by our finding…

用于实例检索的局部保持马尔可夫转移

Locality Preserving Markovian Transition for Instance Retrieval

本文提出局部保持马尔可夫转移（LPMT）框架，通过双向协同扩散、局部状态嵌入和热力学马尔可夫转移过程，在保持局部判别力的同时实现高效全局实例检索。

Classroom Video Assessment and Retrieval via Multiple Instance Learning

We propose a multiple instance learning approach to content-based retrieval of classroom video for the purpose of supporting human assessing the learning environment. The key element of our approach is a mapping between the semantic concepts of the assessment system and features of the video that can be measured using techniques from the fields of computer vision and speech analysis. We report on a formative experiment in content-based video retrieval involving trained experts in the Classroom Assessment Scoring System, a widely used framework for assessment and improvement of learning environ…

Compressive Phase Retrieval: Optimal Sample Complexity with Deep Generative Priors

Advances in compressive sensing provided reconstruction algorithms of sparse signals from linear measurements with optimal sample complexity, but natural extensions of this methodology to nonlinear inverse problems have been met with potentially fundamental sample complexity bottlenecks. In particular, tractable algorithms for compressive phase retrieval with sparsity priors have not been able to achieve optimal sample complexity. This has created an open problem in compressive phase retrieval: under generic, phaseless linear measurements, are there tractable reconstruction algorithms that suc…

Generative Adversarial Nets for Information Retrieval: Fundamentals and Advances

Generative adversarial nets (GANs) have been widely studied during the recent development of deep learning and unsupervised learning. With an adversarial training mechanism, GAN manages to train a generative model to fit the underlying unknown real data distribution under the guidance of the discriminative model estimating whether a data instance is real or generated. Such a framework is originally proposed for fitting continuous data distribution such as images, thus it is not straightforward to be directly applied to information retrieval scenarios where the data is mostly discrete, such as …

用于实例检索的簇感知相似度扩散

Cluster-Aware Similarity Diffusion for Instance Retrieval

针对成对亲和图易传播离群点和其他流形错误信息的问题，提出簇感知相似度扩散CAS，通过局部簇内扩散、双向逆约束及邻居平滑提升实例检索与目标重识别效果。

DiffPhase：基于生成式扩散的 STFT 相位检索

DiffPhase: Generative Diffusion-based STFT Phase Retrieval

本文提出 DiffPhase，将语音增强扩散模型改造为面向 STFT 相位检索的生成式插补方法，并通过语音质量与可懂度指标验证其优于经典及现代方法。

Instance-level Heterogeneous Domain Adaptation for Limited-labeled Sketch-to-Photo Retrieval

Although sketch-to-photo retrieval has a wide range of applications, it is costly to obtain paired and rich-labeled ground truth. Differently, photo retrieval data is easier to acquire. Therefore, previous works pre-train their models on rich-labeled photo retrieval data (i.e., source domain) and then fine-tune them on the limited-labeled sketch-to-photo retrieval data (i.e., target domain). However, without co-training source and target data, source domain knowledge might be forgotten during the fine-tuning process, while simply co-training them may cause negative transfer due to domain gaps.…

Exemplar VAE: Linking Generative Models, Nearest Neighbor Retrieval, and Data Augmentation

We introduce Exemplar VAEs, a family of generative models that bridge the gap between parametric and non-parametric, exemplar based generative models. Exemplar VAE is a variant of VAE with a non-parametric prior in the latent space based on a Parzen window estimator. To sample from it, one first draws a random exemplar from a training set, then stochastically transforms that exemplar into a latent code and a new observation. We propose retrieval augmented training (RAT) as a way to speed up Exemplar VAE training by using approximate nearest neighbor search in the latent space to define a lower…

GENIUS：面向通用多模态搜索的生成式框架

GENIUS: A Generative Framework for Universal Multimodal Search

本文提出通用生成式检索框架 GENIUS，通过模态解耦语义量化和查询增强，在多模态、多领域任务中兼顾检索性能与效率，并在 M-BEIR 等基准上超越先前生成式方法。

基于生成式检索的推荐系统

Recommender Systems with Generative Retrieval

该研究提出基于Semantic ID的生成式检索推荐方法，用Transformer序列模型自回归解码物品语义ID，在多数据集上性能超SOTA，还提升了冷启动物品的检索泛化能力。

多视图标识符增强的生成式检索

Multiview Identifiers Enhanced Generative Retrieval

本文提出基于段落内容生成的合成标识符，并与标题、子串构成多视图标识符，以增强生成式检索的段落表示和排序能力，在三个公开数据集上取得最佳效果。

扩展非自回归模型的词表以实现高效生成式检索

Scaling the Vocabulary of Non-autoregressive Models for Efficient Generative Retrieval

本文提出PIXAR，通过将NAR生成式检索模型词表扩展至最多500万个多词实体和短语并优化推理，在低延迟成本下显著提升检索效果与商业收益。

PinRec：统一生成检索用于Pinterest推荐系统

PinRec: Unified Generative Retrieval for Pinterest Recommender Systems

PinRec是一种统一的生成式检索模型，用于Pinterest的所有推荐界面，通过预训练和微调实现单一模型适应不同界面需求，提升搜索保存率4%。

Class-Specific Blind Deconvolutional Phase Retrieval Under a Generative Prior

In this paper, we consider the highly ill-posed problem of jointly recovering two real-valued signals from the phaseless measurements of their circular convolution. The problem arises in various imaging modalities such as Fourier ptychography, X-ray crystallography, and in visible light communication. We propose to solve this inverse problem using alternating gradient descent algorithm under two pretrained deep generative networks as priors; one is trained on sharp images and the other on blur kernels. The proposed recovery algorithm strives to find a sharp image and a blur kernel in the range…

Learning Deep Representations via Contrastive Learning for Instance Retrieval

Instance-level Image Retrieval (IIR), or simply Instance Retrieval, deals with the problem of finding all the images within an dataset that contain a query instance (e.g. an object). This paper makes the first attempt that tackles this problem using instance-discrimination based contrastive learning (CL). While CL has shown impressive performance for many computer vision tasks, the similar success has never been found in the field of IIR. In this work, we approach this problem by exploring the capability of deriving discriminative representations from pre-trained and fine-tuned CL models. To b…

重新审视生成式检索与密集检索：静态价值先验可逆转性能差距

Revisiting Generative vs. Dense Retrieval: A Static Value Prior Reverses the Gap

该研究对比生成式检索与密集检索在推荐任务中的表现，发现静态物品价值先验可逆转生成式检索的价值领先，且工业场景下大模型仍弱于小型密集编码器。

Towards Sample-Optimal Compressive Phase Retrieval with Sparse and Generative Priors

Compressive phase retrieval is a popular variant of the standard compressive sensing problem in which the measurements only contain magnitude information. In this paper, motivated by recent advances in deep generative models, we provide recovery guarantees with near-optimal sample complexity for phase retrieval with generative priors. We first show that when using i.i.d. Gaussian measurements and an $L$-Lipschitz continuous generative model with bounded $k$-dimensional inputs, roughly $O(k \log L)$ samples suffice to guarantee that any signal minimizing an amplitude-based empirical loss functi…

Misspecified Phase Retrieval with Generative Priors

In this paper, we study phase retrieval under model misspecification and generative priors. In particular, we aim to estimate an $n$-dimensional signal $\mathbf{x}$ from $m$ i.i.d.~realizations of the single index model $y = f(\mathbf{a}^T\mathbf{x})$, where $f$ is an unknown and possibly random nonlinear link function and $\mathbf{a} \in \mathbb{R}^n$ is a standard Gaussian vector. We make the assumption $\mathrm{Cov}[y,(\mathbf{a}^T\mathbf{x})^2] \ne 0$, which corresponds to the misspecified phase retrieval problem. In addition, the underlying signal $\mathbf{x}$ is assumed to lie in the ran…

SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval

We tackle the cross-modal retrieval problem, where learning is only supervised by relevant multi-modal pairs in the data. Although the contrastive learning is the most popular approach for this task, it makes potentially wrong assumption that the instances in different pairs are automatically irrelevant. To address the issue, we propose a novel loss function that is based on self-labeling of the unknown semantic classes. Specifically, we aim to predict class labels of the data instances in each modality, and assign those labels to the corresponding instances in the other modality (i.e., swappi…

Phase Retrieval Under a Generative Prior

The phase retrieval problem asks to recover a natural signal $y_0 \in \mathbb{R}^n$ from $m$ quadratic observations, where $m$ is to be minimized. As is common in many imaging problems, natural signals are considered sparse with respect to a known basis, and the generic sparsity prior is enforced via $\ell_1$ regularization. While successful in the realm of linear inverse problems, such $\ell_1$ methods have encountered possibly fundamental limitations, as no computationally efficient algorithm for phase retrieval of a $k$-sparse signal has been proven to succeed with fewer than $O(k^2\log n)$…

Generative Adversarial User Privacy in Lossy Single-Server Information Retrieval

We propose to extend the concept of private information retrieval by allowing for distortion in the retrieval process and relaxing the perfect privacy requirement at the same time. In particular, we study the trade-off between download rate, distortion, and user privacy leakage, and show that in the limit of large file sizes this trade-off can be captured via a novel information-theoretical formulation for datasets with a known distribution. Moreover, for scenarios where the statistics of the dataset is unknown, we propose a new deep learning framework by leveraging a generative adversarial ne…