How do two distributions of texts differ? Humans are slow at answering this, since discovering patterns might require tediously reading through hundreds of samples. We propose to automatically summarize the differences by "learning a natural language hypothesis": given two distributions $D_{0}$ and $D_{1}$, we search for a description that is more often true for $D_{1}$, e.g., "is military-related." To tackle this problem, we fine-tune GPT-3 to propose descriptions with the prompt: "[samples of $D_{0}$] + [samples of $D_{1}$] + the difference between them is_____." We then re-rank the descriptions by checking how often they hold on a larger set of samples with a learned verifier. On a benchmark of 54 real-world binary classification tasks, while GPT-3 Curie (13B) only generates a description similar to human annotation 7% of the time, the performance reaches 61% with fine-tuning and re-ranking, and our best system using GPT-3 Davinci (175B) reaches 76%. We apply our system to describe distribution shifts, debug dataset shortcuts, summarize unknown tasks, and label text clusters, and present analyses based on automatically generated descriptions.
Bank distress in the news: Describing events through deep learning
Turku Centre for Computer Science – TUCS（图尔库计算机科学中心）；Åbo Akademi University（奥博学术大学）；Goethe University Frankfurt am Main（法兰克福大学）；Hanken School of Economics（汉肯经济学院）；Arcada University of Applied Sciences（阿卡达应用科学大学）
Differences in Text Generated by Diffusion and Autoregressive Language Models
Shanghai Qi Zhi Institute（上海启智研究院）；Institute for Interdisciplinary Information Sciences, Tsinghua University（清华大学交叉信息研究院）；Xiongan AI Institute（雄安人工智能研究院）
On the Difference of BERT-style and CLIP-style Text Encoders
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Hong Kong University of Science and Technology（香港科技大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
