The machine learning community has mainly relied on real data to benchmark algorithms as it provides compelling evidence of model applicability. Evaluation on synthetic datasets can be a powerful tool to provide a better understanding of a model's strengths, weaknesses, and overall capabilities. Gaining these insights can be particularly important for generative modeling as the target quantity is completely unknown. Multiple issues related to the evaluation of generative models have been reported in the literature. We argue those problems can be avoided by an evaluation based on ground truth. General criticisms of synthetic experiments are that they are too simplified and not representative of practical scenarios. As such, our experimental setting is tailored to a realistic generative task. We focus on categorical data and introduce an appropriately scalable evaluation method. Our method involves tasking a generative model to learn a distribution in a high-dimensional setting. We then successively bin the large space to obtain smaller probability spaces where meaningful statistical tests can be applied. We consider increasingly large probability spaces, which correspond to increasingly difficult modeling tasks and compare the generative models based on the highest task difficulty they can reach before being detected as being too far from the ground truth. We validate our evaluation procedure with synthetic experiments on both synthetic generative models and current state-of-the-art categorical generative models.
Physics-Informed Generative Solver: Bridging Data-Driven Priors and Conservation Laws for Stable Spatiotemporal Field Reconstruction
School of Advanced Manufacturing Engineering, Nanjing University(南京大学先进制造工程学院);National Laboratory of Solid State Microstructures, Nanjing University(南京大学固态微结构国家实验室);Suzhou Acoustics Industry Technology Research Institute Co., Ltd.(苏州声学工业技术研究所有限公司);School of Mechanical and Electric Engineering, Soochow University(苏州大学机械与电子工程学院);Shishan Laboratory, Nanjing University(仙山实验室)
Bridging the Semantic Gap for Categorical Data Clustering via Large Language Models
School of Computer Science and Technology(计算机科学与技术学院);Guangdong University of Technology(广东技术大学);Department of Computer Science(计算机科学系);Hong Kong Baptist University(香港 Baptist 大学)
EchoAlign: Bridging Generative and Discriminative Learning under Noisy Labels
School of Software, Shandong University, Jinan 250100, China(山东大学软件学院,济南250100,中国);Sydney AI Centre, The University of Sydney, Sydney, NSW 2050, Australia(悉尼大学悉尼人工智能中心,悉尼,新南威尔士州2050,澳大利亚)
Generative Example-Based Explanations: Bridging the Gap between Generative Modeling and Explainability
Center for Artificial Intelligence and Robotics(人工智能与机器人中心);Technical University of Applied Sciences Würzburg-Schweinfurt(应用科学大学魏玛-施维林分校);Bielefeld University(比勒菲尔德大学)
Generative Bridging Network in Neural Sequence Prediction
University of California Santa Barbara(加州大学圣巴巴拉分校);Harbin Institute of Technology(哈尔滨工业大学);Microsoft Research Asia(微软亚洲研究院);University of Science Technology of China(中国科学技术大学);Beijing University of Aeronautics Astronautics(北京航空航天大学)
When Hard Negatives Hurt: Bridging the Generative-Discriminative Gap in Hard Negative Synthesis for Retrieval
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Huawei Technologies Co., Ltd.(华为技术有限公司);City University of Hong Kong(香港城市大学);School of Cyber Science and Technology, Sun Yat-sen University(中山大学信息科学与技术学院);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院);The Hong Kong University of Science and Technology(香港科学与技术大学);Huawei Noah’s Ark Lab(华为诺亚实验室)
Bridging Generalization Gap of Heterogeneous Federated Clients Using Generative Models
School of Computing Technologies(计算技术学院);RMIT University(皇家墨尔本理工大学);Department of Computing Technologies(计算技术系);Swinburne University of Technology(斯威本理工大学)
Bridging Generative and Discriminative Noisy-Label Learning via Direction-Agnostic EM Formulation
Cornell Tech(康奈尔科技);Cornell University(康奈尔大学);Department of Radiology, Stanford University(斯坦福大学放射科);Oxford University(牛津大学);Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey(视觉、语音和信号处理中心(CVSSP)、塞维利亚大学)
Bridging Generative and Discriminative Learning: Few-Shot Relation Extraction via Two-Stage Knowledge-Guided Pre-training
University of Electronic Science and Technology of China(电子科技大学);Nanyang Technological University(南洋理工大学);Information Engineering University(信息工程大学);National University of Defense Technology(国防科技大学)
Generative Adversarial Networks Bridging Art and Machine Intelligence
Imperial College London(帝国理工学院);The University of Texas at Dallas(德克萨斯大学达拉斯分校);Indiana University(印第安纳大学);Xi’an Jiaotong-Liverpool University(西交利物浦大学);Georgia Institute of Technology(佐治亚理工学院);Kyoto University(京都大学);AppCubic;Rutgers University(罗格斯大学);Purdue University(普渡大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);National Taiwan Normal University(台湾师范大学);University of Hawaii(夏威夷大学);Hong Kong University of Science and Technology(香港科技大学);School of Visual Arts(视觉艺术学院);Emory University(埃默里大学);Aarhus University(奥胡斯大学);Zhejiang University(浙江大学);University of Edinburgh(爱丁堡大学);National Tsing Hua University(国立清华大学);University of Manchester(曼彻斯特大学);Pingtan Research Institute of Xiamen University(厦门大学平潭研究院);University of Liverpool(利物浦大学);JTB Technology Corp.(JTB科技公司)