Pix2Code: Learning to Compose Neural Visual Concepts as Programs
TU Darmstadt（达姆施塔特工业大学）；Hessian Center for AI (hessian.AI)（黑森人工智能中心）；Eindhoven University of Technology（埃因霍温理工大学）；Centre for Cognitive Science, TU Darmstadt（达姆施塔特工业大学认知科学中心）；German Research Center for AI (DFKI)（德国人工智能研究中心）
Antonia Wüst, Wolfgang Stammer, Quentin Delfosse, Devendra Singh Dhami, Kristian Kersting
The challenge in learning abstract concepts from images in an unsupervised fashion lies in the required integration of visual perception and generalizable relational reasoning. Moreover, the unsupervised nature of this task makes it necessary for human users to be able to understand a model's learnt concepts and potentially revise false behaviours. To tackle both the generalizability and interpretability constraints of visual concept learning, we propose Pix2Code, a framework that extends program synthesis to visual relational reasoning by utilizing the abilities of both explicit, compositional symbolic and implicit neural representations. This is achieved by retrieving object representations from images and synthesizing relational concepts as lambda-calculus programs. We evaluate the diverse properties of Pix2Code on the challenging reasoning domains, Kandinsky Patterns and CURI, thereby testing its ability to identify compositional visual concepts that generalize to novel data and concept configurations. Particularly, in stark contrast to neural approaches, we show that Pix2Code's representations remain human interpretable and can be easily revised for improved performance.
JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence
The University of Hong Kong（香港大学）；Shanghai AI Laboratory（上海人工智能实验室）；Nanjing University（南京大学）；Carnegie Mellon University（卡内基梅隆大学）；Shanghai Innovation Institute（上海创新研究院）
