The web provides a rich, open-domain environment with textual, structural, and spatial properties. We propose a new task for grounding language in this environment: given a natural language command (e.g., "click on the second article"), choose the correct element on the web page (e.g., a hyperlink or text box). We collected a dataset of over 50,000 commands that capture various phenomena such as functional references (e.g. "find who made this site"), relational reasoning (e.g. "article by john"), and visual reasoning (e.g. "top-most article"). We also implemented and analyzed three baseline models that capture different phenomena present in the dataset.
Mapping the Web of Science, a large-scale graph and text-based dataset with LLM embeddings
Tim Kunt Digital Data and Information for Society, Science, and Culture, Zuse Institute Berlin, Takustr. 7, 14195 Berlin, Germany(Zuse Institute Berlin);Annika Buchholz Digital Data and Information for Society, Science, and Culture, Zuse Institute Berlin, Takustr. 7, 14195 Berlin, Germany(Zuse Institute Berlin);Imene Khebouri Digital Data and Information for Society, Science, and Culture, Zuse Institute Berlin, Takustr. 7, 14195 Berlin, Germany(Zuse Institute Berlin);Thorsten Koch Software and Algorithms for Discrete Optimization, Technische Universität Berlin, Straße des 17. Juni 135, 10623 Berlin, Germany Applied Optimization, Zuse Institute Berlin, Takustr. 7, 14195 Berlin, Germany(Technische Universität Berlin, Zuse Institute Berlin);Ida Litzel Digital Data and Information for Society, Science, and Culture, Zuse Institute Berlin, Takustr. 7, 14195 Berlin, Germany(Zuse Institute Berlin);Thi Huong Vu Digital Data and Information for Society, Science, and Culture, Zuse Institute Berlin, Takustr. 7, 14195 Berlin, Germany(Zuse Institute Berlin);Institute of Mathematics, Vietnam Academy of Science and Technology, 10072 Hanoi, Vietnam(Vietnam Academy of Science and Technology)