Learning Action Conditions from Instructional Manuals for Instruction Understanding
University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of Cambridge(剑桥大学);Information Sciences Institute, University of Southern California(南加州大学信息科学学院)
Te-Lin Wu, Caiqi Zhang, Qingyuan Hu, Alex Spangher, Nanyun Peng
The ability to infer pre- and postconditions of an action is vital for comprehending complex instructions, and is essential for applications such as autonomous instruction-guided agents and assistive AI that supports humans to perform physical tasks. In this work, we propose a task dubbed action condition inference, and collecting a high-quality, human annotated dataset of preconditions and postconditions of actions in instructional manuals. We propose a weakly supervised approach to automatically construct large-scale training instances from online instructional manuals, and curate a densely human-annotated and validated dataset to study how well the current NLP models can infer action-condition dependencies in the instruction texts. We design two types of models differ by whether contextualized and global information is leveraged, as well as various combinations of heuristics to construct the weak supervisions. Our experimental results show a >20% F1-score improvement with considering the entire instruction contexts and a >6% F1-score benefit with the proposed heuristics.
LLMs can be easily Confused by Instructional Distractions
IPAI, Seoul National University(IPAI,首尔国立大学);LG AI Research(LG人工智能研究所);Dept. of ECE, Seoul National University(电子工程系,首尔国立大学);SNU-LG AI Research Center(SNU-LG人工智能研究所)
Instructional Goal-Aligned Question Generation for Student Evaluation in Virtual Lab Settings: How Closely Do LLMs Actually Align?
Department of EdPsych, University of Wisconsin-Madison(威斯康星大学麦迪逊分校教育心理学系);Department of CS, Wichita State University(威斯康星州立大学Wichita分校计算机科学系);Department of CSSE, Auburn University(阿伯茨罕大学计算机科学与工程系)
A Dataset for Medical Instructional Video Classification and Question Answering
Lister Hill National Center for Biomedical Communications(利斯特·希尔国家生物医学通信中心);National Library of Medicine(国家医学图书馆);National Institutes of Health(美国国立卫生研究院)