Instruction tuning plays a critical role in aligning large language models (LLMs) with human preference. Despite the vast amount of open instruction datasets, naively training a LLM on all existing instructions may not be optimal and practical. To pinpoint the most beneficial datapoints, data assessment and selection methods have been proposed in the fields of natural language processing (NLP) and deep learning. However, under the context of instruction tuning, there still exists a gap in knowledge on what kind of data evaluation metrics can be employed and how they can be integrated into the selection mechanism. To bridge this gap, we present a comprehensive review on existing literature of data assessment and selection especially for instruction tuning of LLMs. We systematically categorize all applicable methods into quality-based, diversity-based, and importance-based ones where a unified, fine-grained taxonomy is structured. For each category, representative methods are elaborated to describe the landscape of relevant research. In addition, comparison between the latest methods is conducted on their officially reported results to provide in-depth discussions on their limitations. Finally, we summarize the open challenges and propose the promosing avenues for future studies. All related contents are available at https://github.com/yuleiqin/fantastic-data-engineering.
CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；King Abdullah University of Science and Technology（科廷大学）；Fudan University（复旦大学）
MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；ARC Lab, Tencent PCG（腾讯PCG ARC实验室）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）
NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models
AIML, University of Adelaide（阿德莱德大学AIML）；Adobe Research（奥多比研究院）；University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）；University of California, Santa Cruz（加利福尼亚大学圣克鲁兹分校）
VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models
School of Data Science, Fudan University（复旦大学数据科学学院）；ByteDance（字节跳动）；School of Computer Science, Fudan University（复旦大学计算机科学学院）；Research Institute of Intelligent and Complex Systems, Fudan University（复旦大学智能与复杂系统研究院）；Shanghai Innovation Institute（上海创新研究院）
Proximity QA: Unleashing the Power of Multi-Modal Large Language Models for Spatial Proximity Analysis
School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）；School of Electronics Engineering and Computer Science, Peking University（北京大学电子工程与计算机科学学院）；Intel Lab China（英特尔中国实验室）
HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding
Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；SenseTime Research（商汤科技研究院）；Nanjing University（南京大学）
