Small-scale models offer various computational advantages, and yet to which extent size is critical for problem-solving abilities remains an open question. Specifically for solving grade school math, the smallest model size so far required to break the 80\% barrier on the GSM8K benchmark remains to be 34B. Our work studies how high-quality datasets may be the key for small language models to acquire mathematical reasoning. We introduce \texttt{TinyGSM}, a synthetic dataset of 12.3M grade school math problems paired with Python solutions, generated fully by GPT-3.5. After finetuning on \texttt{TinyGSM}, we find that a duo of a 1.3B generation model and a 1.3B verifier model can achieve 81.5\% accuracy, outperforming existing models that are orders of magnitude larger. This also rivals the performance of the GPT-3.5 ``teacher'' model (77.4\%), from which our model's training data is generated. Our approach is simple and has two key components: 1) the high-quality dataset \texttt{TinyGSM}, 2) the use of a verifier, which selects the final outputs from multiple candidate generations.
Domain-Adapted Small Language Models with Hybrid Post-Processing: Achieving Cost-Efficient, Low-Latency Multi-Label Structured Prediction via LoRA Fine-Tuning on Scarce Data
MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering
University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);The Hong Kong Polytechnic University(香港理工大学);Huawei(华为)
World Models with Hints of Large Language Models for Goal Achieving
The Ohio State University(俄亥俄州立大学);University of Maryland, College Park(马里兰大学学院公园分校);Tsinghua University(清华大学);Shanghai Qi Zhi Institute(上海期智研究院);Shanghai AI Lab(上海人工智能实验室)
MEUV: Achieving Fine-Grained Capability Activation in Large Language Models via Mutually Exclusive Unlock Vectors
Xin Tong School of Information and Cyber Security People’s Public Security University of China(信息与网络安全学院 中国人民公安大学);Zhi Lin School of Safety Science Tsinghua University(安全科学学院 清华大学);Jingya Wang School of Information and Cyber Security People’s Public Security University of China(信息与网络安全学院 中国人民公安大学);Meng Han Intelligent Fusion Research Center Zhejiang University(智能融合研究中心 浙江大学);Bo Jin* The Third Research Institute of the Ministry of Public Security of China(中华人民共和国公安部第三研究所)