Large language models (LLMs) continue to struggle with mathematical reasoning, and common post-training pipelines often reduce each generated solution to a binary outcome: correct or incorrect. This perspective is limiting in practice, as failures in chain-of-thought (CoT) reasoning are frequently structured; solutions may appear convincing while containing subtle logical, algebraic, or numerical flaws. Meanwhile, reinforcement learning from human feedback (RLHF) variants that rely on large reward models or LLM-as-a-judge signals are often expensive, difficult to scale, and unstable to iterate. We propose a lightweight and pragmatic post-training pipeline that targets such structured errors under realistic compute budgets. Starting from supervised fine-tuning (SFT) on MetaMathQA-style CoT data, we introduce a compact MathVerifier that decomposes a candidate solution into a six-dimensional error profile and aggregates it into interpretable wrongness and absurdity scores. These verifier signals serve two roles: (i) mining hard negatives that are near-correct yet structurally flawed, and (ii) defining per-sample importance weights that emphasize the most informative preference pairs. We integrate both into an offline Direct Preference Optimization (DPO) objective via a verifier-guided weighted formulation. Experiments on a 1.5B-parameter Qwen2.5 model show that verifier-guided, weighted DPO yields more targeted improvements than vanilla SFT and unweighted DPO, particularly on problems where solutions are numerically close to correct but logically inconsistent, while avoiding the overhead of training large reward models or relying on external judges.
Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering
University of Minnesota（明尼苏达大学）；Virginia Tech（弗吉尼亚理工大学）；Xi’an University of Technology（西安理工大学）；Lahore University of Management Sciences（拉合尔管理科学大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
DavIR: Data Selection via Implicit Reward for Large Language Models
ByteDance, Inc.（字节跳动公司）；Cold Spring Harbor Laboratory（冷泉港实验室）；Shanghai Jiao Tong University（上海交通大学）；National University of Singapore（新加坡国立大学）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）
The variational autoencoder (VAE) framework remains a popular option for training unsupervised generative models, especially for discrete data where generative adversarial networks (GANs) require workaround to create gradient for the generator. In our work modeling US postal addresses, we show that our discrete VAE with tree recursive architecture demonstrates limited capability of capturing field correlations within structured data, even after overcoming the challenge of posterior collapse with scheduled sampling and tuning of the KL-divergence weight $β$. Worse, VAE seems to have difficulty …
The variational autoencoder (VAE) framework is a popular option for training unsupervised generative models, featuring ease of training and latent representation of data. The objective function of VAE does not guarantee to achieve the latter, however, and failure to do so leads to a frequent failure mode called posterior collapse. Even in successful cases, VAEs often result in low-precision reconstructions and generated samples. The introduction of the KL-divergence weight $β$ can help steer the model clear of posterior collapse, but its tuning is often a trial-and-error process with no guidin…
Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees
National Key Laboratory for Novel Software Technology（计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；Alibaba International Digital Commerce（阿里巴巴国际数字商业集团）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance
Massachusetts Institute of Technology, Boston, MA, USA（麻省理工学院）；Harvard University, Boston, MA, USA（哈佛大学）；MIT-IBM Watson AI Lab, Boston, MA, USA（MIT-IBM Watson AI实验室）；University of Illinois Urbana-Champaign, Urbana, IL, USA（伊利诺伊大学厄巴纳-香槟分校）
