超越代理：用于离线GFlowNet训练的轨迹蒸馏指导
Beyond the Proxy: Trajectory-Distilled Guidance for Offline GFlowNet Training
AI总结：
提出轨迹蒸馏GFlowNet（TD-GFN），利用逆强化学习从离线轨迹中提取稠密边奖励，通过DAG剪枝和优先反向采样指导策略，避免代理模型，提升离线GFlowNet训练的收敛速度和样本质量。
摘要：
生成流网络（GFlowNets）擅长采样多样化的高奖励对象。在许多实际应用中，由于无法进行主动奖励查询，这些模型必须使用静态离线数据集进行训练。主流的训练方法通常依赖代理模型为在线采样的轨迹提供奖励反馈。然而，由于数据稀缺或评估成本高，构建可靠的代理往往具有挑战性。虽然现有的无代理方法试图解决这一问题，但它们通常施加粗糙的约束，限制了模型有效探索的能力。为了克服这些限制，我们提出了轨迹蒸馏GFlowNet（TD-GFN），一种新颖的无代理训练框架。TD-GFN利用逆强化学习（IRL）从离线轨迹中提取稠密的、转移级别的边奖励，为高效探索提供丰富的结构指导。关键的是，为了确保鲁棒性，这些奖励通过DAG剪枝和优先反向采样间接指导策略。这种设计确保梯度更新仅依赖于数据集中的真实终端奖励，从而防止错误传播。实验结果表明，TD-GFN在收敛速度和样本质量上显著优于广泛的现有基线，为离线GFlowNet训练建立了更鲁棒和高效的范式。
英文摘要：
Generative Flow Networks (GFlowNets) excel at sampling diverse, high-reward objects. In many practical applications where active reward queries are infeasible, these models must be trained using static offline datasets. Prevailing training methods typically rely on a proxy model to provide reward feedback for online sampled trajectories. However, constructing a reliable proxy is often challenging due to data scarcity or high evaluation costs. While existing proxy-free approaches attempt to address this, they often impose coarse constraints that limit the model's ability to explore effectively. To overcome these limitations, we propose Trajectory-Distilled GFlowNet (TD-GFN), a novel proxy-free training framework. TD-GFN utilizes inverse reinforcement learning (IRL) to extract dense, transition-level edge rewards from offline trajectories, providing rich structural guidance for efficient exploration. Crucially, to ensure robustness, these rewards guide the policy indirectly through DAG pruning and prioritized backward sampling. This design ensures that gradient updates rely exclusively on ground-truth terminal rewards from the dataset, thereby preventing error propagation. Empirical results demonstrate that TD-GFN significantly outperforms a broad range of existing baselines in both convergence speed and sample quality, establishing a more robust and efficient paradigm for offline GFlowNet training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
