NeurIPS 2026
Likelihood-Free Generative Policy Optimization
Improve diffusion and flow policies without evaluating intractable action likelihoods.
Abstract
Diffusion and flow models offer expressive policies for continuous control, but their action likelihoods are generally intractable. This makes many standard policy optimization methods difficult to apply. We introduce Likelihood-Free Generative Policy Optimization (LFGPO), a two-stage framework that separates policy improvement from generative model training. A lightweight ratio model first captures the desired policy update. A conditional drift-matching objective then transfers that update to a diffusion or flow policy. The framework accommodates update rules including PPO and GRPO. Our analysis relates the target policy to the Doob h-transform, proves gradient equivalence between the practical and ideal drift-matching objectives, and establishes a minimum-control-energy interpretation in the diffusion setting. Experiments on MuJoCo continuous-control tasks evaluate both diffusion and flow implementations.
Key results
Separate improvement from fitting
Learn a likelihood-ratio model for the policy update, then distill the update into an expressive generative policy.
Diffusion and flow policies
Use conditional drift matching to support multiple policy classes and update rules without exact action-likelihood evaluation.
An optimal-control view
Characterize the target through the Doob h-transform and connect the diffusion objective to minimum-energy control under the specified endpoint constraints.
BibTeX
Provisional citation based on the authors' acceptance information. Official BibTeX pending.
% Provisional citation; official BibTeX pending.
@inproceedings{su2026lfgpo,
title = {{Likelihood-Free Generative Policy Optimization}},
author = {Maojiang Su and Qijie Zhu and Shuyang Yu and Yu Xiao and Minshuo Chen and Zhaoran Wang and Han Liu},
booktitle = {Neural Information Processing Systems (NeurIPS) 2026},
year = {2026}
}